feat(client/v3/naming): add Attributes into Endpoint and deprecate Metadata #19785

amosehiguese · 2025-04-23T14:09:08Z

This implementation adds the new Attributes *attributes.Attributes into Endpoint and deprecate Metadata accordingly. The metadata field is still supported for backward compatibility

Fixes #19706

cc: @ahrtr @siyuanfoundation

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

k8s-ci-robot · 2025-04-23T14:09:13Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: amosehiguese
Once this PR has been reviewed and has the lgtm label, please assign serathius for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-04-23T14:09:19Z

Hi @amosehiguese. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ahrtr · 2025-04-23T14:19:45Z

client/v3/naming/resolver/resolver.go

 			r.cc.UpdateState(gresolver.State{Endpoints: eps})
 		}
 	}
 }

-func convertToGRPCEndpoint(ups map[string]*endpoints.Update) []gresolver.Endpoint {
+func convertToGRPCEndpoint(lg *zap.Logger, ups map[string]*endpoints.Update) []gresolver.Endpoint {


suggest not to change the signature of the function. We don't know how users use the resovler, it may generate huge number of warning message.

Suggested change

func convertToGRPCEndpoint(lg *zap.Logger, ups map[string]*endpoints.Update) []gresolver.Endpoint {

func convertToGRPCEndpoint(ups map[string]*endpoints.Update) []gresolver.Endpoint {

I was thinking since it is a private function and only called within the watch function(in the entire etcd repo), it wouldn't have much impact.

That said, do i revert or leave it as is?

I could take another approach though

please revert this change.

usually it isn't a good pattern to print log in a lower level or utility package

I will revert then. Thank you for the feedback

ahrtr · 2025-04-23T14:46:55Z

please squash the commits.

We also need to update the doc https://etcd.io/docs/v3.5/dev-guide/grpc_naming/

the Metadata has deprecated, use Attributes instead
Mark the gRPC naming and discovery feature as experimental, depending on how the resolver package in grpc-go evolves.

amosehiguese · 2025-04-23T14:49:57Z

Alright. I'll do just that.

ahrtr · 2025-04-23T15:56:30Z

the Metadata has deprecated, use Attributes instead

let's update this first together with this PR.

after this PR gets merged, backport it to 3.6 and 3.5.
raise a PR in etcd-io/website to update the doc

Mark the gRPC naming and discovery feature as experimental, depending on how the resolver package in grpc-go evolves.

we will discuss & resolve this separately.

cc @dfawley @fuweid @serathius @ivanvc

ahrtr · 2025-04-23T15:58:43Z

/ok-to-test

ahrtr · 2025-04-23T16:17:01Z

client/v3/naming/resolver/resolver.go

-					Metadata: up.Endpoint.Metadata,
+					Addr:       up.Endpoint.Addr,
+					Metadata:   up.Endpoint.Metadata,
+					Attributes: up.Endpoint.Attributes,


@dfawley

Probably eventually we might want to use Endpoint.Attributes instead of Address.Attributes, as we only care about the LB policy?

Currently we assume each endpoint has only one address, but actually it may contain multiple addresses (we won't change the behavior for now).

Yeah, I saw it and then realised that the attributes field in the Endpoint was specific to LB-Policy which is the usecase for etcd and wondered why that wasn't used instead. But I have clarity on that now. Thanks

codecov · 2025-04-23T16:21:50Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.04%. Comparing base (8f933a5) to head (da7ba13).
Report is 2 commits behind head on main.

Additional details and impacted files

Files with missing lines	Coverage Δ
client/v3/naming/endpoints/endpoints.go	`100.00% <ø> (ø)`
client/v3/naming/endpoints/endpoints_impl.go	`75.20% <100.00%> (+3.50%)`	⬆️
client/v3/naming/resolver/resolver.go	`80.70% <100.00%> (+0.34%)`	⬆️
server/proxy/grpcproxy/register.go	`71.15% <100.00%> (-4.41%)`	⬇️

... and 24 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #19785      +/-   ##
==========================================
- Coverage   68.81%   67.04%   -1.78%     
==========================================
  Files         421      421              
  Lines       35863    35886      +23     
==========================================
- Hits        24678    24058     -620     
- Misses       9754    10425     +671     
+ Partials     1431     1403      -28

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8f933a5...da7ba13. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ahrtr · 2025-04-23T16:27:50Z

You also need to persist the new added attributes (line 41 in internal/update.go), and read it back in List,

etcd/client/v3/naming/endpoints/endpoints_impl.go

Lines 175 to 192 in 8f933a5

    
           func (m *endpointManager) List(ctx context.Context) (Key2EndpointMap, error) { 
        
           	key := m.target + "/" 
        
           	resp, err := m.client.Get(ctx, key, clientv3.WithPrefix(), clientv3.WithSerializable()) 
        
           	if err != nil { 
        
           		return nil, err 
        
           	} 
        
           	eps := make(Key2EndpointMap) 
        
           	for _, kv := range resp.Kvs { 
        
           		var iup internal.Update 
        
           		if err := json.Unmarshal(kv.Value, &iup); err != nil { 
        
           			continue 
        
           		} 
        
           		eps[string(kv.Key)] = Endpoint{Addr: iup.Addr, Metadata: iup.Metadata} 
        
           	} 
        
           	return eps, nil 
        
           }

amosehiguese · 2025-04-23T16:27:59Z

let's update this first together with this PR.

Alright.

after this PR gets merged, backport it to 3.6 and 3.5.

I'm on standby.

raise a PR in etcd-io/website to update the doc

I will hold off on this then

ahrtr · 2025-04-23T16:33:11Z

You might also need to update grpcproxy

etcd/server/proxy/grpcproxy/register.go

Line 75 in 8f933a5

endpoint := endpoints.Endpoint{Addr: addr, Metadata: getMeta()}

amosehiguese · 2025-04-23T16:37:51Z

You also need to persist the new added attributes (line 41 in internal/update.go), and read it back in List,

etcd/client/v3/naming/endpoints/endpoints_impl.go

Lines 175 to 192 in 8f933a5

func (m *endpointManager) List(ctx context.Context) (Key2EndpointMap, error) {

key := m.target + "/"

resp, err := m.client.Get(ctx, key, clientv3.WithPrefix(), clientv3.WithSerializable())

if err != nil {

return nil, err

}

eps := make(Key2EndpointMap)

for _, kv := range resp.Kvs {

var iup internal.Update

if err := json.Unmarshal(kv.Value, &iup); err != nil {

continue

}

eps[string(kv.Key)] = Endpoint{Addr: iup.Addr, Metadata: iup.Metadata}

}

return eps, nil

}

Got it.

amosehiguese · 2025-04-23T16:45:12Z

You might also need to update grpcproxy

Pretty much adjust every other usage of the endpoint struct to account for the new Attributes field.

Will do that.

…tadata This implementation adds the new Attributes *attributes.Attributes into Endpoint and deprecate Metadata accordingly. The metadata field is still supported for backward compatibility Signed-off-by: amosehiguese <[email protected]>

amosehiguese · 2025-04-24T00:49:13Z

Two files may also need adjustments but I thought to run it by you first.

👉 endpoints_test.go [integration tests]

etcd/tests/integration/clientv3/naming/endpoints_test.go

Line 46 in 8f933a5

e1 := endpoints.Endpoint{Addr: "127.0.0.1", Metadata: "metadata"}

👉 cluster.go [grpc-proxy]
In MembersList, If proxies are registered, it will lists them using their metadata and endpoint addresses.

etcd/server/proxy/grpcproxy/cluster.go

Line 146 in 8f933a5

m, err := decodeMeta(fmt.Sprint(upt.Metadata))

What would you have me do to these files?

ahrtr · 2025-04-24T09:26:31Z

I think

we need update (or add) the test to cover the new field Attributes
we shouldn't break any usage on the existing field Metadata.

amosehiguese · 2025-04-24T09:38:37Z

I think

we need update (or add) the test to cover the new field Attributes

we shouldn't break any usage on the existing field Metadata.

Alright then.

ahrtr · 2025-04-24T10:01:47Z

I just mark this as a priority/important-soon task.

As mentioned above #19785 (comment), I think we need to backport this PR to release-3.6 and release-3.5, of course we will evaluate the impact. The goal is to get it included in 3.6.0, so that we can remove it in 3.7; otherwise, we will have to completely get rid of the deprecated Metadata field in 3.8 (~6 years later based on current cadence)

cc @fuweid @serathius @ivanvc @jmhbnz

ahrtr · 2025-04-24T10:02:35Z

also cc grpc-go tech-lead @dfawley, thx for all the help so far.

amosehiguese · 2025-04-24T13:42:16Z

@ahrtr
Quick question. Are flaky tests expected in the end-to-end test suite? I’ve been running make test-e2e and sometimes it passes, but other times it fails without me changing anything. Just want to be sure if that’s expected or if something else is going on, because it’s been slowing down my push.

...and those tests take quite a bit of time to run

amosehiguese · 2025-04-24T13:43:48Z

I just mark this as a priority/important-soon task.

I will be done with this from my end once I'm able to resolve whether or not flaky tests are expected in the e2e suite

ahrtr · 2025-04-24T13:48:36Z

Quick question. Are flaky tests expected in the end-to-end test suite?

You need to ensure the test failures are not caused by your changes.

amosehiguese · 2025-04-24T14:27:33Z

Quick question. Are flaky tests expected in the end-to-end test suite?

You need to ensure the test failures are not caused by your changes.

I think I’ll have to run it against the main branch a couple of times to be sure.

amosehiguese · 2025-04-24T17:09:26Z

Same error occured on the main branch after running successfully in the previous run. Seems to look like a connection timeout.

What do you reckon I do? push?

cc: @ahrtr

ahrtr · 2025-04-24T17:36:30Z

Same error occured on the main branch after running successfully in the previous run. Seems to look like a connection timeout.

Windows is only tier 3 support, please try to test it on a linux platform.

What do you reckon I do? push?

Pushing to your dev branch won't break anything. Please review your PR yourself first to ensure it's the best you can deliver.

amosehiguese · 2025-04-24T17:44:23Z

Oh, I am actually working on a linux machine running on Azure cloud. This is just my backup pc, so I am forced to doing everything on the cloud.

Pushing to your dev branch won't break anything. Please review your PR yourself first to ensure it's the best you can deliver.

I definitely will. Thanks for the feedback

k8s-ci-robot added the area/clientv3 label Apr 23, 2025

k8s-ci-robot added the needs-ok-to-test label Apr 23, 2025

k8s-ci-robot added the size/M label Apr 23, 2025

ahrtr reviewed Apr 23, 2025

View reviewed changes

ahrtr mentioned this pull request Apr 23, 2025

Remove the deprecated grpc.DialContext #19772

Open

k8s-ci-robot added size/S and removed size/M labels Apr 23, 2025

amosehiguese force-pushed the issue-19706/deprecate-address-metadata branch from 0c64765 to cbf86a8 Compare April 23, 2025 15:36

k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels Apr 23, 2025

ahrtr reviewed Apr 23, 2025

View reviewed changes

amosehiguese force-pushed the issue-19706/deprecate-address-metadata branch from cbf86a8 to da7ba13 Compare April 24, 2025 00:03

k8s-ci-robot added size/M and removed size/S labels Apr 24, 2025

ahrtr added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Apr 24, 2025

ahrtr mentioned this pull request Apr 24, 2025

Remove the dependency on grpc-go's experimental API (e.g. resolver & balancer) #15145

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(client/v3/naming): add Attributes into Endpoint and deprecate Metadata #19785

feat(client/v3/naming): add Attributes into Endpoint and deprecate Metadata #19785

amosehiguese commented Apr 23, 2025 •

edited

Loading

k8s-ci-robot commented Apr 23, 2025

k8s-ci-robot commented Apr 23, 2025

ahrtr Apr 23, 2025

amosehiguese Apr 23, 2025 •

edited

Loading

ahrtr Apr 23, 2025

ahrtr Apr 23, 2025

amosehiguese Apr 23, 2025 •

edited

Loading

ahrtr commented Apr 23, 2025

amosehiguese commented Apr 23, 2025

ahrtr commented Apr 23, 2025

ahrtr commented Apr 23, 2025

ahrtr Apr 23, 2025

amosehiguese Apr 23, 2025 •

edited

Loading

codecov bot commented Apr 23, 2025 •

edited

Loading

ahrtr commented Apr 23, 2025

amosehiguese commented Apr 23, 2025 •

edited

Loading

ahrtr commented Apr 23, 2025

amosehiguese commented Apr 23, 2025

amosehiguese commented Apr 23, 2025

amosehiguese commented Apr 24, 2025

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025

ahrtr commented Apr 24, 2025

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025 •

edited

Loading

amosehiguese commented Apr 24, 2025 •

edited

Loading

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025

amosehiguese commented Apr 24, 2025

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025 •

edited

Loading

	func convertToGRPCEndpoint(lg zap.Logger, ups map[string]endpoints.Update) []gresolver.Endpoint {
	func convertToGRPCEndpoint(ups map[string]*endpoints.Update) []gresolver.Endpoint {

feat(client/v3/naming): add Attributes into Endpoint and deprecate Metadata #19785

Are you sure you want to change the base?

feat(client/v3/naming): add Attributes into Endpoint and deprecate Metadata #19785

Conversation

amosehiguese commented Apr 23, 2025 • edited Loading

k8s-ci-robot commented Apr 23, 2025

k8s-ci-robot commented Apr 23, 2025

ahrtr Apr 23, 2025

Choose a reason for hiding this comment

amosehiguese Apr 23, 2025 • edited Loading

Choose a reason for hiding this comment

ahrtr Apr 23, 2025

Choose a reason for hiding this comment

ahrtr Apr 23, 2025

Choose a reason for hiding this comment

amosehiguese Apr 23, 2025 • edited Loading

Choose a reason for hiding this comment

ahrtr commented Apr 23, 2025

amosehiguese commented Apr 23, 2025

ahrtr commented Apr 23, 2025

ahrtr commented Apr 23, 2025

ahrtr Apr 23, 2025

Choose a reason for hiding this comment

amosehiguese Apr 23, 2025 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Apr 23, 2025 • edited Loading

Codecov Report

ahrtr commented Apr 23, 2025

amosehiguese commented Apr 23, 2025 • edited Loading

ahrtr commented Apr 23, 2025

amosehiguese commented Apr 23, 2025

amosehiguese commented Apr 23, 2025

amosehiguese commented Apr 24, 2025

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025

ahrtr commented Apr 24, 2025

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025 • edited Loading

amosehiguese commented Apr 24, 2025 • edited Loading

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025

amosehiguese commented Apr 24, 2025

ahrtr commented Apr 24, 2025

amosehiguese commented Apr 24, 2025 • edited Loading

amosehiguese commented Apr 23, 2025 •

edited

Loading

amosehiguese Apr 23, 2025 •

edited

Loading

amosehiguese Apr 23, 2025 •

edited

Loading

amosehiguese Apr 23, 2025 •

edited

Loading

codecov bot commented Apr 23, 2025 •

edited

Loading

amosehiguese commented Apr 23, 2025 •

edited

Loading

amosehiguese commented Apr 24, 2025 •

edited

Loading

amosehiguese commented Apr 24, 2025 •

edited

Loading

amosehiguese commented Apr 24, 2025 •

edited

Loading