Skip to content

Unexpected Unknown grpc code for ErrNotPrimary #21671

@ants

Description

@ants

Bug report criteria

What happened?

Due to a performance hiccup there was a leader switch, which apparently caused /v3/lease/keepalive requests return an error: {"code": 2, "message": "not a primary lessor"}. Everything is as expected, except the Unknown error code.

What did you expect to happen?

Some error code that signifies that retry is in order. Unavailable maybe.

How can we reproduce it (as minimally and precisely as possible)?

Have s.ensureLeadership() check fail in EtcdServer.LeaseRenew().

Anything else we need to know?

No response

Etcd version (please run commands below)

3.6.7

Etcd configuration (command line flags or environment variables)

No response

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

No response

Relevant log output

{"level":"warn","ts":"2026-04-21T22:04:25.607018+0200","caller":"etcdserver/raft.go:387","msg":"leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk","to":"95012eee2c61378b","heartbeat-interval":"250ms","expected-duration":"500ms","exceeded-duration":"1.484060419s"}
{"level":"warn","ts":"2026-04-21T22:04:25.607340+0200","caller":"etcdserver/raft.go:387","msg":"leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk","to":"836bb8986617b0d6","heartbeat-interval":"250ms","expected-duration":"500ms","exceeded-duration":"1.484390506s"}
{"level":"warn","ts":"2026-04-21T22:04:25.742549+0200","caller":"etcdserver/v3_server.go:911","msg":"waiting for ReadIndex response took too long, retrying","sent-request-id":10377592809537642783,"retry-timeout":"500ms"}
{"level":"info","ts":"2026-04-21T22:04:26.006035+0200","logger":"raft","caller":"v3@v3.6.0/raft.go:1101","msg":"a77ddc750103e129 [logterm: 216, index: 26712135, vote: a77ddc750103e129] ignored MsgPreVote from 95012eee2c61378b [logterm: 216, index: 26712130] at term 216: lease is not expired (remaining ticks: 1)"}
{"level":"info","ts":"2026-04-21T22:04:26.006102+0200","logger":"raft","caller":"v3@v3.6.0/raft.go:1101","msg":"a77ddc750103e129 [logterm: 216, index: 26712135, vote: a77ddc750103e129] ignored MsgVote from 95012eee2c61378b [logterm: 216, index: 26712130] at term 216: lease is not expired (remaining ticks: 1)"}
{"level":"info","ts":"2026-04-21T22:04:26.006137+0200","logger":"raft","caller":"v3@v3.6.0/raft.go:1116","msg":"a77ddc750103e129 [term: 216] received a MsgHeartbeat message with higher term from 95012eee2c61378b [term: 217]"}
{"level":"info","ts":"2026-04-21T22:04:26.006207+0200","logger":"raft","caller":"v3@v3.6.0/raft.go:897","msg":"a77ddc750103e129 became follower at term 217"}
{"level":"info","ts":"2026-04-21T22:04:26.006238+0200","logger":"raft","caller":"v3@v3.6.0/node.go:372","msg":"raft.node: a77ddc750103e129 changed leader from a77ddc750103e129 to 95012eee2c61378b at term 217"}
{"level":"warn","ts":"2026-04-21T22:04:26.006316+0200","caller":"etcdserver/server.go:939","msg":"Failed to check current member's leadership","error":"etcdserver: leader changed"}
{"level":"info","ts":"2026-04-21T22:04:26.008988+0200","logger":"raft","caller":"v3@v3.6.0/log.go:158","msg":"found conflict at index 26712131 [existing term: 216, conflicting term: 217]"}
{"level":"info","ts":"2026-04-21T22:04:26.009030+0200","logger":"raft","caller":"v3@v3.6.0/log_unstable.go:203","msg":"replace the unstable entries from index 26712131"}
{"level":"info","ts":"2026-04-21T22:04:26.019329+0200","caller":"etcdserver/v3_server.go:903","msg":"first commit in current term: resending ReadIndex request"}
{"level":"warn","ts":"2026-04-21T22:04:26.039277+0200","caller":"etcdserver/server.go:946","msg":"Current member isn't a leader","local-member-id":12069044971509113129,"new-lead":10736914587109635979}
{"level":"warn","ts":"2026-04-21T22:04:26.039397+0200","caller":"etcdserver/server.go:946","msg":"Current member isn't a leader","local-member-id":12069044971509113129,"new-lead":10736914587109635979}
...

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions