When service isn't shut down gracefully and therefore not deregistered from consul, when the new instance is spawned on the same host:port, Consul doesn't know which instance is actually alive. So all zombie instances send pings to the same healthy single instance and since they get 200, Consul thinks all instances are alive.
I tried to attach ServiceID either as query param or header (since Consul supports custom headers), and in the ping endpoint checked if the received ID matches the actual service ID.
If it matches, it returns 200, otherwise 404, and it works. Consul receives 200 only from the single alive instance.
So my proposal is to add either query param or some header with Convey's ServiceId to health checks.
When service isn't shut down gracefully and therefore not deregistered from consul, when the new instance is spawned on the same
host:port, Consul doesn't know which instance is actually alive. So all zombie instances send pings to the same healthy single instance and since they get200, Consul thinks all instances are alive.I tried to attach
ServiceIDeither as query param or header (since Consul supports custom headers), and in the ping endpoint checked if the received ID matches the actual service ID.If it matches, it returns 200, otherwise 404, and it works. Consul receives 200 only from the single alive instance.
So my proposal is to add either query param or some header with Convey's
ServiceIdto health checks.