Description
When using WatchAsync with authentication enabled, if the server returns an invalid auth token error (e.g., due to token expiration or a server-side session flush), the watcher stops receiving events but fails to notify the client.
Specifically:
The Action callback is not invoked with response.Canceled == true.
The Task returned by WatchAsync does not throw an exception or complete in a faulted state.
The internal gRPC stream appears to be disposed or canceled silently, leaving the application in a "zombie" state where it believes it is watching but never receives further updates.
This forces developers to implement hacky workarounds (like using reflection to access private _channel and _authToken fields) to maintain a reliable connection.
Environment
- dotnet-etcd version: 8.1
- .NET version: .NET 8.0
- OS: almalinux 8
- etcd version: 3.6.7
Steps to Reproduce
Initialize an EtcdClient with valid credentials and enable Auth on the etcd server.
Start a WatchAsync on a specific prefix.
Wait for the authToken to expire, or manually revoke the user's session/token on the etcd server.
Modify a key under the watched prefix.
Observed Behavior:
The modification event is not received.
No error is logged by the SDK.
The application remains unaware that the watch stream has been terminated by the server.
Expected Behavior
The SDK should either:
Trigger the Action callback with Canceled = true and provide the CancelReason.
Or, the WatchAsync task should throw an RpcException (Unauthenticated) to allow for a standard try-catch reconnection logic.
Evidence: Connection Loss Logs
During my testing, the underlying gRPC connection frequently encountered the following error when the token became invalid or the connection was jittery, yet the WatchAsync task remained in a "hanging" state without triggering any callback:
Watch stream connection lost: Unavailable - Status(StatusCode="Unavailable", Detail="Error reading next message. HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)", DebugException="System.Net.Http.HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)")
This log indicates that the HttpClient (HTTP/2) detected a premature response end, but this critical RpcException was swallowed or not propagated to the Action callback or the WatchAsync Task.
Suggested Fix
The internal Watch implementation should properly propagate gRPC stream status changes. When the underlying ResponseStream.ReadAllAsync() or equivalent iterator encounters an error or a Canceled flag from the server, it must ensure the high-level API is notified.
Current workaround involves manually creating a WatchClient from the private _channel and injecting the token into Metadata manually, which proves the underlying gRPC flow works if handled correctly.
Description
When using WatchAsync with authentication enabled, if the server returns an invalid auth token error (e.g., due to token expiration or a server-side session flush), the watcher stops receiving events but fails to notify the client.
Specifically:
The Action callback is not invoked with response.Canceled == true.
The Task returned by WatchAsync does not throw an exception or complete in a faulted state.
The internal gRPC stream appears to be disposed or canceled silently, leaving the application in a "zombie" state where it believes it is watching but never receives further updates.
This forces developers to implement hacky workarounds (like using reflection to access private _channel and _authToken fields) to maintain a reliable connection.
Environment
Steps to Reproduce
Initialize an EtcdClient with valid credentials and enable Auth on the etcd server.
Start a WatchAsync on a specific prefix.
Wait for the authToken to expire, or manually revoke the user's session/token on the etcd server.
Modify a key under the watched prefix.
Observed Behavior:
The modification event is not received.
No error is logged by the SDK.
The application remains unaware that the watch stream has been terminated by the server.
Expected Behavior
The SDK should either:
Trigger the Action callback with Canceled = true and provide the CancelReason.
Or, the WatchAsync task should throw an RpcException (Unauthenticated) to allow for a standard try-catch reconnection logic.
Evidence: Connection Loss Logs
During my testing, the underlying gRPC connection frequently encountered the following error when the token became invalid or the connection was jittery, yet the WatchAsync task remained in a "hanging" state without triggering any callback:
Watch stream connection lost: Unavailable - Status(StatusCode="Unavailable", Detail="Error reading next message. HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)", DebugException="System.Net.Http.HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)")This log indicates that the HttpClient (HTTP/2) detected a premature response end, but this critical RpcException was swallowed or not propagated to the Action callback or the WatchAsync Task.
Suggested Fix
The internal Watch implementation should properly propagate gRPC stream status changes. When the underlying ResponseStream.ReadAllAsync() or equivalent iterator encounters an error or a Canceled flag from the server, it must ensure the high-level API is notified.
Current workaround involves manually creating a WatchClient from the private _channel and injecting the token into Metadata manually, which proves the underlying gRPC flow works if handled correctly.