unit test flake: panic: grpc: the server has been stopped

[BlaineEXE]

Bug Report

What happened:

There is a fairly frequent unit test flake showing panic: grpc: the server has been stopped. This occurs in sidecar tests.

I haven't observed this in local testing. Only in the prow CI. Presumably due to Prow CI running in resource-constrained virtual environments that have occasional slowness.

I think this is happening when the unit test's gRPC server starts slower than tests run but am not sure. It is a race condition. We should find out what will allow the tests to wait until the gRPC server is running before proceeding.

What you expected to happen:

Unit tests should be stable.

How to reproduce this bug (as minimally and precisely as possible):

Hard to repro. Run prow CI multiple times to observe. I have not observed it on my local machine, presumably because it is faster.

Anything else relevant for this bug report?:

Resources and logs to submit:

Copy all relevant COSI resources here in yaml format:

# BucketClass
# BucketAccessClass
# BucketClaim
# Bucket
# BucketAccess

# Copy COSI controller pod logs here

# Copy COSI sidecar logs here for the relevant driver

Environment:

• Kubernetes version (use kubectl version), please list client and server:
• Sidecar version (provide the release tag or commit hash):
• Provisoner name and version (provide the release tag or commit hash):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others: