fix(es): add delay in TestPasswordFromFile to prevent flakiness

[Br1an67]

Fixes #4743

Summary

The TestPasswordFromFile test was flaky because the Elasticsearch client has background health check goroutines that may continue running briefly after Close() is called. When the password changes and a new client is created, the old client is closed but its health check goroutine might still make requests with the old password for a brief moment after the client swap.

Changes

• Added a 100ms delay after the client changes in the test to allow the old client's background goroutines to fully shut down before proceeding with the new password verification

Testing

• Ran the test 5 times in a row without any failures:

  go test -v -run TestPasswordFromFile ./internal/storage/v1/elasticsearch/... -count=5
  

• All 5 runs passed successfully

Root Cause

The olivere/elastic client library's Stop() method signals background goroutines to stop but does not wait for them to complete. This can cause a race condition where:

1. Password file changes
2. New client is created with new password
3. Old client is swapped out and Close() is called
4. Test writes span2 and verifies new password
5. Old client's health check goroutine (still running) makes a request with old password

The delay gives the old client's background operations time to fully terminate before the test proceeds.

[Copilot]

Pull request overview

Addresses flakiness in TestPasswordFromFile caused by background Elasticsearch client goroutines continuing briefly after Close() during password rotation, leading to occasional requests with the old credentials.

Changes:

• Adds a short delay after detecting the client swap in runPasswordFromFileTest to reduce race-related flakiness.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Using a fixed time.Sleep here makes the test slower and can still be flaky on slow/loaded CI (the healthcheck goroutine may take longer than 100ms to stop). Since this test only validates auth headers on write requests, consider disabling background health checks in the test configuration (e.g., cfg.DisableHealthCheck = true) or replacing the sleep with a deterministic wait condition tied to observed requests (e.g., wait until no further requests with the old password are seen for a short window).

Suggested change

	// Give the old client time to fully shut down its background goroutines
	// (e.g., health checks) after being closed in onClientPasswordChange.
	// This prevents flakiness where the old client might still make requests
	// with the old password after the client has been swapped.
	time.Sleep(100 * time.Millisecond)

[yurishkuro]

timeouts do not improve flakiness, the solution needs to be deterministic.