benchmark/client: add context for cancellation

[wdsjk]

Fixes: #8596

Added cancellation by context in benchmark/client instead of custom channel approach. Not sure about returning ctx.Err() (as mentioned in the original issue under point 4), so please review

RELEASE NOTES: None

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: wdsjk / name: Ivan Mamaev (08c45bc, 123e0d4, 13f3600, 7161064, 798974a, 87d80e9, b1db5d5, c92ee83, cf5b7ef, e09a7f5, ea7453f, f3c53df)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.19%. Comparing base (ab1ca08) to head (e09a7f5).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8614      +/-   ##
==========================================
+ Coverage   82.05%   82.19%   +0.13%     
==========================================
  Files         417      417              
  Lines       40793    40793              
==========================================
+ Hits        33473    33530      +57     
+ Misses       5963     5919      -44     
+ Partials     1357     1344      -13     

see 30 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wdsjk]

Could you please add "Type: Internal Cleanup" label? Thanks

[arjan-bal]

Apologies for the delayed review and thanks for the contribution!

A few main points:

1. We should pass the context though function params.
2. We should try to derive contexts from other contexts as much as possible instead of using context.Background.
3. We can avoid a lot of the ctx.Err() checks by letting RPCs fail and handling the case when the error had a status of CANCELLED.

[wdsjk]

Thank you very much for the review, it is very helpful!

[wdsjk]

Everything is ready, review please @arjan-bal

[arjan-bal]

LGTM, thanks for the contribution! Adding a second reviewer. I'll also run the benchmarks once to verify there aren't any regressions.

[arjan-bal]

There's a significant drop in unary performance from 625k qps to 351k qps compared to the master branch. I'm investigating further.

Investigating performance regression in unary benchmark performance.

[arjan-bal]

There's a significant drop in unary performance from 625k qps to 351k qps compared to the master branch. I'm investigating further.

Using the cancellable context to make RPCs seems to be the reason for the performance drop. After reverting to using context.Background() for making RPCs (see 1d2714b), the performance was back to initial levels.

[arjan-bal]

Left some comments to avoid the performance cost of using a cancellable context for RPCs. The main changes are:

• We should continue to propagate the context until the point of making a stream or unary call.
• We should check for context expiry before making the unary call or stream request. In most cases this means replacing the following

select {
	case <-bc.stop:
		return
	default:
}

with

if ctx.Err() != nil {
        return
}

• Continue to use context.Background() for requests as before.

[wdsjk]

Review please @arjan-bal

[wdsjk]

Sorry, misclicked

[arjan-bal]

LGTM, adding a second reviewer.