Add configuration option for test executor arguments #2636
Conversation
freakboy3742
requested review from
joerick and
mhsmith
and removed request for
mhsmith
| ### `test-execution-args` {: #test-execution-args toml env-var } | ||
|
|
||
| > Define additional arguments that will be passed to the command that runs the tests. |
There was a problem hiding this comment.
test-execution-args still sounds like they might be arguments to the tests themselves. Would test-runner-args be a clearer name?
There was a problem hiding this comment.
I picked test-execution-args based on the discussion that lead to this PR; but test-runner-args would also work. The exact naming isn't a hill I'll die on, and it's easy to change; would be interested in hearing other opinions.
| # has problems using any simulator that isn't pre-warmed. | ||
| # Unfortunately, the simulator that it picks by default isn't | ||
| # pre-warmed. So - we need to explicitly select a simulator. | ||
| test_execution_args: '--simulator "iPhone 16e,OS=18.5"' |
There was a problem hiding this comment.
Could we make this the default, at least when running on GitHub Actions, so each project doesn't need to specify it? Experience so far has shown that every required bit of manual configuration will trip up a significant number of package developers, even if the documentation is clear enough.
There was a problem hiding this comment.
The problem is that the "working default" can (and based on recent evidence, does) change somewhat arbitrarily. If a value is hard coded into cibuildwheel, then it needs a cibuildwheel release to fix it.
Even the fact that an override is needed at all is a temporary measure - once GitHub/Apple resolve this hypervisor performance issue, there presumably won't be a need for an override.
There was a problem hiding this comment.
An alternative idea – though I guess this might need to be done on the CPython side – if we need to use a simulator which is warmed up, can we detect which simulators are warmed up and use them by default? That would give better performance even after the current issue is fixed,
Again, this could be a CI-specific behavior, but there is precedent for doing that to work around issues which are likely to keep happening, such as the Android emulator permissions.
docs/platforms.md
Outdated
| The iOS test environment can't support running shell scripts, so the [`test-command`](options.md#test-command) value must be specified as if it were a command line being passed to `python -m ...`. | ||
|
|
||
| The test process uses the same testbed used by CPython itself to run the CPython test suite. It is an Xcode project that has been configured to have a single Xcode "XCUnit" test - the result of which reports the success or failure of running `python -m <test-command>`. | ||
| Any arguments specified using [`test-execution-args`](options.md#test-execution-args) will be passed as arguments to the Python script that starts the testbed project. The most common argument to use will be `--simulator`, which allows the specification of a specific device or iOS version for the test simulator. For example, `test_execution_args = ["--simulator", "iPhone 16e,OS=18.5"]` would specify the use of an iPhone 16e simulator running OS 18.5. |
There was a problem hiding this comment.
This should state what the default arguments are on both Android and iOS.
There was a problem hiding this comment.
It should also say what the default emulator/simulator is, so the user can judge whether an override is necessary.
Co-authored-by: Malcolm Smith <[email protected]>
…the default args to the test runner.
| automatically be skipped. | ||
|
|
||
| Any arguments specified using [`test-execution-args`](options.md#test-execution-args) will be passed as arguments to the Python script that starts the testbed project. The most common arguments to use will be `--managed minVersion` or `--managed maxVersion`, specifying the use of a managed Android emulator with the minimum or maximum supported Android version; or `--connected <emulator ID>`, specifying the use of an existing booted Android emulator or device. | ||
| Any arguments specified using [`test-execution-args`](options.md#test-execution-args) will be passed as arguments to the Python script that starts the testbed project. cibuildwheel will automatically start the testbed project with `--site-packages` and `--cwd` arguments matching your test environment, as well as enabling verbose output with `-v` if [`build_verbosity`](options.md#build_verbosity) is enabled. The most common additional arguments to use will be `--managed minVersion` or `--managed maxVersion`, specifying the use of a managed Android emulator with the minimum or maximum supported Android version; or `--connected <serial>`, specifying the use of an existing booted Android emulator or device. |
There was a problem hiding this comment.
If we're going into this level of detail, I guess we should link to the script documentation so people can find out what options are available, and similarly for iOS.
2 participants
iOS and Android test suites are executed using a testbed app. This app needs to be started, and the test arguments (the value of
test_command) is passed to that testbed.However, the testbed app itself has configuration arguments, and it can sometimes be useful to be able to use these arguments.
This issue has become mildly critical as a result of the recent changes to the
macos-15test runner. At present, the simulator that is picked by default by the iOS test runner is entirely valid from the perspective of Xcode, but then refuses to launch.This is a known issue with the macOS-15 runner that GitHub is investigating; but it's not a once-off issue. When Xcode 16.4 was released, it removed the "iPhone SE" simulator as a default image, and as there was no way to configure the simulator used at runtime, a full update of the iOS support packages and a cibuildwheel release was necessary to resolve the issue.
So - this PR adds a
CIBW_TEST_EXECUTION_ARGSsetting (with related platform-specific alternatives). These arguments are passed to the testbed runner on iOS and Android.As Android requires explicitly specifying the simulator to use as part of the runner arguments, the value of
CIBW_TEST_EXECUTION_ARGSis lightly processed. If there's a--managedor--connectedvalue inCIBW_TEST_EXECUTION_ARGS, that value will be used by the testbed runner; if no such argument exists,--managed maxVersionwill be used by the test runner.I'm not entirely sure how best to test this; it's in a portion of the code that can't be easily tested without actually starting simulators, and... well... the problem that led to this PR is that we can't use other simulators :-) So - I've gone with testing this empirically through the Github Actions configuration, passing in the necessary arguments so that on GitHub, simulators are specified on iOS and Android x86 simulator. On other platforms, the default behavior will continue to operate.