Support test retry filtering

[NGloreous]

Description

Add an extensibility point to VSTest to enable custom logic to determine if a failed test should be retried.

The OneDrive team currently uses a custom built test harness to run our tests. To deal with test flakyness we need to retry tests and we have built up custom logic to determine when it's acceptable to retry tests and how many times a test can retry. We would like to migrate to VSTest but we need this support to do it. We're open to contributing.

The extension should have the following context provided:

• Fully qualified test name
• The test assembly
• The failure Exception
• Ideally the full test context (to have access settings/parameters)

The extension should be able to control

• If the test gets retried (also possibly be able to control when it gets retried, e.g. right away vs at the end of other tests)
• What to do with the failed results, e.g. drop it, change the outcome.

The retry filter extension to use should be able to be specified via command line and/or run settings.

[acesiddhu]

@NGloreous our recommendation is to use translation layer. The logic which determines what needs to be re run etc needs to be the guy which is invoking translation layer. We follow a same mechanism in Visual Studio Test Task today.
We shouldnt add this in vstest because thats not what vstest.console.exe is supposed to do. It's main work is to run test and provide apis to client (TEstExplorer, VsTEst task, custom clients like yours) to enable any kind of work flow these clients want to execute. We shouldnt overload vstest in anyways.

[cltshivash]

@NGloreous Listing the possible options below.

-> Leveraging a custom attribute seems to be the best route here. Is it possible to revisit the option of updating the test code ?
-> Using translation layer. But that would mean authoring a client and not leveraging the out of the box vstest task in the pipeline.
-> Having a custom adapter. But this will be a lot of work and maintenance.
-> Exposing the new extension point. This is going to be a lot of changes and not something worth going after.

[NGloreous]

@cltshivash for a new extension point, how much work are we talking? We are willing to do this work and feel it would be worth doing.

We'd like to avoid the custom test attribute and don't feel it gives us the flexibility we want.

[cltshivash]

@NGloreous Can you add more information on the flexibility part ?

All the current execution extension is driven through attributes and a new extension not declared on the code will need a place to be declared (probably run settings ?) and second it needs to support all the attribute driven extensions which are supported today.

[NGloreous]

@cltshivash the flexibility to onboard new code without having to change it all. Also we would like the flexibility to delay retries until the end of the test run vs inline retries immediately. I don't think attributes would allow us to do this.

[NGloreous]

@cltshivash another thing is we don't necessarily want to do an "inline" retry (e.g. just call the test method a second time if the test fails). We've found doing a retry like this doesn't help much for our tests. What does help is starting "fresh" by re-creating the test class object or by retrying in a new process (this is needed because often time failures are caused by tests "leaking" state).

I don't think using custom attributes would support this. Can this even be accomplished from within testfx? I would imagine if we want to retry in a new process this logic would need to go into vstest?

[kiragengis]

Hi @NGloreous I am doing this in my team. I am running all the tests calling the VSTest.Console for each test. I read the results and I decided when to re-run the test and how many times the test should re-run. As you said, would be great to have this capability here.

[brumlemann]

The simplest of solutions, from a user perspective, would be a switch on vstest.console /RerunAttempts:{number} with a default value of 0. Seems to me this would be the simplest solution from an implementation perspective too.

Gather up all the failing tests and rerun, of course observing all other switches but tweaking [test file names] and /Tests:[test name].

In particular /Parallell is important so only the test container(s) that contain failing tests are rerun in separate runs, rescheduled immediately at the back of the queue when that particular container run has finished with failed tests.

This would be most efficient eliminating the wait for a complete run before rescheduling failing tests. On big runs with many test containers there is always a trailing part were the parallelism is weak, simply because not all test container are equally weighted. So in this phase typically cores/threads are available but not exploited.

This rather simple first approximation would therefore have the benefit of exploiting "dead time" at the end of a big /Parallell run filling in the blanks re-running tests that failed, basically on free time.

And I agree, starting up a separate process is the most desirable way to do this because it catches a lot of failing issues in the simplest way possible.

Then you could extend by adding attributes to control single tests/classes, but that wouldn't be my first priority.

[Evangelink]

Moving this issue over to vstest.