Mutually exclusive test cases/groups

[zlondrej]

My aim is to parallelize tests as much as possible, but I'm also bit restricted by shared resources. I would like a functionality that allows me to mark any test case/group with an exclusivity tag. Any test with an exclusivity tag will not run in parallel with another test with same exclusivity tag, but can run in parallel with other tests that have different or no exclusivity tag.

I can achieve a resource lock by using MVar () or similar synchronization mechanism at the start of the test but this will not prevent tests to be scheduled in parallel and sequentialize, while blocking other tests that could be run in parallel in the meantime.

It's a bit similar to sequentialTestGroup, but there are no dependencies (so more like inOrderTestGroup introduced by #448) and no specific execution order. The main difference to aforementioned functions is that this focuses on scheduling tests with shared resources, rather than their order.

Is there a place for such functionality in this framework, or is this something that should be implemented outside? I was considering if it could be built on top of withResource, but I'm not sure because the resource is shared in the entire sub-tree:

IO a is an action which returns the acquired resource. Despite it being an IO action, the resource it returns will be acquired only once and shared across all the tests in the tree.

Plus I have a feeling that it would just block at test scheduler or test execution level if the withResource was applied on top of TestCase by folding TestTree.

What would be the best way to address such case?

[Bodigrim]

Tasty's approach to test scheduling is implicit and is not suited well for this kind of fine-grained control.

It could probably work if we extend data Status = ... | Suspended and data Outcome = ... | TryAgainLater, but I'm not too keen to explore such approach, because it is a breaking change for limited gains.

#392 is related, although would require a user to provide a function capable to sort test list in the order, which keeps usage of shared resources saturated.

Other than that, you can write your own TestManager to implement whatever execution strategy you want. But that's quite challenging.

[zlondrej]

I don't necessarily need to change the order of tests or maximize the shared resource usage. We have quite large test suite that takes few minutes to run and only maybe 20% needs the exclusive access to certain resources.

I think functionality is really bit similar to what after does. After thinking for some time about it, it could probably be implemented in non-breaking manner by injecting After dependencies into test tree.

E.g.

[ exclusively "my-resource-X" $ testGroup "exclusive group X"
    [ testCase "A"
    , testCase "B"
    , testCase "C"
    ]
, exclusively "my-resource-Y" $ testGroup "exclusive group Y"
    [ testCase "D"
    , testCase "E"
    , testCase "F"
    ]
, testGroup "non-exclusive group Z"
    [ testCase "G"
    , testCase "H"
    , testCase "I"
    ]
]

Could become something like:

[ TestGroup "exclusive group X"
    [ SingleTest "A" _
    , After AllFinish "exclusive group X.A" $ SingleTest "B" _
    , After AllFinish "exclusive group X.B" $ SingleTest "C" _
    ]
, TestGroup "exclusive group Y"
    [ SingleTest "D" _
    , After AllFinish "exclusive group Y.D" $ SingleTest "E" _
    , After AllFinish "exclusive group Y.E" $ SingleTest "F" _
    ]
, TestGroup "non-exclusive group Z"
    [ SingleTest "G" _
    , SingleTest "H" _
    , SingleTest "I" _
    ]
]

And that should pretty much work without any additional tracking, because After should essentially serialize specific tests. One caveat is that the After tagging has to be done after any filtering, because it needs to go through final test tree, otherwise it wouldn't tag test cases correctly.

I'm also not sure that After doesn't do any blocking either. There are definitively some TVars involved in resolveDeps. But it also seems that it actually doesn't schedule the test until it reports ActionReady in findBool so dependencies seem to be the key here.

---

I'd be happy to write my own TestManager and release is as separate tasty package if possible, but I'm bit worried whether it would be actually possible with currently exposed API, or how other managers could interfere with it. So at the moment, I'm just digging for ideas/hints from someone with deeper knowledge of tasty.

[Bodigrim]

After thinking for some time about it, it could probably be implemented in non-breaking manner by injecting After dependencies into test tree.

I think this could work out. If you can confirm that injecting After dependencies is enough, we can discuss how to do it automatically.