Conversation
…-regression-testing
|
ok this is ready to review now the race condition is sorted @bstancil |
bstancil
left a comment
There was a problem hiding this comment.
Ok, I'm into this conceptually, but are there easy ways to bypass it (or ways to make it so that we elect into running it?) These things can take some time to run, so blocking every PR on it feels very heavy?
Y'all have a lot more experience / opinions about this than I do, so i'll take your judgement on it. But if the test suite grows, this could become vey cumbersome.
|
At the moment the time between "PR open" and "PR approved" is normally multiple hours, and the suite finishes in ~10 minutes, so I think we've got a fair bit of headroom before it becomes the limiting factor. It's also skipping markdown changes etc so quick cleanups shouldn't be an issue. You can imagine a Slim CI setup which only runs modified tests, I couldn't be bothered faffing with that right now but the good naming conventions should make that reasonably approachable when the time comes. Enabling auto-merge also means that a PR can be human reviewed and approved pending successful regression tests, without having to remember to come back and do it later. So I think we should be ok for the foreseeable! |
bstancil
left a comment
There was a problem hiding this comment.
aight, let's do it. and if we need to change it, that's easy enough to do when we want to.
Make sure that changes to a task, shared project, etc, don't cause an experiment to fail when run against the sage. And, check that all experiments (except
analytics_engineering001which is designed to test a no-op) fail when run with no agentRunning each project type with a single concurrent run to avoid #76's race conditions, but this can be changed once that’s solved