Could the default for reportTestSuiteErrors move to being enabled in a future release?

[Niallfitzy1]

We recently tripped over this foot-gun with our testing suites.

It feels like an odd choice to default to ignoring that test suites fail to run, considering it gives the false impression that tests have passed when they've actually failed entirely.

Could a safer default be added in the next major release?

There seems to be a bit of discussion here: #220 but the explanation doesn't seem to justify the downsides considering that users could still opt-out if they're sensitive to these kind of results.

I don't think the default behaviour users expect for a test reporter is to suppress failures of any kind.

For context, we use this reporter with a Github Action to get rich test reporting for our pull requests.

[palmerj3]

Absolutely not.

Here is the main issue and why that is the case.

As you know, you can configure jest-junit to set suiteNameTemplate, className, etc so that your junit file is configured the way you want.

Each of those test suites and test cases now are essentially a unique identifier.

Many people ingest junit files into tooling to keep track of tests over time, so stability of identifiers matter.

So why is this a problem?

Well if you enable reportTestSuiteErrors and a test suite fails to load, we have to default to using file path as the identifier. If you aren't already configuring your test suites to utilize file path then we've effectively just introduced a new unique identifier you may not have expected.

Systems like jenkins have plugins which can keep track of test flakiness so we have to keep default settings such that unique identifiers remain stable.

Hope this helps.

[Niallfitzy1]

Thanks for the response. It answers why the default can't be changed in a bubble, but I don't think it helps anything about how un-intuitive the default behaviour is.

So maybe my question is better phrased as:

Is there anything that can be done so that out of the box we can have reporting of test suite errors?

• Is there an updated default configuration that could enable this?
• Could we report them differently to have some kind of stability when flagging a failure?

Keeping this in mind:

we have to default to using file path as the identifier. If you aren't already configuring your test suites to utilize file path then we've effectively just introduced a new unique identifier

Could the default be to use file path as the default identifier? Would that enable this?

Again, if someone really dislikes being told the suite is completely broken with a new unique identifier, nothing would stop them disabling this behaviour. I'm not advocating for removing the choice. I'm advocating for safe defaults.

[palmerj3]

Couple of things at play here.

A test suite is essentially a describe block in jest. Given that you can have multiple describe blocks within a test file it would be problematic if I defaulted to the the suite name to be simply a file path.

And this is also the crux of why, when a test file fails to be interpreted by jest, I don't by default write a file path empty test suite to the junit file. In previous, successful, runs that junit file would be populated with test suites and test cases that contain whatever strings people wrote for the describe and it blocks.

So that is a nuanced reason why I cannot simply default to writing a test suite failure to the junit file. Unlike java junit, jest does not force uniqueness for test suites (describe blocks) and does not limit you to one suite per file. If it did then what you're suggesting would be a super reasonable default for everyone.

The final bit here I'd like to say is: I would not recommend reading a junit file as the sole mechanism for determining if tests failed. Jest itself sets an appropriate error code when there is an exception and a failure occurs. Not sure if this is what you're doing, but I have seen folks basically pass a github action step if there are no listed failures in a junit file, but ignoring the actual error codes of jest itself and that is not recommended. junit.xml is intended to be a summary of what tests ran along with metadata about the tests / suites.