Prune org.junit.start from exceptions

[mpkorstanje]

When using JUnit.start the pruneStackTrace algorithm immediately sees the HelloTest.main frame and assumes that this is the test method because the test class name matches.

org.opentest4j.AssertionFailedError: expected: <11> but was: <12>
	at [email protected]/org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:158)
	at [email protected]/org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:139)
	at [email protected]/org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:201)
	at [email protected]/org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:152)
	at [email protected]/org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:147)
	at [email protected]/org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:558)
	at com.examp.project/com.example.project.HelloTest.stringLength(HelloTest.java:14)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
...
	at [email protected]/org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:81)
	at [email protected]/org.junit.start.JUnit.run(JUnit.java:63)
	at [email protected]/org.junit.start.JUnit.run(JUnit.java:37)
	at com.examp.project/com.example.project.HelloTest.main(HelloTest.java:9)

By checking if org.junit.start is involved further down the stack we exclude this scenario.

---

I hereby agree to the terms of the JUnit Contributor License Agreement.

---

Definition of Done

• There are no TODOs left in the code
• Method preconditions are checked and documented in the method's Javadoc
• Coding conventions (e.g. for logging) have been followed
• Change is covered by automated tests including corner cases, errors, and exception handling
• Public API has Javadoc and @API annotations
• Change is documented in the User Guide and Release Notes

[sormuras]

Copied from junit-team/junit-examples#656 (comment)

Here, I'd like to keep some detailed information about where the control flow was and where the exception was generated. Hiding too much details hinders to understand which part of the software was in control.

What does the stacktrace shown in the initial description look like when this pruning is applied?

[mpkorstanje]

Right now it would look like identical to a tests that was launched without JUnit.start, e.g. the Console Launcher:

org.opentest4j.AssertionFailedError: expected: <11> but was: <12>
	at [email protected]/org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:158)
	at [email protected]/org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:139)
	at [email protected]/org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:201)
	at [email protected]/org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:152)
	at [email protected]/org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:147)
	at [email protected]/org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:558)
	at com.examp.project/com.example.project.HelloTest.stringLength(HelloTest.java:14)

And after #5159 that would then reduce to:

org.opentest4j.AssertionFailedError: expected: <11> but was: <12>
	at com.examp.project/com.example.project.HelloTest.stringLength(HelloTest.java:14)

[mpkorstanje]

Here, I'd like to keep some detailed information about where the control flow was and where the exception was generated. Hiding too much details hinders to understand which part of the software was in control.

Are you thinking of something like this:

org.opentest4j.AssertionFailedError: expected: <11> but was: <12>
	at com.examp.project/com.example.project.HelloTest.stringLength(HelloTest.java:14)
	at com.examp.project/com.example.project.HelloTest.main(HelloTest.java:9)

If so, why so for JUnit.start but not for other users of the JUnit Platform?

[sormuras]

Are you thinking of something like this:

org.opentest4j.AssertionFailedError: expected: <11> but was: <12>
	at com.examp.project/com.example.project.HelloTest.stringLength(HelloTest.java:14)
	at com.examp.project/com.example.project.HelloTest.main(HelloTest.java:9)

Yes, something like that. Perhaps with 1-2 additional frames from Platform and Jupiter between the two. And at least the initial call to Jupiter's Assertions API.

If so, why so for JUnit.start but not for other users of the JUnit Platform?

Because the I want to keep the trace of the flow of control consistent and easy to follow. It should feel the similar to as if the user wrote a local test method calling a local assertion method. An exception there would be full of HelloTest-based frames, as you would expect from a minimal/starter program.

Delegating to JUnit.run() in a test program within its main() method is for new to Java/JUnit users already a "leap of faith" that somehow the @Test-annotated methods are called. So, if an exception is thrown, the paths taken by the framework should not be totally hidden.

[marcphilipp]

Team decision: If possible, let's keep the following stack frames:

org.opentest4j.AssertionFailedError: expected: <11> but was: <12>
	at [email protected]/org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:558)
	at com.examp.project/com.example.project.HelloTest.stringLength(HelloTest.java:14)
	at [email protected]/org.junit.start.JUnit.run(JUnit.java:37)
	at com.examp.project/com.example.project.HelloTest.main(HelloTest.java:9)