Skip to content

Increase Fidelity of the sbt.testing.Framework Implementation #1107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Increase Fidelity of the sbt.testing.Framework Implementation #1107

wants to merge 1 commit into from

Conversation

dubinsky
Copy link

@dubinsky dubinsky commented Apr 4, 2025 •
edited
Loading

  • when Selectors supplied include only TestSelectors and TestWildcardSelectors, filter properties to run by matching their names against the Selectors;
  • added a test that actually runs ScalaCheck via the SBT Test Interface and demonstrates the (now correct) treatment of the Selectors;
  • test also demonstrates two unfixable infidelities in the treatment of the nested properties;
  • thankfully, ScalaCheck's implementation of sbt.testing.Framework is, unlike in some other test frameworks, shared between the platforms (JVM, Scala.js, Scala Native), so the fixes do not have to be replicated for each platform, but:
  • the test needs to supply a testClassLoader: ClassLoader parameter when calling sbt.testing.Framework.runner(); on platforms other than JVM, getClass.getClassLoader is not available, so Platform.getClassLoader: ClassLoader method was added to every Platform; on platforms other than JVM, it returns null - which is fine, since on those platforms sbt.testing.Framework.runner() ignores the testClassLoader parameter anyway.

fixes #1105

This was referenced Mar 16, 2025
Closed
Open
@dubinsky dubinsky force-pushed the increase-fidelity-of-the-sbt-testing-framework-implementation branch from 05bd678 to 7ce3e5f Compare April 4, 2025 20:24
@dubinsky dubinsky force-pushed the increase-fidelity-of-the-sbt-testing-framework-implementation branch from 7ce3e5f to da9cb92 Compare April 28, 2025 05:04
@dubinsky
Copy link
Author

@satorg any thoughts?

@dubinsky dubinsky force-pushed the increase-fidelity-of-the-sbt-testing-framework-implementation branch from da9cb92 to 4a468a9 Compare April 28, 2025 08:44
@dubinsky
Copy link
Author

@satorg I am starting to suspect that the plan here is to let this fix wither on the wine like the previous contribution in this area (#1031); if this is true, and my contribution is for some reason not wanted, please say so; otherwise - what do I need to do to get this merged?

Thank you!

@satorg
Copy link
Contributor

satorg commented Jun 15, 2025

@dubinsky thank you for the PR and sorry for the long reply!

Personally, I'd love to get both PRs (this and #1031) merged ultimately. I just don't feel very confident about this ScalaCheckFramework implementation, because I personally don't use it and mostly rely on other test framework runners.

Do you think your PR complements #1031 or supersedes it completely?

@dubinsky
- when Selectors supplied include only TestSelectors and `TestWil…
2e33740 
…dcardSelector`s, filter properties to run by matching their names against the `Selector`s;

- added a test that actually runs ScalaCheck via the `SBT Test Interface` and demonstrates the (now correct) treatment of the `Selector`s;
- test also demonstrates two unfixable infidelities in the treatment of the nested properties;
- thankfully, ScalaCheck's implementation of `sbt.testing.Framework` is, unlike in some other test frameworks, shared between the platforms (JVM, Scala.js, Scala Native), so the fixes do not have to be replicated for each platform, but:
- the test needs to supply a `testClassLoader: ClassLoader` parameter when calling `sbt.testing.Framework.runner()`; on platforms other than JVM, `getClass.getClassLoader` is not available, so `Platform.getClassLoader: ClassLoader` method was added to every `Platform`; on platforms other than the JVM, it returns `null`, which is fine since on those platforms `sbt.testing.Framework.runner()` ignores the `testClassLoader` parameter anyway.

fixes typelevel#1105
@dubinsky dubinsky force-pushed the increase-fidelity-of-the-sbt-testing-framework-implementation branch from 4a468a9 to 2e33740 Compare June 16, 2025 05:00
@dubinsky
Copy link
Author

Personally, I'd love to get both PRs (this and #1031) merged ultimately. I just don't feel very confident about this ScalaCheckFramework implementation, because I personally don't use it and mostly rely on other test framework runners.

Code changes have comments, tests are added that verify the change, and issue #1105 has detailed explanations. Is there anything else I can do? Are you going to involve other contributors/maintainers that are more familiar with this area or do you have specific questions?

Do you think your PR complements #1031 or supersedes it completely?

My pull request completely supersedes #1031: it handles both TestSelector and TestWildcardSelector, whereas #1031 handles just TestSelector.

Thank you!

@dubinsky dubinsky mentioned this pull request Jul 27, 2025
Open
@dubinsky dubinsky mentioned this pull request Aug 5, 2025
Open
satorg
satorg reviewed Aug 12, 2025
View reviewed changes
core/shared/src/main/scala/org/scalacheck/ScalaCheckFramework.scala
Comment on lines +102 to +107
// For TestSelector, exact comparison with both the full name and the name with
// the suite prefix (properties.name) stripped off works for non-nested suite,
// but results in a false negative for nested suites: test "A.B.test" can not be
// selected by its short name "test";
// instead, selected test name is matched using `endsWith`.
case s: TestSelector => name.endsWith(s.testName())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dubinsky , just to clarify: for nested test suites, shouldn't NestedTestSelector be used instead?

Copy link
Author

@dubinsky dubinsky Aug 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I understand, for selecting the tests to run, only TestSelector and TestWildcardSelector are used; NestedTestSelector should be used for reporting status of a test that is nested. As I mentioned further down, inability to correctly report nested tests is one of the unfixables in ScalaCheck; at least it can be fixed to run the correct tests ;)

@satorg
Copy link
Contributor

satorg commented Aug 12, 2025

@dubinsky , thank you for your work on this issue. I found some time to wrap my head around SBT testing API, so that now I think understand what is going on there to some extent. If you are still interested, I guess we can get your PR merged.

However, I wouldn't like to dismiss the effort made in #1031 too, since it was kinda preceding yours. How about if I merge the former PR first, then you'll update yours overwriting conflicting parts and we'll merge your PR right after? Thereby everyone will get involved. @Duhemm, in a case you are still around, let us know if you are OK with this plan please. Thanks everyone!

@satorg satorg mentioned this pull request Aug 12, 2025
Open
@dubinsky
Copy link
Author

@dubinsky , thank you for your work on this issue.

My pleasure!

If you are still interested, I guess we can get your PR merged.

Yes.

However, I wouldn't like to dismiss the effort made in #1031 too, since it was kinda preceding yours. How about if I merge the former PR first, then you'll update yours overwriting conflicting parts and we'll merge your PR right after?

Of course! Thank you!

@satorg
Copy link
Contributor

satorg commented Aug 12, 2025

@dubinsky , do you have any clue what serializeTask/deserializeTask methods of ScalaCheckRunner could be for? They look like they are supposed to be called via reflection, but I didn't manage to figure out when and how exactly.

@dubinsky
Copy link
Author

@dubinsky , do you have any clue what serializeTask/deserializeTask methods of ScalaCheckRunner could be for? They look like they are supposed to be called via reflection, but I didn't manage to figure out when and how exactly.

They are called by the Test Interface/Test Bridge on Scala.js and Scala Native to transport the task information from JVM to Node.js/Native code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@satorg satorg satorg left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

sbt ScalaCheckRunner: loss of test selection fidelity
2 participants
@dubinsky @satorg