pr05 Visual Regression Testing #1: Initial Visual Testing Framework

[Vaivaswat2244]

Visual Testing Framework for Processing

Summary

This PR introduces a comprehensive visual regression testing framework for Processing, enabling automated detection of visual changes in rendering output across different platforms. This also opens up the scope for further discussions and modifications to the approach.

Implementation Details

Core Components

1. Sketch Runner with Controlled Execution

• Executes Processing sketches in a controlled test environment
• Captures single frame after setup() and one draw() call
• Automatic window cleanup after screenshot capture
• Configurable render wait times for complex sketches

2. Test Runner (VisualTestRunner.java)

• Captures screenshots from Processing sketches
• Manages baseline images with platform-specific naming
• Compares actual output against saved baselines
• Automatically creates baselines on first run

3. JUnit 5 Integration (VisualTest.java)

• Base test class extending JUnit 5
• Provides assertVisualMatch() helper method
• Supports test tagging and suite organization
• Handles first-run baseline creation gracefully

4. Image Comparison Algorithm (ImageComparator.java)

• Implements a sophisticated pixel-matching algorithm
• Detects and filters out minor rendering artifacts (line shifts, anti-aliasing differences)
• Configurable thresholds for pass/fail criteria
• Generates diff images for failed tests showing exact pixel differences

Project Structure

Key Features

1. Hierarchical Screenshot Organization

• Screenshots organized by test suite: __screenshots__/suite-name/test-name-platform.png

2. Test Suite Organization

• Tests organized by feature area (shapes, rendering, text, etc.)
• JUnit 5 @Tag annotations for flexible test selection
• For now, only two suites have been added, i.e., shapes and rendering. These will obviously be extended as tests grow

3. Intelligent Pixel Matching

• Handles minor anti-aliasing differences
• Detects and ignores line-shift artifacts

4. Gradle Tasks

./gradlew :visual-tests:test                    # Run all tests
./gradlew :visual-tests:visualTests             # Run only visual tests
./gradlew :visual-tests:testBasicShapes         # Run specific suite

Usage Example

@Tag("shapes")
public class BasicShapesTest extends VisualTest {
    
    @Test
    @DisplayName("Red circle renders correctly")
    public void testRedCircle() {
        assertVisualMatch("basic-shapes/red-circle", new ProcessingSketch() {
            @Override
            public void setup(PApplet p) {
                p.noStroke();
            }
            
            @Override
            public void draw(PApplet p) {
                p.background(255);
                p.fill(255, 0, 0);
                p.ellipse(p.width/2, p.height/2, 100, 100);
            }
        });
    }
}

Testing Strategy

First Run:

• Test captures screenshot from sketch
• No baseline exists → saves as baseline
• Test marked as "BASELINE CREATED"

Subsequent Runs:

• Test captures screenshot
• Compares against platform-specific baseline
• Reports PASS/FAIL with mismatch details
• Saves diff image if failed

cc @catilac @Stefterv @mingness @SableRaf

[Stefterv]

I added a few comments related to the Gradle setup. I will look into the behaviour and functionality at another stage

[mingness]

@Vaivaswat2244 It seems that the running of the tests is orchestrated via the gradle file, when instead, I think it should be orchestrated by JUnit. I'm not sure how the reporting system might work - maybe it needs gradle for that, but for the tests themselves, there should be a way of structuring them so they are simply the tests, any data needed by them, and they you invoke JUnit to run the tests. In our call tomorrow, let's look closer at this.

[Vaivaswat2244]

Hey @Stefterv and @mingness, I've made the changes and also ensured the tests are orchestrated using JUnit.
Also, I was in the process of porting tests from p5 (currently done for the shapes) and have added those in these commits.

[mingness]

@Vaivaswat2244 just to make the conversation history clean in this PR, I recommend to respond to each of the issues that @Stefterv raised in his previous review with a short comment of what you implemented, or just "DONE", and then mark the issues as resolved. And when you're ready for Stef to rereview, click on "re-request review".

[mingness]

Looks good to me! I only have a few formatting suggestions: There are changes in the root level gradle files that are actually just formatting - I suggest to revert these files. Also in other places where there are substantial formatting only changes (some lines in ./core/build.gradle.kts) I suggest to revert the formatting changes, so that all the changes are functional.

[Stefterv]

Looking good! Thank you @Vaivaswat2244

One thing that I think would be useful would be some way to know if there differences between the different platforms, as a warning or something similar

I also noticed that Intellij gives a link to see the differences between the asserts, if you see a chance to make that the two images that would be awesome!

@catilac is it an idea to also move this to a feature branch for now?

[Stefterv]

Was running it locally on my Mac and it was running in pixeldensity 2 causing the tests to fail

[Vaivaswat2244]

Okay, so I can try forcing the tests to render at a standard pixel-density, another thought is of saving separate baselines for both the pixel-densities though this might be an issue if we see devices with better renderers.

[Stefterv]

I think pixeldensity = 1 fine for now

[Vaivaswat2244]

Hey @Stefterv, Is this resolved now?

[Vaivaswat2244]

Looks good to me! I only have a few formatting suggestions: There are changes in the root level gradle files that are actually just formatting - I suggest to revert these files. Also in other places where there are substantial formatting only changes (some lines in ./core/build.gradle.kts) I suggest to revert the formatting changes, so that all the changes are functional.

Sure, I'll revert the formatting changes.

[Vaivaswat2244]

Looking good! Thank you @Vaivaswat2244

One thing that I think would be useful would be some way to know if there differences between the different platforms, as a warning or something similar

I also noticed that Intellij gives a link to see the differences between the asserts, if you see a chance to make that the two images that would be awesome!

@catilac is it an idea to also move this to a feature branch for now?

Sure, I think the differences between platforms can be identified, we can compare the running sketch to all the baselines which we'll be storing (linux, mac and win), though this can increase the test time significantly, maybe we can do this periodically, using a cron job or something similar.

[catilac]

great question @Stefterv! @Vaivaswat2244 if this project will require multiple phases or multiple PRs, it could be helpful to create a feature branch for it. If it's all self contained in this, then no problem.

And in general, small PRs around certain logical boundaries are nice for digesting changes when it's a large effort.

up to you @Stefterv !

[Vaivaswat2244]

Hey @catilac, Addition to this pr, there should be atleast 3 more prs till the completion.
1- adding more tests
2- CI integration and report generation (p5 version of reporting system)
3- documentation

Changing to feature branch or staying in main, I am open to both the options.

[Stefterv]

@Vaivaswat2244 can you fix the last comment and merge the latest changes from main into your branch?

[Vaivaswat2244]

Hey @Stefterv , I just fixed the last comment, but in order to merge the latest changes from main to my branch, I had to resolve the merge conflicts between the branches. Please take a look and let me know if I've made a git mess...