Is there a possibility to run Verify checks in CI without file creation? Memorize received output and check with existing verified file contents?

[blazkovicz]

I did not find any correlating documentation on the matter. Please help!

We use C#, MSTest, Verify.MSTest

[SimonCropp]

sorry. i don’t understand what u r asking for

[blazkovicz]

hi! Thanks for quick reaction! I created tests in my net project, using Verify.MSTest nuget package. İt creates .received files and checks against .verified files, all good. When we run tests in jenkins Cİ we don’t want any file creation there, just want actual snapshot contents to be checked on-the-fly against existing .verified file contents and to either succeed or fail with error. How to achieve that?

[SimonCropp]

enable AutoVerify on the build server:

    public static class ModuleInitializer
    {
        [ModuleInitializer]
        public static void Init()
        {
            if (BuildServerDetector.Detected)
            {
                VerifierSettings.AutoVerify();
            }
        }
    }

then u can run a script to push any changes back. here is how u woud do it in a github action

    - name: Push changes
      run: |
        git config --local user.email "action@github.com"
        git config --local user.name "GitHub Action"
        git commit -m "changes" -a || echo "nothing to commit"
        remote="https://${GITHUB_ACTOR}:${{secrets.GITHUB_TOKEN}}@github.com/${GITHUB_REPOSITORY}.git"
        branch="${GITHUB_REF:11}"
        git push "${remote}" ${branch} || echo "nothing to push"
      shell: bash

[blazkovicz]

This autoverify does the opposite we want
In some scenarios it makes sense to auto-accept any changes as part of a given test run.
we don't want changes to be auto-accepted in CI run, we want changes to fail the test and no file creation or edit attempt.

and we don't want anything to be pushed anywhere, of course

[blazkovicz]

İs it possible to skip this writing with some env variable os special setup in the code and just get the mismatch description in the console error?

[SimonCropp]

not currently.

it is helpful for people to promote those received files to the build output. so u can then download them and compare them locally

so r u saying you are not allowed to write files in your build process?

[SimonCropp]

also note that the result of a test using verify can be any file format. eg images are common. in those scenarios writing to the console would not work

[SimonCropp]

here are file promotion docs https://github.com/VerifyTests/Verify/blob/main/docs/build-server.md#getting-received-files-from-ci

[blazkovicz]

Okay for non text files console error can just state then, that files had mismatched. We want to be able to run test tool in very strict environment with minimal file system permissions. Basically, run in CI should be green always, if its red - we rerun locally and solve problems. We don’t want mismatch error to cause other type of errors, related not to tests but environmental setup, which will cause longer time to understand the root of the problem.

[SimonCropp]

happy to consider a pull request that adds this feature