Create a mypy-primer like workflow in CI for Sphinx

[AA-Turner]

Is your feature request related to a problem? Please describe.

When e.g. adding new warnings, it is good to know what the impact is likely to be. In the static typing ecosystem, @hauntsaninja's mypy-primer is used automatically in CI to report on many popular open source projects (including Sphinx!). I think we should explore doing something similar for Sphinx, as an aide to pre-release testing and to give us confidence when adding new warnings. For example, I don't currently have a good estimate for #13400 and if we could immediately make it a warning (cc @jayaddison).

Describe the solution you'd like

I don't know how much / if there is anything we could take from mypy-primer as scaffolding here. @hauntsaninja may be able to advise (hope you don't mind the ping!). Ideally we'd have something in CI that we could run, either nightly / on request / on every PR. We'd need to capture the environment setup for building docs for each chosen project. Probably isolated environments? We'd then need to choose a set of initial OSS projects to test against.

Describe alternatives you've considered

Doing nothing.

Additional context

• 7.3.0 release plan #12242 (comment) (cc @picnixz)

[hauntsaninja]

I don't know that there is too much to take directly from mypy_primer.

The number one thing to steal is the Github Actions set up. This is a little fiddly to do correctly and securely. See:

• https://github.com/python/mypy/blob/master/.github/workflows/mypy_primer.yml
• https://github.com/python/mypy/blob/master/.github/workflows/mypy_primer_comment.yml

You could also borrow some elements of the diffing logic, though you could probably do something simpler like

d = difflib.Differ()
old_lines = self.old_result.output.splitlines()
new_lines = self.new_result.output.splitlines()
diff = d.compare(old_lines, new_lines)
return "\n".join(line for line in diff if line[0] in ("+", "-"))

The main file in mypy_primer that could be interesting to skim through is https://github.com/hauntsaninja/mypy_primer/blob/master/mypy_primer/model.py

[picnixz]

One thing I can tell is that using difflib.unified_diff may be slow when there is a lot of output (depending on what you're matching, e.g., matching the debug messages as well) and it's more efficient to use the diff command directly. I encountered a diff in my current job that was 80 MB and difflib was very slow. I do not know if using Differ.compare would be more efficient though but using diff -u2 --suppress-common-lines --speed-large-files solved my issue.

We need to know:

• the set of projects we're matching against,
• what we'll be matching against: do we want to match the debug messages?
• which extensions to use: all? a combination of many? etc

[hauntsaninja]

(difflib isn't the fastest thing ever, but it's fine for mypy_primer. I've had luck using https://pypi.org/project/cdifflib/ in the past too)