feat: Adding e2e hook tests - #990
Closed
pen-pal wants to merge 12 commits into
Closed
Conversation
pen-pal
requested review from
MaxymVlasov,
antonbabenko and
yermulnik
as code owners
May 28, 2026 08:26
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds end-to-end test infrastructure: a bash test runner, README, GitHub Actions workflow, and two Terraform-focused e2e cases with inputs, expected outputs, and requirements files. ChangesEnd-to-End Test Infrastructure
Sequence Diagram(s)sequenceDiagram
participant Workflow
participant RunScript as tests/e2e/run_e2e_tests.sh
participant PreCommit as pre-commit
participant Git as git diff
Workflow->>RunScript: run tests in container
RunScript->>PreCommit: pre-commit run --all-files
PreCommit-->>RunScript: exit code and file changes
RunScript->>Git: compare produced tree to expected/
Git-->>RunScript: diff result
RunScript-->>Workflow: PASS / FAIL summary
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/e2e-tests.yaml:
- Around line 31-33: The workflow uses env.IMAGE set to
ghcr.io/antonbabenko/pre-commit-terraform:latest which allows image drift;
change the IMAGE value to a digest-pinned reference
(ghcr.io/antonbabenko/pre-commit-terraform@sha256:<digest>) so the e2e runner
always pulls an immutable artifact, updating any references to IMAGE used in the
workflow (the env block and any docker pull/docker run invocations); obtain the
exact sha256 digest for the desired image tag via a docker registry query or
docker pull + docker inspect (or GHCR API) and replace :latest with
`@sha256`:<digest>.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0f0b0b52-8698-431c-922f-5536c499b88a
📒 Files selected for processing (18)
.github/workflows/e2e-tests.yamltests/e2e/README.mdtests/e2e/cases/terraform_fmt/reformats-misaligned-hcl/.pre-commit-config.yamltests/e2e/cases/terraform_fmt/reformats-misaligned-hcl/expected/main.tftests/e2e/cases/terraform_fmt/reformats-misaligned-hcl/expected_returncodetests/e2e/cases/terraform_fmt/reformats-misaligned-hcl/input/main.tftests/e2e/cases/terraform_fmt/reformats-misaligned-hcl/requirestests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/.pre-commit-config.yamltests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/main.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/versions.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/wrappers/README.mdtests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/wrappers/main.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/wrappers/outputs.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/wrappers/variables.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/expected/wrappers/versions.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/input/main.tftests/e2e/cases/terraform_wrapper_module_for_each/generates-wrapper-for-root-module/input/versions.tftests/e2e/run_e2e_tests.sh
yermulnik
reviewed
May 28, 2026
yermulnik
left a comment
yermulnik
left a comment
Collaborator
There was a problem hiding this comment.
Thank you for the contribution.
I'm out of context around tests and therefore I've added solely tech comments around Bash code.
pen-pal
added a commit
to pen-pal/pre-commit-terraform
that referenced
this pull request
May 29, 2026
Per review on antonbabenko#990: - replace the `RETURN` trap with explicit cleanup at the function's single exit (the `RETURN` trap is Bash-only and confused reviewers); - read the expected return code with `$(< file)` instead of `$(cat file)`; - quote the `true`/`false` flag values to avoid confusion with the builtins; - use `&>` instead of `> ... 2>&1`; - drop the redundant trailing slash on the copy destination; - print the summary with `printf` instead of a loop. No behavior change; the suite still passes natively and inside the image. Signed-off-by: penpal <unameme@proton.me>
pen-pal
added a commit
to pen-pal/pre-commit-terraform
that referenced
this pull request
May 29, 2026
Per review on antonbabenko#990, pin `ghcr.io/antonbabenko/pre-commit-terraform` by `sha256` digest instead of the floating `:latest` tag, so the bundled toolchain can't drift between runs. A Renovate annotation keeps the digest updated. Signed-off-by: penpal <unameme@proton.me>
yermulnik
reviewed
May 29, 2026
yermulnik
left a comment
yermulnik
left a comment
Collaborator
There was a problem hiding this comment.
@MaxymVlasov Shell code looks good to me, though I'm not really good with the rest of the tests logic — could you please have a look when you've got a moment? Thank you.
Introduce `tests/e2e/run_e2e_tests.sh`, a behavioral test runner that exercises each hook the way a user would -- through `pre-commit` against fixture files -- and compares the result against a committed golden tree. It auto-discovers `cases/<hook>/<case>/`, builds a temp git repo from `input/`, renders the case's `.pre-commit-config.yaml` (`__PCT_REPO__` -> checkout root), runs the hook via `pre-commit run --all-files`, then asserts the exit code matches `expected_returncode` (default 0) and the working tree is byte-identical to `expected/`. The tree compare uses `git diff --no-index` so it works with the BusyBox `diff` shipped in the project image. A `requires` file skips a case when a needed CLI tool is absent. Part of antonbabenko#823. Signed-off-by: penpal <unameme@proton.me>
Add `generates-wrapper-for-root-module`: a root module (with no
`provider_meta`) is wrapped, and the generated
`wrappers/{main,outputs,variables,versions}.tf` plus `README.md` are
checked against the golden tree. Exits 0 (only new files are created).
Part of antonbabenko#823.
Signed-off-by: penpal <unameme@proton.me>
Add `reformats-misaligned-hcl`: misaligned HCL is rewritten to canonical style. The hook modifies a tracked file, so `pre-commit` exits 1 (`expected_returncode=1`). The case requires `terraform`. Part of antonbabenko#823. Signed-off-by: penpal <unameme@proton.me>
Add `.github/workflows/e2e-tests.yaml`. Checkout runs on the host (the JS action needs Node), then the suite runs inside the published multi-tool image (`ghcr.io/antonbabenko/pre-commit-terraform:latest`), which supplies terraform/hcledit/pre-commit while the hook and test code come from the checkout -- so the image tag need not be in sync. Part of antonbabenko#823. Signed-off-by: penpal <unameme@proton.me>
Add `tests/e2e/README.md` describing the case layout, how to run the suite natively and inside the project image, and how to add a new case. Part of antonbabenko#823. Signed-off-by: penpal <unameme@proton.me>
Per review on antonbabenko#990: - replace the `RETURN` trap with explicit cleanup at the function's single exit (the `RETURN` trap is Bash-only and confused reviewers); - read the expected return code with `$(< file)` instead of `$(cat file)`; - quote the `true`/`false` flag values to avoid confusion with the builtins; - use `&>` instead of `> ... 2>&1`; - drop the redundant trailing slash on the copy destination; - print the summary with `printf` instead of a loop. No behavior change; the suite still passes natively and inside the image. Signed-off-by: penpal <unameme@proton.me>
Per review on antonbabenko#990, pin `ghcr.io/antonbabenko/pre-commit-terraform` by `sha256` digest instead of the floating `:latest` tag, so the bundled toolchain can't drift between runs. A Renovate annotation keeps the digest updated. Signed-off-by: penpal <unameme@proton.me>
Signed-off-by: penpal <unameme@proton.me>
pen-pal
force-pushed
the
feat/823-e2e-hook-tests
branch
from
May 30, 2026 05:10
235410e to
a6052ac
Compare
MaxymVlasov
reviewed
Jun 1, 2026
MaxymVlasov
left a comment
MaxymVlasov
left a comment
Collaborator
There was a problem hiding this comment.
Thank you for introducing this simple but powerful e2e test strategy!
I have a few requests for you before we proceed.
| env: | ||
| # renovate: datasource=docker depName=ghcr.io/antonbabenko/pre-commit-terraform | ||
| # yamllint disable-line rule:line-length | ||
| IMAGE: ghcr.io/antonbabenko/pre-commit-terraform:latest@sha256:4ef4b8323b27fc263535ad88c9d2f20488fcb3b520258e5e7f0553ed5f6692b5 |
Collaborator
There was a problem hiding this comment.
If we pin it to hash, then there's no sense to use the latest at all, which contradicts the main idea - test againt latest tool set & against the latest stable hook version.
In other words, we need 2 tests: against latest w/o pinning and against the latest available hook version, which can be bumped by renovate, but I'd like to have it as part of the release process.
Comment on lines
+71
to
+72
| docker build -t pct:e2e --build-arg INSTALL_ALL=true . | ||
| docker run --rm -v "$PWD:/lint" -w /lint --entrypoint bash pct:e2e \ |
Collaborator
There was a problem hiding this comment.
And what's the reason to build a separate image?
Just use a exisitng one https://github.com/antonbabenko/pre-commit-terraform#1-install-dependencies
Collaborator
There was a problem hiding this comment.
Let's add a section about requesting the addition of e2e hooks to https://github.com/antonbabenko/pre-commit-terraform/blob/master/.github/CONTRIBUTING.md#add-new-hook
|
This PR has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this PR will be closed in 10 days |
github-actions
Bot
added
the
stale
Co-authored-by: Maksym Vlasov <MaxymVlasov@users.noreply.github.com>
Co-authored-by: Maksym Vlasov <MaxymVlasov@users.noreply.github.com>
Co-authored-by: Maksym Vlasov <MaxymVlasov@users.noreply.github.com>
Contributor
Author
|
will apply the fixes based on comments |
github-actions
Bot
removed
the
stale
4 tasks
|
This PR has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this PR will be closed in 10 days |
github-actions
Bot
added
the
stale
|
This PR was automatically closed because of stale in 10 days |
MaxymVlasov
added a commit
that referenced
this pull request
Aug 18, 2026
…est (#1004) ### Description of your changes #### What <sub>This section was generated by AI.</sub> - Add `tests/pytest/tool_version_test.py`: black-box pytest suite for `--hook-config=--tool-version=` / `--tool-version-mode=` / `--tf-path` resolution added on `downloadable_and_version_controlled_hooks`. Tests invoke real `hooks/*.sh` scripts as subprocesses and assert only on exit code / cache filesystem state / output - never on bash-internal function names - so they need minimal changes if a hook is ever rewritten in another language. - **Assertions prove which binary actually ran.** Each stub echoes a unique marker; every network-free test asserts that marker plus the absence of the losing candidate's. `--tool-version-mode=strict` in particular logs its NOTE *before* choosing a binary. - Hooks now run against a coreutils-only `PATH` (no wrapped CLI tool) and an allowlisted environment (`PRE_COMMIT_COLOR=never`, `LC_ALL=C`, no inherited `PCT_TFPATH`/`TERRAGRUNT_TFPATH`/`TF_*`), so a host install or an exported variable cannot change what a test resolves. The temp repo is created with `git init --template=` and committed with `--no-verify`, because this project's own README tells users to point `init.templateDir` at a directory with pre-commit installed. - Coverage: cache-hit reuse, `strict` vs `prefer-local`, the `--tf-path=terraform|opentofu|tofu` selector and its invalid-value error, `--tf-path` auto-detect **including Terraform winning when both binaries are on `$PATH`**, `--tf-path` literal passthrough when `--tool-version` is *not* set, the `XDG_CACHE_HOME` and `$HOME/.cache` cache roots, the documented `checkov` no-op, the actionable "tool not found" error, the `tool_name` wired by **each of the 14 resolving hooks**, and one real end-to-end download on a cache miss. - `fix(hooks)`: resolve a pinned tool **once** per run. `common::per_dir_hook` now takes the already-resolved `tool_path` instead of re-resolving it, so hooks that need it before the per-dir loop (`tflint --init`; the terragrunt hooks choosing `hcl format` vs `hclfmt`) no longer trigger a duplicate NOTE log and, on a cache miss, a duplicate download. - `fix(hooks)`: reject an unrecognized `--tool-version-mode` instead of treating it as `strict`, so a typo like `prefer_local` no longer silently does the opposite of what was asked. Documented in README. - Package `hooks/`, `lib_getopt` and `tools/install/` into the sdist (`hatch.toml`), since the CI `tests` job runs pytest against a built sdist and needs the hook scripts this suite subprocesses into. - Fix bash 3.2 (macOS default) `${var^^}` usage in `hooks/_common.sh` / `tools/install/_common.sh` (portable `tr` pipeline) and an OpenTofu release-asset version-regex bug. - Wire `GITHUB_TOKEN` into the tox test job so the real-download test authenticates its `api.github.com` call instead of sharing the unauthenticated per-IP rate limit across the whole CI matrix. - Mark the download test `network` (registered in `pytest.ini`, which `--strict-markers` requires) so `-m 'not network'` runs the suite offline. - Flat test functions rather than test classes, matching this repo's existing `_cli_test.py` / `terraform_docs_replace_test.py` convention; parametrized cases are bundled into `NamedTuple`s to satisfy `PLR0913`/`WPS211` without suppressions. `.flake8` ignores `WPS226` (string-literal overuse, same precedent as `_cli_test.py`) plus `WPS202`/`WPS204`, which a wide table-driven suite trips by construction. Windows-skip coverage pragmas satisfy the `covdefaults` 100% coverage gate. - Merge `stderr` into `stdout` in `_run_hook`, because `common::colorify` writes every diagnostic to stderr while a wrapped tool's own output goes to stdout - one ready-to-grep string per caller. #### Why > [!IMPORTANT] > Somehow related/conflicts with #990. So far - I can't say what will be chosen, waiting for python-guru's review Main idea - make some black-box testing for bash hooks on pytest, which then can be applied on hooks reimplementation on Python, when/if it time comes Or if that's a shity approach - then just make some black-box testing, not in pytest ### How can we test changes <sub>This section was generated by AI.</sub> - Offline: `pytest tests/pytest/ -m 'not network'` - 43 passed, 1 deselected. - Full suite: `pytest tests/pytest/` - 44 passed, coverage 100.00% (the `covdefaults` gate). Network test alone: `pytest -m network` - 1 passed. - **Mutation-tested.** Each of these, applied on its own, turns the suite red: strict mode falling back to `$PATH`; inverting the `--tf-path` auto-detect precedence; dropping the `XDG_CACHE_HOME` fallback; a `terraform-docs` -> `terraform_docs` `tool_name` typo; `--tf-path` passthrough hijacked by the pinning code; `prefer-local` ignored; the `opentofu` -> `tofu` binary rename forgotten; the `--tool-version-mode` validation removed; and re-introducing the duplicate resolution inside `common::per_dir_hook`. - Each new commit was checked out into a separate `git worktree` and its test suite run there, to confirm no intermediate commit is broken. - This PR targets `downloadable_and_version_controlled_hooks` (not `master`) so this repo's GH Actions matrix runs the suite for real, including the network-dependent download test. - All pre-commit hooks pass locally: `shellcheck`, `shfmt`, `ruff check`, `ruff format`, `wemake-python-styleguide`, `mypy` (py3.10/3.12/3.14).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
Increment toward #823: a behavioral end-to-end test harness for the hooks. It runs each hook the way a user would - through
pre-commitagainst fixture files — and compares the result against a committed "golden" output tree.Addresses #823 (lays the foundation and seeds it with two hooks). It does not close the issue, since most hooks still need cases.
Harness -
tests/e2e/run_e2e_tests.shtests/e2e/cases/<hook_id>/<case_name>/. Adding a hook is just adding a dir — no runner changes.input/, renders the case's.pre-commit-config.yaml(__PCT_REPO__→ checkout root), runs the hook viapre-commit run --all-files.expected_returncode(default0) and the working tree is byte-identical toexpected/.requiresfile skips a case when a needed CLI tool is absent.Seed cases
terraform_fmtreformats-misaligned-hclpre-commitexits1(hook modifies a tracked file)terraform_wrapper_module_for_eachgenerates-wrapper-for-root-modulewrappers/{main,outputs,variables,versions}.tf+README.mdgenerated for aCI -
.github/workflows/e2e-tests.yamlRuns the suite on every PR inside the published multi-tool image (
ghcr.io/antonbabenko/pre-commit-terraform:latest). The image supplies the tooling (terraform,hcledit, …); the hook and test code come from the checkout, so the image tag need not be in sync.How can we test changes
Locally (needs
pre-commit,git, and the hooks' tools —terraform,hcledit):bash tests/e2e/run_e2e_tests.sh # => 2 passed, 0 failed, 0 skippedInside the project image (matches CI):