Repository navigation
Week 4: rigorous testing - fuzz, integration, gap coverage, python coverage gate - #4
Merged
Merged
Conversation
…erage gate Pure test-rigor work. No product surface changes. Lifts Go coverage 88.6% -> 95.0%, Python coverage 0% -> 96.8%, adds three new fuzzers and a live MCP-source-to-ACP integration test. Go fuzzers (1.4M+ executions across 4 fuzzers, zero crashes locally): - internal/sources/mcp/FuzzCompactSchema: random JSON-Schema shapes, asserts every emitted compact form uses only legal mini-language tokens (string/int/float/bool/json/bytes plus '?' and '[]' suffixes, enum:..., ref:...). - internal/sources/mcp/FuzzInferCapabilities: arbitrary tool names + extras, asserts capabilities are sorted, deduped, lowercase, no single-char tokens. - internal/builder/FuzzDependsOnFor: random dependency graphs, asserts no self-cycles in the output, sorted unique IDs, every returned id present in the available map. Live integration test: - tests/integration/mcp_source_to_acp_test.go boots a fake MCP server with verbose JSON-Schema descriptors, imports it via internal/sources/mcp.Importer, builds a manifest through the real ACP server stack, and asserts the manifest payload is strictly smaller than the upstream MCP tools/list body. This is the 'ACP on top of MCP' positioning made executable. - Local run: 622-byte manifest vs 1017-byte tools/list (39% reduction on a 2-tool fixture; production S4 hits 97.4%). Gap coverage: - internal/server/server_extra_test.go covers handleFeedback (bad JSON, GET method, sink error -> 500), CryptoIDSource (1000 unique IDs, no collisions), ListenAndServe (graceful context-cancel + bad-addr error path), unauth response shape (JSON envelope, never plain text). - internal/proxy/gates_test.go covers AlwaysApprove + AlwaysDeny (was 0% before). - pkg/acp/api_error_test.go covers APIError.Error() formatting and IsAPIError() (was 0% before). Python tests (35 -> 61): - adapters/python/tests/test_acp_common_properties.py: random schema atoms, random graph permutations, k-fold diamonds, cycle-injection must always raise CycleError. - adapters/python/tests/test_acp_langgraph_build.py: stubs the langgraph.graph module via sys.modules so build_graph runs the full StateGraph construction path without the framework dep. - benchmark/tests/test_harness_determinism.py: count_tokens purity, zero-MCP-tokens edge case, p95 monotonicity, RunResult JSON wire shape stability. - benchmark/tests/test_harness_main.py: request_acp_manifest + measure_run + main() against a stubbed urlopen, including the HTTPError -> exit 2 path. - benchmark/tests/test_report.py: render() against the real S1+S4 sample numbers, including main() with tmp_path. CI: - pytest now runs with --cov on adapters and benchmark trees. - --cov-fail-under=85 hard gate. - pytest-cov added to the python install step. Validation: - go vet ./...: clean - go test -race -count=1 -timeout 60s ./...: 9 packages green - go coverage on internal+pkg: 95.0% (was 88.6%) - staticcheck ./...: clean (after S1017 cleanup in fuzz test) - govulncheck ./...: zero vulnerabilities - pytest -q --cov-fail-under=85: 61 passed, 96.84% coverage Signed-off-by: Shreyansh Sancheti <43677304+shreyanshjain7174@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pure test-rigor work, no product surface changes. Lifts:
What landed
Go fuzzers
Live integration
`tests/integration/mcp_source_to_acp_test.go` boots a fake MCP server with verbose JSON-Schema descriptors, imports it via `internal/sources/mcp.Importer`, builds a manifest through the real ACP server, and asserts the manifest is strictly smaller than the upstream MCP `tools/list` body. Local run: 622-byte manifest vs 1017-byte tools/list (39% reduction on a 2-tool fixture; production S4 hits 97.4%).
Gap-coverage
Python tests
CI
Validation
```
go vet ./... # clean
go test -race -count=1 -timeout 60s ./... # 9 packages green
go coverage on internal+pkg # 95.0% (was 88.6%)
staticcheck ./... # clean
govulncheck ./... # zero vulnerabilities
pytest --cov-fail-under=85 # 61 passed, 96.84% coverage
```
Verifier
Independent verifier subagent ran before commit: PASS. Cosmetic suggestion to use stdlib helpers in the integration test was applied in the same branch.