fix: implement pagination for workflow jobs fetching#188
Merged
Conversation
- Replace direct API call with octokit.paginate() to fetch all jobs - Add warning for workflows with >100 jobs about memory usage - Update test mocks to include paginate method - Fixes issue where only first 100 jobs were fetched from large workflows
- Add bugfix requirements document outlining pagination limitation and expected behavior - Add design document with fault conditions, preservation requirements, and implementation approach - Add tasks document with implementation steps and test requirements - Add Kiro spec configuration for workflow-jobs-pagination-fix bugfix - Update github.test.ts with pagination-related test cases - Ensures complete job fetching for workflows exceeding 100 jobs limit
- Remove warning log requirements (2.5, 2.6) from bugfix specification - Remove warning property validation from design specification - Replace warning implementation with code comment documenting risk acceptance - Update tasks to add explanatory comment instead of warning logic - Document OOM error implications and future considerations in code - Update test coverage to reflect simplified implementation (97.52%)
- Create target workflow with 200 matrix jobs to test pagination - Create collector workflow to validate telemetry collection - Workflows trigger on push to test pagination fix - Collector verifies at least 200 job spans are captured
- Remove trailing whitespace from test-pagination-200-jobs-collector.yml - Reformat task descriptions in tasks.md to improve readability and maintain consistent line length - Improve documentation formatting for better maintainability
paper2
force-pushed
the
fix/workflow-jobs-pagination
branch
from
608639e to
68c800d
Compare
…eters - Delete redundant test-pagination-200-jobs-collector.yml workflow - Rename test-pagination-200-jobs-target.yml to test-with-many-jobs.yml for clarity - Simplify trigger conditions to push events without branch restrictions - Add verification job that calls validate-action-output.yml reusable workflow - Extend validate-action-output.yml with new input parameters for selective validation - Add validate-traces, validate-metrics, validate-job-count, and expected-job-count inputs - Update job naming and step structure for better readability - Streamline workflow to use reusable workflow pattern instead of separate collector workflow
- Add `set -euo pipefail` to all shell script blocks for strict error handling - Add `-r` flag to jq command in job count validation for raw output - Normalize whitespace and line breaks for consistency across validation steps - Improve script robustness by ensuring pipelines fail on any error and undefined variables are caught
… docs - Remove trailing whitespace from validate-action-output.yml workflow file - Add clarification to README about OpenTelemetry environment variable support - Document that OTEL_ variables are automatically read from job steps - Reference OpenTelemetry SDK environment variable specification for extended configuration options - Improve documentation clarity for users configuring the action
paper2
force-pushed
the
fix/workflow-jobs-pagination
branch
from
860da06 to
c1a83f6
Compare
paper2
force-pushed
the
fix/workflow-jobs-pagination
branch
from
0a5764e to
22ae585
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix workflow jobs pagination to support 100+ jobs
#185
Problem
The action was only fetching the first 100 jobs from workflows due to missing pagination support in the GitHub API call. Workflows with more than 100 jobs would have incomplete telemetry data, causing metrics and traces to be inaccurate.
Solution
octokit.paginate()to fetch all workflow jobs without artificial limitsChanges
Core Fix:
src/github/github.ts: Replaced single API call with paginated fetching usingoctokit.paginate()Testing Infrastructure:
.github/workflows/test-many-jobs.yml: New test workflow with 200 jobs to validate pagination.github/workflows/validate-action-output.ymlwith:validate-traces,validate-metrics)set -euo pipefailTest Organization:
push-event-tests.yml→test-push-event.ymlworkflow-run-tests.yml→test-workflow-run-validation.ymlworkflow-run-tests-target-workflow.yml→test-workflow-run-event.ymlDocumentation:
.kiro/specs/workflow-jobs-pagination-fix/Testing