Skip to content

PIPELINE_SETUP.md

Latest commit

 

History

History
418 lines (321 loc) · 9.76 KB

PIPELINE_SETUP.md

File metadata and controls

418 lines (321 loc) · 9.76 KB
Raw

Pipeline Setup Summary

This document summarizes the complete CI/CD pipeline and testing infrastructure for the llms.txt Generator GitHub Action.

Overview

The project now has a comprehensive pipeline for testing, building, bundling, and releasing the action to GitHub.

Pipeline Components

1. CI Workflow (.github/workflows/ci.yml)

Triggers: Push to main/develop, Pull requests

Jobs:

  • Test: Runs unit tests

    • Checkout code
    • Setup Node.js 20
    • Install dependencies
    • Run unit tests
    • Build TypeScript

  • Integration Test: Tests complete workflow

    • Creates test fixtures dynamically
    • Runs action against fixtures
    • Validates generated files
    • Checks file structure and content
    • Uploads test artifacts

  • Bundle: Creates distribution bundle

    • Builds and bundles action
    • Checks bundle size
    • Uploads bundle artifact

  • Lint: Code quality checks

    • TypeScript compilation check
    • No syntax errors

2. Release Workflow (.github/workflows/release.yml)

Triggers: Push of version tags (v*)

Process:

  1. Checkout code with full history
  2. Setup Node.js 20
  3. Install dependencies
  4. Run full test suite
  5. Build and bundle action
  6. Create release archive
  7. Generate changelog from git commits
  8. Create GitHub release with assets
  9. Update major version tag (e.g., v1 → v1.2.0)

Outputs:

  • GitHub release with changelog
  • Release archive (.tar.gz)
  • Updated major version tag

3. Test Scenarios Workflow (.github/workflows/test-scenarios.yml)

Triggers: Manual dispatch, Weekly schedule

Scenarios:

  • Basic Usage: Simple markdown files
  • Complex Structure: Multiple directories and sections
  • Special Characters: Filenames with special chars
  • Exclude Patterns: File filtering
  • Empty Files: Edge case handling
  • Different Extensions: .md, .mdx, .markdown

Each scenario:

  • Creates realistic test fixtures
  • Runs the action
  • Validates output
  • Checks for expected content

4. Dependabot (.github/dependabot.yml)

Updates:

  • npm dependencies (weekly)
  • GitHub Actions versions (weekly)

Configuration:

  • Auto-labels PRs
  • Conventional commit messages
  • Limits open PRs

Testing Infrastructure

Unit Tests

Location: src*.test.ts

Framework: Vitest

Coverage:

  • File scanning
  • Markdown parsing
  • Content generation
  • URL formatting
  • Section categorization

Run: npm test

Integration Tests

Location: tests/integration/

Script: test-runner.sh

Tests:

  1. Basic functionality
  2. Multiple sections
  3. Exclude patterns
  4. Error handling

Run: npm run test:integration

Test Fixtures

Integration tests create temporary fixtures:

  • Realistic markdown files
  • Various directory structures
  • Edge cases (empty files, special chars)
  • Files to be excluded

Documentation

User Documentation

  • README.md: Project overview and quick start
  • QUICKSTART.md: 5-minute setup guide
  • src/README.md: Complete action documentation

Developer Documentation

  • CONTRIBUTING.md: Contribution guidelines
  • TESTING.md: Comprehensive testing guide
  • PUBLISHING.md: Release process
  • RELEASE_CHECKLIST.md: Pre-release checklist
  • PIPELINE_SETUP.md: This document

Templates

  • Bug Report: .github/ISSUE_TEMPLATE/bug_report.md
  • Feature Request: .github/ISSUE_TEMPLATE/feature_request.md
  • Pull Request: .github/PULL_REQUEST_TEMPLATE.md

Release Process

Automated Release

  1. Update version:

    cd src
npm version patch  # or minor, major

  2. Build and bundle:

    npm run package

  3. Commit bundle:

    git add dist/
git commit -m "chore: update bundle for release"

  4. Push with tags:

    git push origin main --tags

  5. Automated steps:

    • CI runs all tests
    • Release workflow triggers
    • GitHub release created
    • Changelog generated
    • Major version tag updated

Version Tags

Users can reference:

  • Specific: @v1.0.0 (pinned)
  • Major: @v1 (latest v1.x.x)
  • Branch: @main (not recommended)

Package Scripts

{
  "build": "tsc",
  "bundle": "ncc build srcmain.ts -o dist",
  "package": "npm run build && npm run bundle",
  "test": "vitest --run",
  "test:integration": "bash tests/integration/test-runner.sh",
  "test:all": "npm test && npm run test:integration"
}

Directory Structure

.
├── .github/
│   ├── actions/
│   │   └── llms-txt-generator/
│   │       ├── src                    # TypeScript source
│   │       │   ├── main.ts
│   │       │   ├── file-scanner.ts
│   │       │   ├── parser.ts
│   │       │   ├── generator.ts
│   │       │   ├── types.ts
│   │       │   └── *.test.ts           # Unit tests
│   │       ├── tests/
│   │       │   └── integration/        # Integration tests
│   │       ├── dist/                   # Bundled output
│   │       ├── action.yml              # Action metadata
│   │       ├── package.json
│   │       └── README.md
│   ├── workflows/
│   │   ├── ci.yml                      # Main CI pipeline
│   │   ├── release.yml                 # Release automation
│   │   └── test-scenarios.yml          # Comprehensive tests
│   ├── dependabot.yml                  # Dependency updates
│   ├── ISSUE_TEMPLATE/
│   │   ├── bug_report.md
│   │   └── feature_request.md
│   └── PULL_REQUEST_TEMPLATE.md
├── sample/                             # Example data (not used in CI)
├── CONTRIBUTING.md
├── PUBLISHING.md
├── QUICKSTART.md
├── TESTING.md
├── RELEASE_CHECKLIST.md
├── PIPELINE_SETUP.md
└── README.md

CI/CD Flow Diagram

┌─────────────────┐
│  Push to main   │
│  or PR created  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   CI Workflow   │
├─────────────────┤
│ • Unit tests    │
│ • Integration   │
│ • Build         │
│ • Bundle        │
│ • Lint          │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Tests pass?    │
└────┬───────┬────┘
     │       │
    Yes      No
     │       │
     │       └──► ❌ Fail PR
     │
     ▼
┌─────────────────┐
│  Merge to main  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Create tag     │
│  (npm version)  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Release Workflow│
├─────────────────┤
│ • Run tests     │
│ • Build/bundle  │
│ • Create release│
│ • Update v1 tag │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ ✅ Published!   │
└─────────────────┘

Quality Gates

Before Merge

  • ✅ All unit tests pass
  • ✅ All integration tests pass
  • ✅ TypeScript compiles
  • ✅ Bundle builds successfully
  • ✅ Code review approved

Before Release

  • ✅ All tests pass
  • ✅ Documentation updated
  • ✅ CHANGELOG updated
  • ✅ Version bumped
  • ✅ Bundle committed

Monitoring

After Release

  • Check GitHub Actions runs
  • Monitor for issues
  • Review error reports
  • Respond to user feedback

Metrics to Track

  • Test pass rate
  • Bundle size
  • Build time
  • Action usage
  • Issue response time

Maintenance

Regular Tasks

  • Weekly: Review Dependabot PRs
  • Monthly: Review and update documentation
  • Quarterly: Security audit
  • As needed: Bug fixes and features

Dependency Updates

Dependabot automatically:

  • Creates PRs for updates
  • Runs tests on updates
  • Labels appropriately

Review and merge when tests pass.

Troubleshooting

CI Fails

  1. Check workflow logs
  2. Run tests locally
  3. Verify Node.js version matches
  4. Check for environment differences

Release Fails

  1. Verify tag format (v*)
  2. Check permissions
  3. Review release workflow logs
  4. Ensure bundle is committed

Integration Tests Fail

  1. Check test fixtures
  2. Verify action inputs
  3. Review generated output
  4. Check for path issues

Next Steps

For New Contributors

  1. Read CONTRIBUTING.md
  2. Setup development environment
  3. Run tests locally
  4. Make changes
  5. Submit PR

For Maintainers

  1. Review RELEASE_CHECKLIST.md
  2. Follow release process
  3. Monitor after release
  4. Respond to issues

For Users

  1. Read QUICKSTART.md
  2. Add workflow to repository
  3. Customize as needed
  4. Report issues if found

Resources

Support

Pipeline Status: ✅ Fully Configured

Last Updated: [Date]

Maintained By: [Your Team]