CHANGELOG.md

Changelog

[Unreleased]

[0.0.6] - 2025-11-14

Security

• Git History: Removed sensitive files from repository history
  • Used git-filter-repo to rewrite history and eliminate sensitive data exposure

Changed

• .gitignore: Enhanced to prevent committing sensitive configuration files
  • Deduplicated entries and organized into logical sections
  • Added patterns to exclude sensitive mapping files: config/test_mapping.json, config/*_mapping.json
  • Added patterns to exclude archive files: config/*.zip
  • Consolidated Python, testing, and environment-specific ignores
  • Ensured config/auth.json, config/*.local.json, and config/private/ are properly excluded

[0.0.5] - 2025-10-16

Fixed

• Logging: Fixed KeyError when logging script arguments in log_script_start() function
  • Changed args parameter to script_args to avoid conflict with Python's LogRecord.args attribute
  • Resolves "Attempt to overwrite 'args' in LogRecord" error when running scripts with logging
  • All scripts now start successfully without logging errors
    • fetch_specs_with_auth.py: Made OAuth configuration platform-agnostic and environment-driven
      • Replaced --tenant-token-url with --oauth-token-url for clarity
      • Added OAUTH_TOKEN_URL environment variable support for any OAuth 2.0 provider
      • Removed Microsoft-specific defaults and invalid URL fallbacks
      • Added validation to ensure OAuth token URL is configured when needed
    • batch_fetch_register_scan.py: Added OAuth token URL support for batch workflows
      • Added --oauth-token-url parameter to pass through to fetch_specs_with_auth.py
      • Updated usage examples to show OAuth environment variable and CLI usage
      • Enables complete batch automation with OAuth-protected APIs
    • Documentation: Updated chaining recipes with new OAuth authentication patterns
      • Added OAuth setup section showing OAUTH_TOKEN_URL environment variable usage
      • Updated config examples to remove deprecated tokenUrl field
      • Added Python batch automation section with OAuth examples
  • Updated integration tests to reflect new parameter names and behavior
• Integration Tests: Fixed 14 failing integration tests to properly validate coded integration contracts
  • fetch_specs_with_auth.py: Added missing --debug argument and updated test expectations to match resilient script behavior
  • register_app_with_host_and_spec.py: Fixed output directory security guard issues in tests
  • All 220 integration tests now passing ✅
• CONTRIBUTING.md: Fixed broken documentation links

[0.0.4] - 2025-10-07

Added

• Explained how receipts system works w example
• Validated 100% compliance with architectural standards
• Documented updates including new chaining recipes
• New troubleshooting guide: docs/Troubleshooting_Git_Pull.md
  • Documents the Pythonic way to resolve uv.lock conflicts during git pull
  • Explains why lock file conflicts are normal and expected
  • Provides best practices for lock file management
  • Includes prevention strategies with pre-commit hooks
• Added git pull troubleshooting reference to docs/QUICK_START.md
• New validation script: scripts/validate_chain.py
  • Validates chaining workflow components without requiring VPN/API access
  • Checks that all required scripts exist and are accessible
  • Validates receipt file structure and JSON format
  • Enables offline validation of fetch → register → scan chain
• New batch automation script: scripts/examples/batch_fetch_register_scan.py
  • Complete Python automation for fetch → register → scan workflow
  • One-command execution with proper error handling and progress tracking
  • Supports importAPI.json config files for multiple APIs
  • Generates audit receipts and machine-readable JSON output
  • Replaces shell scripting with robust Python implementation
• Enhanced Makefile: Added comprehensive targets for all workflows
  • New batch automation targets: batch-automation, fetch-specs
  • Utility targets: validate-chain, list-instances, delete-app, setup-app
  • Organized help system with clear sections and examples
  • Support for CONFIG_FILE variable for batch operations

[0.0.3] - 2025-10-03

Fixed

Critical: fetch_specs_with_auth.py - Complete refactor to follow project patterns

Breaking Changes:

• Output location changed from ./tmp/ to output/ directory
• --debug flag replaced with --verbose flag
• Script now outputs JSON envelope to stdout (was integer exit code)

Eight Critical Fixes Applied:

1. Output Directory Pattern ✅

   • Changed from hardcoded ./tmp/ to proper output/ directory
   • Uses ensure_output_dir() from lib/common.py
   • Respects APISEC_OUTPUT_DIR environment variable
   • Added --output-dir flag for custom locations
   • Added --force flag for security guard bypass

2. Receipt Pattern Implementation ✅

   • Creates timestamped receipts: fetch_specs_with_auth-YYYYMMDDTHHMMSSZ.json
   • Uses write_receipt() from lib/common.py
   • Contains: config file, success/failure counts, fetched spec metadata
   • Enables audit trail and batch processing validation

3. JSON Envelope Output ✅

   • Outputs ok_json() envelope to stdout (machine-parseable)
   • Proper err_json() for all error conditions
   • Includes: successes, failures, total, fetched_specs, output_dir, receipt_file
   • Enables pipeline chaining with jq and other tools

4. Enhanced Error Handling ✅

   • Config file not found → actionable err_json() with file path
   • Invalid JSON → parse error details with suggestions
   • No API entries → validation error with configuration guidance
   • All errors include: code, message, why, suggestion, exit_code

5. Security Logging ✅

   • All API keys sanitized before logging via sanitize_token()
   • Client secrets sanitized in OAuth flows
   • Access tokens sanitized after acquisition
   • Uses log_script_start(), log_script_complete(), log_script_error()
   • Correlation IDs propagated through all operations

6. Integration with Common Library ✅

   • Uses ensure_output_dir() for directory management
   • Uses write_receipt() for audit trails
   • Uses ok_json() and err_json() for structured output
   • Uses get_env() for environment variable handling
   • Follows patterns from register_app_from_spec.py

7. Enhanced Spec Tracking ✅

   • Tracks metadata: index, URL, auth_type, output_file, name
   • Logs file size for each saved spec
   • Consistent JSON formatting with indent=2
   • All data available in receipt and final output envelope

8. Improved Configuration Loading ✅

   • Robust load_config() with correlation_id tracking
   • FileNotFoundError → structured error with actionable message
   • JSONDecodeError → parse error details with suggestions
   • Type validation → specific guidance for fixes

Impact:

• Enables batch processing of 6-30+ protected OpenAPI specs
• Creates full audit trail via receipts
• Enables pipeline chaining and automation
• Provides actionable error messages for troubleshooting
• Security team can audit operations without credential exposure

Changed

Documentation: docs/Fetch_Specs_With_Auth.md - Complete rewrite

Complete modernization reflecting all improvements:

1. Structure and Organization

   • Updated to 2025-10-03 with script metadata
   • Clear purpose statement with key features list
   • Three-tier output documentation (stdout, stderr, files)
   • Added "what this is NOT" section

2. Configuration Section Enhanced

   • Detailed authentication type explanations (API Key, OAuth 2.0)
   • Example configuration with both apikey and oauth
   • Batch processing guidance (6-30+ endpoints)
   • Configuration notes with scope patterns

3. Usage Examples Modernized

   • Basic usage with uv run (recommended approach)
   • Custom OAuth tenant URL examples with MICROSOFT_LOGIN_GUID
   • Custom output directory examples
   • Verbose logging with LOG_LEVEL environment variable
   • Traditional venv activation for backwards compatibility

4. Output Documentation Comprehensive

   • JSON envelope format with complete example
   • Operational logs description with structured logging
   • File outputs (specs and receipts) with paths
   • Receipt structure documentation
   • All three output channels clearly explained

5. Common Outcomes Table

   • Structured table format for quick reference
   • Status, meaning, and action columns
   • All major HTTP status codes covered (200, 400, 401, 403, 500)
   • WAF detection guidance

6. Pipeline Chaining Section (NEW)

   • Extract fetched spec files with jq examples
   • Process specs programmatically with receipt files
   • Register specs into APIsec (next step workflow)
   • Integration with register_app_from_spec.py
   • Demonstrates machine-readable output value

7. Troubleshooting Greatly Enhanced

   • WAF/Network issues with curl verification commands
   • OAuth authentication failures with detailed solutions
   • API key authentication failures with test commands
   • Invalid JSON/OpenAPI document diagnostics
   • Debug logging with LOG_LEVEL=DEBUG examples

8. FAQ Section Expanded

   • Network access requirements clarification
   • APIsec integration vs spec fetching distinction
   • Batch processing guidance (100+ endpoints)
   • Partial failure handling explanation
   • Credential storage and security best practices
   • Token sanitization guarantees

9. Security Notes Section (NEW)

   • Token sanitization guarantees documented
   • Credential protection in output explained
   • Directory traversal prevention details
   • Force flag security implications warning
   • .gitignore recommendations for config files

Added

• Enhanced logging with token sanitization throughout all authentication flows
• Support for mixed case authType values ("oauth", "oAuth", "OAuth" all work)
• Redundant config field handling (all auth fields can exist, only relevant ones used)

Technical Debt Paid

• Aligned fetch_specs_with_auth.py with project architectural patterns:
  • Standalone by Design ✅
  • JSON Envelope Pattern ✅
  • Receipt Pattern ✅
  • Security by Default ✅
  • Separation of Concerns ✅
• Followed all logging standards from .cursor/rules/logging-standards.mdc
• Followed all development standards from .cursor/rules/development_standards.mdc
• Consistent with reference implementation in register_app_from_spec.py

Migration Guide for v0.0.3

If you're upgrading from v0.0.2:

1. Update output references: Change ./tmp/ to output/ in scripts/docs
2. Replace --debug flag: Use --verbose instead
3. Update automation: Consume stdout JSON envelope instead of exit codes
4. Add environment variables (optional):

   export APISEC_OUTPUT_DIR="output"  # Default, can customize
   export LOG_LEVEL="INFO"            # or DEBUG for verbose

5. Update pipelines: Use jq to extract data from JSON envelope:

   # Old way (doesn't work anymore)
   python scripts/fetch_specs_with_auth.py --config config.json
   
   # New way
   uv run python scripts/fetch_specs_with_auth.py --config config.json | \
     jq -r '.data.fetched_specs[].output_file'

[0.0.2] - 2025-10-01

Added

• New scripts for spec fetching and authentication:
  • scripts/fetch_spec.py - Download OpenAPI specs from URLs with authentication
  • scripts/fetch_specs_with_auth.py - Fetch multiple protected specs
• Authentication configuration examples in config/auth-examples/:
  • API Key, Basic Auth, Bearer Login, OAuth2, Custom Headers
• Shared constants module lib/constants.py
• Example manifest config/example.importAPI.json for batch operations
• Sample OpenAPI spec openapi.json (crAPI) for testing

Changed

• Enhanced logging in scripts:
  • scripts/register_app_with_host_and_spec.py - Added structured logging
  • scripts/run_scan.py - Added correlation ID tracking
  • scripts/update_app_and_instances.py - Added structured logging

Fixed

• Security: Removed unnecessary --force flag bypasses in internal script calls
  • register_app_from_spec.py - temp directory already validated by parent
  • refresh_instance_spec.py - temp directory already validated by parent
  • Temp directories created inside validated output directory don't need force override

[0.0.1] - 2025-09-25

Added

Core features

• Python standalone scripts for APIsec NG:
  • register_app_from_spec.py: Register an application from OpenAPI (file or URL). Automatically normalizes tokens and records a receipt to output/.
  • register_app_with_host_and_spec.py: Register an app and explicitly add an instance using a host-url.
  • update_app_and_instances.py: Operational utilities:
    • add-instance: Add instance via batch API.
    • show-instance: Fetch instance details.
    • show-app: Fetch application details.
  • refresh_instance_spec.py: Refresh an application's OpenAPI spec from file or URL.
  • run_scan.py: Trigger a scan for an instance; supports --wait with timeout and polling.

Dev UX and tooling

• Makefile with convenience targets mirroring the above scripts:
  • reg-file, reg-url, refresh-file, scan-wait
  • Defaults are overridable (e.g., APP_NAME, SPEC_FILE, SPEC_URL)
• Shell aliases in scripts/aliases.sh:
  • apisec_reg_file, apisec_reg_url, apisec_refresh_file, apisec_scan_wait
  • Designed for quick interactive usage; mirror Makefile behavior
• uv as the Python runner for reproducible, dependency-light execution

Documentation

• Quick Start (docs/QUICK_START.md) with both uv run commands and Makefile/alias usage
• README.md updated to reference make help and aliases
• config/.exampleenv with environment variables for easy setup

Enterprise-readiness and quality

• Consistent JSON envelope across all scripts for pipeline-friendly parsing:
  • Fields: script, version, timestamp, request_id, status, data|error
• Actionable error handling via structured JSON responses; non-zero exit codes for CI signaling
• Output receipts written to output/ with timestamped filenames for auditability
• Guardrails on filesystem writes to avoid accidental writes outside examples/python-scripts unless --force is provided
• Token normalization automatically handles raw tokens and Bearer prefix
• Verbose mode surfaces curl stderr for diagnostics; dry-run where appropriate
• Strong typing where helpful and clear function boundaries in lib/common.py

CI/CD readiness

• Commands suitable for use in GitHub Actions and other CI runners
• Deterministic CLI surfaces with explicit required flags and sane defaults

Housekeeping

• Valid CITATION.cff
• pyproject.toml cleaned of unsupported uv fields; adds readme, license, and authors