Skip to content

TESTING.md

Latest commit

 

History

History
211 lines (171 loc) · 6.92 KB

TESTING.md

File metadata and controls

211 lines (171 loc) · 6.92 KB

Sake Testing Status

Overview

This document summarizes the current state of tests after fixing issues from 6 years of unmaintained development.

Summary

✅ Unit tests: All passing (5/5)
✅ Cross-platform functional tests: All passing (3/3 test suites, 25+ individual tests)
⚠️ OS-specific functional tests: Not running on Windows (test1, test2, test3)

Unit Tests (test_sake.py)

Status: ✅ All passing (5/5 tests)

Fixed Issues:

  1. yaml.load() missing Loader argument - Added Loader=yaml.Loader parameter (required in PyYAML 5.1+)
  2. Path separator incompatibility - Normalized paths in tests to handle Windows backslashes vs Unix forward slashes

Tests:

  • test_clean_path - Path normalization across platforms
  • test_escp - Escaping of paths with spaces
  • test_expand_macros - Macro expansion in Sakefiles
  • test_get_help - Help text generation
  • test_get_all_outputs - Glob expansion for outputs

Functional Tests

Cross-Platform Tests (OS-Independent)

test4: Simple Cross-Platform Functional Test

Status: ✅ All passing (7/7 tests)
Coverage: Basic functionality

  • Clean operation
  • Recon mode (dry run)
  • Full build with dependencies
  • No-op rebuilds
  • Help command
  • Python-only commands (portable)

test5: Parallel Build Testing

Status: ✅ All passing (8/8 tests)
Coverage: Parallel execution and meta-targets

  • Sequential vs parallel build comparison
  • Parallel speedup verification (~2x faster)
  • Meta-targets with multiple atomic targets
  • Rebuild detection after file changes
  • Help with meta-targets
  • Clean operation
  • Uses Python time.sleep() instead of Unix sleep

test6: Advanced Features Testing

Status: ✅ All passing (10/10 tests)
Coverage: Advanced sake features

  • Macro substitution (#! syntax)
  • Conditional macros (?= operator)
  • Force rebuild (-F flag)
  • Force recon (-F -r)
  • Specific meta-target selection
  • CLI macro override (-D flag)
  • Wildcard dependencies
  • Help command
  • Clean operation
  • Python-only implementation

OS-Specific Tests (Unix-only)

test1: Multi-Tool Pipeline

Status: ⚠️ Not portable to Windows
Requires: curl, gnumeric (ssconvert), bash, R
Purpose: Tests complex multi-tool workflow with web fetch and data conversion
Note: Demonstrates real-world usage but requires Unix-specific tools

test2: Parallel Execution Demo

Status: ⚠️ Not portable to Windows (replaced by test5)
Requires: bash, sleep, echo, cat, tr, cowsay
Purpose: Demonstrates parallel builds with timing
Note: test5 provides equivalent cross-platform coverage

test3: Comprehensive API Test

Status: ⚠️ Not portable to Windows
Requires: gcc, bash, pandoc, figlet, tar, perl
Purpose: Comprehensive test of entire sake API with C compilation workflow
Note: Covers many advanced features but requires Unix build tools

Test Coverage Summary

Features Tested Cross-Platform ✅

  • Basic build workflow
  • Dependency tracking
  • Recon/dry-run mode
  • Clean operation
  • Help generation
  • Parallel builds (test5)
  • Meta-targets (test5, test6)
  • Macro substitution (test6)
  • Conditional macros (test6)
  • Force rebuild (test6)
  • CLI macro override (test6)
  • Wildcard dependencies (test6)
  • File change detection (test5, test6)

Features Only Tested on Unix ⚠️

  • C compilation workflows (test3)
  • Complex multi-tool pipelines (test1, test3)
  • Visual/graphviz output (test3)
  • Pattern matching in Sakefiles (test3)

Running Tests

All Unit Tests

python test_sake.py

All Cross-Platform Functional Tests

cd functests
python run_cross_platform_tests.py

Individual Functional Tests

cd functests/test4  # or test5, test6
python functest.py

Unix-Only Tests (Linux/macOS)

cd functests/test3
python functest.py

Changes Made

1. Fixed Unit Tests

  • File: test_sake.py
  • Changes:
    • Added Loader=yaml.Loader to yaml.load() call
    • Added path normalization helper for cross-platform compatibility
    • Normalized all path assertions to use forward slashes

2. Fixed pkg_resources Deprecation Warning

  • Files: setup.py, sakelib/main.py, sake
  • Changes:
    • Migrated from scripts to entry_points in setup.py
    • Created sakelib/main.py as the proper entry point module
    • Modern setuptools now generates scripts without deprecated pkg_resources
    • Maintained backward compatibility with the sake script
  • Issue: Fixed DeprecationWarning: pkg_resources is deprecated as an API

3. Created test5 (Parallel Build Testing)

  • Files: functests/test5/
    • Sakefile.yaml - 4 parallel targets + combine step
    • functest.py - Test runner with timing
    • README.md - Documentation
  • Purpose: Cross-platform replacement for test2's parallel build testing
  • Implementation: Uses Python instead of Unix commands

4. Created test6 (Advanced Features)

  • Files: functests/test6/
    • Sakefile.yaml - Macro-heavy build with meta-targets
    • functest.py - Test runner
    • input.txt - Test data
    • README.md - Documentation
  • Purpose: Cross-platform testing of advanced sake features
  • Implementation: Pure Python, tests macros, force rebuild, wildcards

5. Created Test Runner

  • File: functests/run_cross_platform_tests.py
  • Purpose: Run all OS-independent tests with summary report
  • Features: Colored output, platform detection, failure reporting

Dependencies

Required (for all tests)

  • Python 2.7 or 3.x
  • PyYAML >= 3.0
  • networkx >= 1.0

Optional (for Unix-only tests)

  • gcc (test3)
  • pandoc (test3)
  • figlet (test3)
  • curl (test1)
  • R (test1)
  • gnumeric/ssconvert (test1)

Recommendations

  1. CI/CD: Use run_cross_platform_tests.py for continuous integration on all platforms
  2. Development: Run unit tests (test_sake.py) frequently during development
  3. Unix-specific: Run test3 on Linux/macOS to verify complex workflows
  4. Coverage: The cross-platform tests provide good coverage of core functionality

Migration Status

✅ Complete: All OS-independent functionality has cross-platform test coverage
✅ Unit tests: All fixed and passing
✅ Functional tests: Essential features covered cross-platform (test4, test5, test6)
⚠️ Advanced features: Some Unix-specific tests remain (test1, test3) but are not critical

Test Statistics

  • Total test suites: 9 (1 unit + 8 functional)
  • Cross-platform test suites: 4 (1 unit + 3 functional)
  • Individual test cases: 30+ across all suites
  • Cross-platform test cases: 25+ (unit + functional)
  • Pass rate: 100% on Windows
  • Expected pass rate on Linux/macOS: 100% (all tests should work)