Skip to content

TESTING.md

Latest commit

 

History

History
300 lines (240 loc) · 7.77 KB

TESTING.md

File metadata and controls

300 lines (240 loc) · 7.77 KB
Raw

Testing and Quality Assurance

The VEX Kernel Checker project includes a comprehensive testing framework to ensure reliability, performance, and maintainability. This document outlines the testing strategy and quality assurance processes.

Testing Infrastructure Overview

Test Coverage

  • Unit Tests: Core functionality, configuration analysis, CVE filtering
  • Integration Tests: End-to-end workflows, file system operations
  • Performance Tests: Scalability, memory usage, caching effectiveness
  • Validation Tests: Environment setup, configuration verification

Quality Metrics

  • Code coverage target: >90%
  • Performance benchmarking for all major operations
  • Automated configuration validation
  • Cross-platform compatibility testing

Quick Start Testing

Run All Tests

# Using make (recommended)
make test

# Using test runner directly
python3 tests/run_tests.py

# Run specific test category
python3 tests/test_response_field.py      # Response field tests
python3 tests/test_error_handling.py      # Error handling tests
python3 tests/test_kernel_filtering.py    # Kernel filtering tests

Quick Validation

# Quick smoke test
make test-quick

# Validate your configuration
make validate VEX_FILE=examples/test_real_cve.json KERNEL_CONFIG=/path/to/.config KERNEL_SOURCE=/path/to/kernel

Performance Analysis

# Run performance benchmarks
make benchmark

# Generate detailed performance report
python3 tests/benchmark.py --output performance_report.json

Testing Tools

1. Test Runner (tests/run_tests.py)

Comprehensive test execution with coverage reporting:

  • Automatic test discovery
  • Coverage reporting with HTML output
  • Dependency validation
  • Quick smoke tests for rapid development

2. Configuration Validator (tests/validate_config.py)

Environment and configuration validation:

  • Python environment compatibility
  • File format validation (VEX, kernel config)
  • WebDriver and API key verification
  • Detailed error reporting with recommendations

3. Performance Benchmarker (tests/benchmark.py)

Performance testing and optimization:

  • Multi-scale dataset testing (10-200 CVEs)
  • Memory usage tracking
  • Cache performance analysis
  • Statistical performance reporting

4. Unit Test Suite (tests/test_vex_kernel_checker.py)

Comprehensive unit and integration tests:

  • Core functionality validation
  • Error handling verification
  • Mock API testing
  • Temporary environment testing

Development Workflow

Before Committing

make workflow-fix

This runs:

  • Code formatting
  • Linting
  • Quick tests

Before Creating PR

make workflow-pr

This runs:

  • Full test suite with coverage
  • Code quality checks
  • Performance benchmarking
  • Clean environment validation

Setting Up Development Environment

make setup-dev

This installs all development dependencies and prepares the environment.

Continuous Integration

GitHub Actions

The project includes a comprehensive CI/CD pipeline (.github/workflows/ci.yml):

  • Multi-Python Testing: Tests across Python 3.7-3.11
  • Code Quality: Linting and formatting checks
  • Performance Monitoring: Automated benchmarking
  • Integration Testing: End-to-end workflow validation

Local CI Simulation

# Run the same checks as CI
make ci-test
make ci-lint
make ci-benchmark

Test Categories

Unit Tests

Focus on individual components:

  • Configuration parsing
  • Pattern matching algorithms
  • Data validation
  • Error handling

Integration Tests

Test complete workflows:

  • VEX file processing
  • Kernel source analysis
  • Configuration option detection
  • Report generation

Performance Tests

Ensure scalability:

  • Large dataset processing (200+ CVEs)
  • Memory usage optimization
  • Cache hit rate analysis
  • Concurrent processing

Validation Tests

Environment verification:

  • Dependency availability
  • File format correctness
  • External tool connectivity

Quality Gates

Before Release

All of the following must pass:

  • Full test suite (>90% coverage)
  • Performance benchmarks within acceptable ranges
  • Code quality checks (linting, formatting)
  • Documentation validation
  • Cross-platform compatibility verification

Performance Thresholds

  • CVE processing: <2 seconds per CVE (config-only mode)
  • Memory usage: <500MB for 200 CVEs
  • Cache hit rate: >80% for repeated operations

Code Quality Standards

  • Flake8 compliance (max complexity: 10)
  • Black formatting
  • Comprehensive docstrings
  • Type hints for public APIs

Testing Best Practices

When Adding Features

  1. Write tests first (TDD approach)
  2. Include both positive and negative test cases
  3. Add performance benchmarks for computationally intensive features
  4. Update validation scripts for new configuration requirements

Test Data Management

  • Use temporary directories for file system tests
  • Create realistic test data that mirrors production scenarios
  • Clean up test resources in tearDown methods
  • Mock external dependencies (APIs, WebDriver)

Performance Testing

  • Benchmark before and after changes
  • Test with various dataset sizes
  • Monitor memory usage patterns
  • Validate cache effectiveness

Troubleshooting Tests

Common Issues

Import Errors:

# Validate Python environment
python3 tests/validate_config.py --check-deps

# Ensure VEX Kernel Checker is importable
python3 tests/run_tests.py --quick

Missing Dependencies:

# Install all required packages
make install-dev

# Check specific dependencies
python3 tests/run_tests.py --check-deps

Performance Regression:

# Generate baseline performance report
python3 tests/benchmark.py --output baseline.json

# Compare after changes
python3 tests/benchmark.py --output new_results.json
# (Manual comparison needed)

WebDriver Issues:

# Validate WebDriver setup
python3 tests/validate_config.py --webdriver /path/to/msedgedriver

Test Environment Reset

# Clean all temporary files and caches
make clean

# Reset development environment
make setup-dev

Contributing to Tests

Adding New Tests

  1. Follow naming convention: test_feature_description
  2. Include docstrings explaining test purpose
  3. Use descriptive assertions with custom error messages
  4. Add to appropriate test class based on functionality

Test Organization

  • TestVexKernelChecker: Core functionality tests
  • TestIntegration: End-to-end workflow tests
  • TestPerformance: Performance and scalability tests
  • TestValidation: Configuration and environment tests

Performance Test Guidelines

  • Use realistic test data sizes
  • Measure both time and memory usage
  • Include statistical analysis (mean, std dev)
  • Set performance thresholds and alerts

Future Testing Enhancements

Planned Improvements

  • Docker-based testing environment
  • Automated performance regression detection
  • Cross-platform testing (Windows, macOS)
  • Integration with external CVE databases for testing
  • Property-based testing for edge cases

Testing Infrastructure Roadmap

  • Parallel test execution for faster feedback
  • Test result visualization dashboard
  • Automated test data generation
  • Fuzz testing for robustness validation

Getting Help

Documentation

  • tests/README.md: Detailed testing documentation
  • README.md: General project documentation
  • docs/: Additional technical documentation

Common Commands

make help                    # Show all available commands
make test-quick             # Quick validation
make validate               # Configuration validation help
python3 tests/run_tests.py --help  # Test runner options

Reporting Issues

When reporting test-related issues:

  1. Include Python version and OS information
  2. Provide full error output from failing tests
  3. Include configuration validation results
  4. Specify which test categories are affected