Skip to content

TESTS.md

Latest commit

 

History

History
590 lines (441 loc) · 15.1 KB

TESTS.md

File metadata and controls

590 lines (441 loc) · 15.1 KB

TrustGraph Test Suite

This document provides instructions for running and maintaining the TrustGraph test suite.

Overview

The TrustGraph test suite follows the testing strategy outlined in TEST_STRATEGY.md and implements the test cases defined in TEST_CASES.md. The tests are organized into unit tests, integration tests, and performance tests.

Test Structure

tests/
├── unit/
│   ├── test_text_completion/
│   │   ├── test_vertexai_processor.py
│   │   ├── conftest.py
│   │   └── __init__.py
│   ├── test_embeddings/
│   ├── test_storage/
│   └── test_query/
├── integration/
│   ├── test_flows/
│   └── test_databases/
├── fixtures/
│   ├── messages.py
│   ├── configs.py
│   └── mocks.py
├── requirements.txt
├── pytest.ini
└── conftest.py

Prerequisites

Install TrustGraph Packages

The tests require TrustGraph packages to be installed. You can use the provided scripts:

Option 1: Automated Setup (Recommended)

# From the project root directory - runs all setup steps
./run_tests.sh

Option 2: Step-by-step Setup

# Check what imports are working
./check_imports.py

# Install TrustGraph packages
./install_packages.sh

# Verify imports work
./check_imports.py

# Install test dependencies
cd tests/
pip install -r requirements.txt
cd ..

Option 3: Manual Installation

# Install base package first (required by others)
cd trustgraph-base
pip install -e .
cd ..

# Install vertexai package (depends on base)
cd trustgraph-vertexai  
pip install -e .
cd ..

# Install flow package (for additional components)
cd trustgraph-flow
pip install -e .
cd ..

Install Test Dependencies

cd tests/
pip install -r requirements.txt

Required Dependencies

  • pytest>=7.0.0 - Testing framework
  • pytest-asyncio>=0.21.0 - Async testing support
  • pytest-mock>=3.10.0 - Mocking utilities
  • pytest-cov>=4.0.0 - Coverage reporting
  • google-cloud-aiplatform>=1.25.0 - Google Cloud dependencies
  • google-auth>=2.17.0 - Authentication
  • google-api-core>=2.11.0 - API core
  • pulsar-client>=3.0.0 - Pulsar messaging
  • prometheus-client>=0.16.0 - Metrics

Running Tests

Basic Test Execution

# Run all tests
pytest

# Run tests with verbose output
pytest -v

# Run specific test file
pytest tests/unit/test_text_completion/test_vertexai_processor.py

# Run specific test class
pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIProcessorInitialization

# Run specific test method
pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIProcessorInitialization::test_processor_initialization_with_valid_credentials

Test Categories

# Run only unit tests
pytest -m unit

# Run only integration tests
pytest -m integration

# Run only VertexAI tests
pytest -m vertexai

# Exclude slow tests
pytest -m "not slow"

Coverage Reports

# Run tests with coverage
pytest --cov=trustgraph

# Generate HTML coverage report
pytest --cov=trustgraph --cov-report=html

# Generate terminal coverage report
pytest --cov=trustgraph --cov-report=term-missing

# Fail if coverage is below 80%
pytest --cov=trustgraph --cov-fail-under=80

VertexAI Text Completion Tests

Test Implementation

The VertexAI text completion service tests are located in:

  • Main test file: tests/unit/test_text_completion/test_vertexai_processor.py
  • Fixtures: tests/unit/test_text_completion/conftest.py

Test Coverage

The VertexAI tests include 139 test cases covering:

1. Processor Initialization Tests (6 tests)

  • Service account credential loading
  • Model configuration (Gemini models)
  • Custom parameters (temperature, max_output, region)
  • Generation config and safety settings
# Run initialization tests
pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIProcessorInitialization -v

2. Message Processing Tests (5 tests)

  • Simple text completion
  • System instructions handling
  • Long context processing
  • Empty prompt handling
# Run message processing tests
pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIMessageProcessing -v

3. Safety Filtering Tests (2 tests)

  • Safety settings configuration
  • Blocked content handling
# Run safety filtering tests
pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAISafetyFiltering -v

4. Error Handling Tests (7 tests)

  • Rate limiting (ResourceExhausted → TooManyRequests)
  • Authentication errors
  • Generic exceptions
  • Model not found errors
  • Quota exceeded errors
  • Token limit errors
# Run error handling tests
pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIErrorHandling -v

5. Metrics Collection Tests (4 tests)

  • Token usage tracking
  • Request duration measurement
  • Error rate collection
  • Cost calculation basis
# Run metrics collection tests
pytest tests/unit/test_text_completion/test_vertexai_processor.py::TestVertexAIMetricsCollection -v

Running All VertexAI Tests

Option 1: Simple Tests (Recommended for getting started)

# Run simple tests that don't require full TrustGraph infrastructure
./run_simple_tests.sh

# Or run manually:
pytest tests/unit/test_text_completion/test_vertexai_simple.py -v
pytest tests/unit/test_text_completion/test_vertexai_core.py -v

Option 2: Full Infrastructure Tests

# Run all VertexAI tests (requires full TrustGraph setup)
pytest tests/unit/test_text_completion/test_vertexai_processor.py -v

# Run with coverage
pytest tests/unit/test_text_completion/test_vertexai_processor.py --cov=trustgraph.model.text_completion.vertexai

# Run with detailed output
pytest tests/unit/test_text_completion/test_vertexai_processor.py -v -s

Option 3: All VertexAI Tests

# Run all VertexAI-related tests
pytest tests/unit/test_text_completion/ -k "vertexai" -v

Test Configuration

Pytest Configuration

The test suite uses the following configuration in pytest.ini:

[tool:pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts = 
    -v
    --tb=short
    --strict-markers
    --disable-warnings
    --cov=trustgraph
    --cov-report=html
    --cov-report=term-missing
    --cov-fail-under=80
asyncio_mode = auto
markers =
    slow: marks tests as slow (deselect with '-m "not slow"')
    integration: marks tests as integration tests
    unit: marks tests as unit tests
    vertexai: marks tests as vertex ai specific tests

Test Markers

Use pytest markers to categorize and filter tests:

@pytest.mark.unit
@pytest.mark.vertexai
async def test_vertexai_functionality():
    pass

@pytest.mark.integration
@pytest.mark.slow
async def test_end_to_end_flow():
    pass

Test Development Guidelines

Following TEST_STRATEGY.md

  1. Mock External Dependencies: Always mock external services (APIs, databases, Pulsar)
  2. Test Business Logic: Focus on testing your code, not external infrastructure
  3. Use Dependency Injection: Make services testable by injecting dependencies
  4. Async Testing: Use proper async test patterns for async services
  5. Comprehensive Coverage: Test success paths, error paths, and edge cases

Test Structure Example

class TestServiceName(IsolatedAsyncioTestCase):
    """Test service functionality"""

    def setUp(self):
        """Set up test fixtures"""
        self.config = {...}

    @patch('external.dependency')
    async def test_success_case(self, mock_dependency):
        """Test successful operation"""
        # Arrange
        mock_dependency.return_value = expected_result
        
        # Act
        result = await service.method()
        
        # Assert
        assert result == expected_result
        mock_dependency.assert_called_once()

Fixture Usage

Use fixtures from conftest.py to reduce code duplication:

async def test_with_fixtures(self, mock_vertexai_model, sample_text_completion_request):
    """Test using shared fixtures"""
    # Fixtures are automatically injected
    result = await processor.process(sample_text_completion_request)
    assert result.text == "Test response"

Debugging Tests

Running Tests with Debug Information

# Run with debug output
pytest -v -s tests/unit/test_text_completion/test_vertexai_processor.py

# Run with pdb on failures
pytest --pdb tests/unit/test_text_completion/test_vertexai_processor.py

# Run with detailed tracebacks
pytest --tb=long tests/unit/test_text_completion/test_vertexai_processor.py

Common Issues and Solutions

1. Import Errors

Symptom: ModuleNotFoundError: No module named 'trustgraph' or similar import errors

Solution:

# First, check what's working
./check_imports.py

# Install the required packages
./install_packages.sh

# Verify installation worked
./check_imports.py

# If still having issues, check Python path
echo $PYTHONPATH
export PYTHONPATH=/home/mark/work/trustgraph.ai/trustgraph:$PYTHONPATH

# Try running tests from project root
cd /home/mark/work/trustgraph.ai/trustgraph
pytest tests/unit/test_text_completion/test_vertexai_processor.py -v

Common causes:

  • TrustGraph packages not installed (pip install -e . in each package directory)
  • Wrong working directory (should be in project root)
  • Python path not set correctly
  • Missing dependencies (install with pip install -r tests/requirements.txt)

2. TaskGroup/Infrastructure Errors

Symptom: RuntimeError: Essential taskgroup missing or similar infrastructure errors

Solution:

# Try the simple tests first - they don't require full TrustGraph infrastructure
./run_simple_tests.sh

# Or run specific simple test files
pytest tests/unit/test_text_completion/test_vertexai_simple.py -v
pytest tests/unit/test_text_completion/test_vertexai_core.py -v

Why this happens:

  • The full TrustGraph processors require async task groups and Pulsar infrastructure
  • The simple tests focus on testing the core logic without infrastructure dependencies
  • Use simple tests to verify the VertexAI logic works correctly

3. Async Test Issues

# Use IsolatedAsyncioTestCase for async tests
class TestAsyncService(IsolatedAsyncioTestCase):
    async def test_async_method(self):
        result = await service.async_method()
        assert result is not None

3. Mock Issues

# Use proper async mocks for async methods
mock_client = AsyncMock()
mock_client.async_method.return_value = expected_result

# Use MagicMock for sync methods
mock_client = MagicMock()
mock_client.sync_method.return_value = expected_result

Continuous Integration

Running Tests in CI

# Install dependencies
pip install -r tests/requirements.txt

# Run tests with coverage
pytest --cov=trustgraph --cov-report=xml --cov-fail-under=80

# Run tests in parallel (if using pytest-xdist)
pytest -n auto

Test Reports

The test suite generates several types of reports:

  1. Coverage Reports: HTML and XML coverage reports
  2. Test Results: JUnit XML format for CI integration
  3. Performance Reports: For performance and load tests
# Generate all reports
pytest --cov=trustgraph --cov-report=html --cov-report=xml --junitxml=test-results.xml

Adding New Tests

1. Create Test File

# tests/unit/test_new_service/test_new_processor.py
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
from unittest import IsolatedAsyncioTestCase

from trustgraph.new_service.processor import Processor

class TestNewProcessor(IsolatedAsyncioTestCase):
    """Test new processor functionality"""
    
    def setUp(self):
        self.config = {...}
    
    @patch('trustgraph.new_service.processor.external_dependency')
    async def test_processor_method(self, mock_dependency):
        """Test processor method"""
        # Arrange
        mock_dependency.return_value = expected_result
        processor = Processor(**self.config)
        
        # Act
        result = await processor.method()
        
        # Assert
        assert result == expected_result

2. Create Fixtures

# tests/unit/test_new_service/conftest.py
import pytest
from unittest.mock import MagicMock

@pytest.fixture
def mock_new_service_client():
    """Mock client for new service"""
    return MagicMock()

@pytest.fixture
def sample_request():
    """Sample request object"""
    return RequestObject(id="test", data="test data")

3. Update pytest.ini

markers =
    new_service: marks tests as new service specific tests

Performance Testing

Load Testing

# Run performance tests
pytest -m performance tests/performance/

# Run with custom parameters
pytest -m performance --count=100 --concurrent=10

Memory Testing

# Run with memory profiling
pytest --profile tests/unit/test_text_completion/test_vertexai_processor.py

Best Practices

1. Test Naming

  • Use descriptive test names that explain what is being tested
  • Follow the pattern: test_<method>_<scenario>_<expected_result>

2. Test Organization

  • Group related tests in classes
  • Use meaningful class names that describe the component being tested
  • Keep tests focused on a single aspect of functionality

3. Mock Strategy

  • Mock external dependencies, not internal business logic
  • Use the most specific mock type (AsyncMock for async, MagicMock for sync)
  • Verify mock calls to ensure proper interaction

4. Assertions

  • Use specific assertions that clearly indicate what went wrong
  • Test both positive and negative cases
  • Include edge cases and boundary conditions

5. Test Data

  • Use fixtures for reusable test data
  • Keep test data simple and focused
  • Avoid hardcoded values when possible

Troubleshooting

Common Test Failures

  1. Import Errors: Check PYTHONPATH and module structure
  2. Async Issues: Ensure proper async/await usage and AsyncMock
  3. Mock Failures: Verify mock setup and expected call patterns
  4. Coverage Issues: Check for untested code paths

Getting Help

  • Check the TEST_STRATEGY.md for testing patterns
  • Review TEST_CASES.md for comprehensive test scenarios
  • Examine existing tests for examples and patterns
  • Use pytest's built-in help: pytest --help

Future Enhancements

Planned Test Additions

  1. Integration Tests: End-to-end flow testing
  2. Performance Tests: Load and stress testing
  3. Security Tests: Input validation and authentication
  4. Contract Tests: API contract verification

Test Infrastructure Improvements

  1. Parallel Test Execution: Using pytest-xdist
  2. Test Data Management: Better fixture organization
  3. Reporting: Enhanced test reporting and metrics
  4. CI Integration: Automated test execution and reporting

This testing guide provides comprehensive instructions for running and maintaining the TrustGraph test suite. Follow the patterns and guidelines to ensure consistent, reliable, and maintainable tests across all services.