Add automated reconstruction quality analysis workflow for truth-reconstruction comparisons

.github/requirements.txt

@@ -2,3 +2,4 @@ awkward
 deepdiff
 scipy
 uproot
+numpy

.github/workflows/linux-eic-shell.yml

@@ -1277,9 +1277,94 @@ jobs:
           name: capybara_${{ github.event.pull_request.number }}.md
           path: capybara_${{ github.event.pull_request.number }}.md
 
+  reconstruction-quality-analysis:
+    runs-on: ubuntu-24.04
+    needs:
+      - eicrecon-gun
+      - eicrecon-dis
+    strategy:
+      matrix:
+        include:
+          # Gun reconstruction analysis
+          - particle: pi
+            detector_config: craterlake
+            file_type: gun
+          - particle: e
+            detector_config: craterlake
+            file_type: gun
+          # DIS reconstruction analysis
+          - beam: 10x100
+            minq2: 1000
+            detector_config: craterlake_tracking_only
+            file_type: dis
+          - beam: 18x275
+            minq2: 1000
+            detector_config: craterlake_18x275
+            file_type: dis
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v5
+        with:
+          sparse-checkout: scripts
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+      - name: Install Python dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install uproot awkward numpy
+      - name: Download gun reconstruction artifacts
+        if: matrix.file_type == 'gun'
+        uses: actions/download-artifact@v5
+        with:
+          name: rec_${{ matrix.particle }}_1GeV_20GeV_${{ matrix.detector_config }}.edm4eic.root
+          path: ./
+      - name: Download DIS reconstruction artifacts
+        if: matrix.file_type == 'dis'
+        uses: actions/download-artifact@v5
+        with:
+          name: rec_dis_${{ matrix.beam }}_minQ2=${{ matrix.minq2 }}_${{ matrix.detector_config }}.edm4eic.root
+          path: ./
+      - name: Run reconstruction quality analysis (gun)
+        if: matrix.file_type == 'gun'
+        run: |
+          chmod +x scripts/compare_reconstruction_truth.py
+          echo "Running reconstruction quality analysis for particle gun..."
+          python scripts/compare_reconstruction_truth.py rec_${{ matrix.particle }}_1GeV_20GeV_${{ matrix.detector_config }}.edm4eic.root --output reconstruction_quality_${{ matrix.particle }}_${{ matrix.detector_config }}.txt || echo "Analysis failed for gun file"
+      - name: Run reconstruction quality analysis (DIS)
+        if: matrix.file_type == 'dis'
+        run: |
+          chmod +x scripts/compare_reconstruction_truth.py
+          echo "Running reconstruction quality analysis for DIS..."
+          python scripts/compare_reconstruction_truth.py rec_dis_${{ matrix.beam }}_minQ2=${{ matrix.minq2 }}_${{ matrix.detector_config }}.edm4eic.root --output reconstruction_quality_dis_${{ matrix.beam }}_${{ matrix.detector_config }}.txt || echo "Analysis failed for DIS file"
+      - name: Display results
+        run: |
+          echo "=== RECONSTRUCTION QUALITY ANALYSIS RESULTS ==="
+          for result_file in reconstruction_quality_*.txt; do
+            if [ -f "$result_file" ]; then
+              echo "Results from: $result_file"
+              echo "----------------------------------------"
+              cat "$result_file"
+              echo ""
+              echo "========================================"
+              echo ""
+            fi
+          done
+          if ! ls reconstruction_quality_*.txt 1> /dev/null 2>&1; then
+            echo "No analysis results found"
+          fi
+      - name: Upload analysis results
+        uses: actions/upload-artifact@v4
+        with:
+          name: reconstruction-quality-${{ matrix.file_type }}-${{ matrix.particle || matrix.beam }}-${{ matrix.detector_config }}
+          path: reconstruction_quality_*.txt
+          if-no-files-found: error
+
   deploy-docs:
     needs:
       - collect-docs
+      - reconstruction-quality-analysis
     permissions:
       pages: write
       id-token: write

.gitignore

@@ -168,4 +168,9 @@ src/tools/default_flags_table/fieldmaps/
 
 src/examples/test_data_generator/__pycache__/
 
+# Python cache files
+__pycache__/
+*.py[cod]
+*$py.class
+
 .run/

docs/design/reconstruction_quality.md

@@ -0,0 +1,127 @@
+# Reconstruction Quality Analysis
+
+This document describes the automated reconstruction quality analysis that compares reconstructed particle quantities against Monte Carlo truth.
+
+## Overview
+
+The EICrecon GitHub Actions workflow includes automated quality assessment that:
+
+1. **Analyzes reconstruction outputs** from both particle gun and deep inelastic scattering (DIS) events
+2. **Compares reconstructed quantities to truth** using MC-reconstruction associations
+3. **Calculates performance metrics** including accuracy, precision, and efficiency
+4. **Provides quality assessments** to help identify reconstruction issues
+
+## Implementation
+
+### Analysis Script
+
+The analysis is performed by `scripts/compare_reconstruction_truth.py`, which:
+
+- Reads EDM4hep ROOT files produced by EICrecon
+- Extracts MC truth and reconstructed particle information
+- Uses association collections to match truth and reconstructed particles
+- Calculates ratios (reconstructed/truth) for key quantities
+- Provides statistical analysis and quality assessments
+
+### GitHub Actions Integration
+
+The `reconstruction-quality-analysis` job in the CI workflow:
+
+- Depends on `eicrecon-gun` and `eicrecon-dis` jobs
+- Downloads reconstruction artifacts from those jobs
+- Runs analysis for multiple detector configurations
+- Uploads results as artifacts for review
+- Displays summary in the workflow output
+
+### Analyzed Collections
+
+The script analyzes the following data when associations are available:
+
+1. **Reconstructed Particles**:
+   - Uses `MCParticles`, `ReconstructedParticles`, and `ReconstructedParticleAssociations`
+   - Compares momentum components, total momentum, and energy
+
+2. **Central Tracking**:
+   - Uses `CentralCKFTracks` and `CentralCKFTrackAssociations`
+   - Compares track-level momentum measurements
+
+3. **Truth-Seeded Tracking**:
+   - Uses `CentralCKFTruthSeededTracks` and associations
+   - Provides baseline performance with perfect seeding
+
+## Metrics and Quality Assessment
+
+### Key Metrics
+
+For each quantity, the analysis reports:
+
+- **Mean ratio**: Average of reconstructed/truth ratios
+- **Resolution**: Standard deviation relative to mean (σ/μ)
+- **Efficiency**: Fraction of MC particles successfully reconstructed
+- **Percentile ranges**: Distribution characteristics
+
+### Quality Grades
+
+The analysis provides qualitative assessments:
+
+- **EXCELLENT**: Mean ratio 0.9-1.1, resolution < 20%
+- **GOOD**: Mean ratio 0.8-1.2, resolution < 30%
+- **FAIR**: Mean ratio 0.7-1.3, resolution < 50%
+- **POOR**: Outside fair range
+
+### Example Output
+
+```
+=== Reconstructed Particles Analysis ===
+Total MC particles: 1500
+Total reconstructed particles: 1342
+Successfully matched: 1298
+Reconstruction efficiency: 86.5%
+
+Momentum:
+  Mean ratio: 0.995 ± 0.087
+  Median ratio: 0.998
+  RMS: 1.002
+  Resolution (σ/μ): 0.087
+  N matched: 1298 / 1298 total
+  25th-75th percentile: [0.942, 1.047]
+  Quality assessment: EXCELLENT
+```
+
+## Usage
+
+### Manual Analysis
+
+To run the analysis manually:
+
+```bash
+# Analyze a single file
+python scripts/compare_reconstruction_truth.py reconstruction_output.edm4eic.root
+
+# Analyze multiple files with output to file
+python scripts/compare_reconstruction_truth.py *.edm4eic.root --output results.txt
+```
+
+### Interpreting Results
+
+- **Ratios near 1.0** indicate accurate reconstruction
+- **Small resolution values** indicate precise/consistent reconstruction
+- **High efficiency** indicates good particle-finding capability
+- **Compare different detector configurations** to assess design choices
+- **Compare gun vs DIS events** to understand performance in different environments
+
+## Integration with CI/CD
+
+The analysis runs automatically on:
+
+- Pull requests (for changed code validation)
+- Main branch pushes (for continuous monitoring)
+- Scheduled runs (for regression detection)
+
+Results are:
+
+- Displayed in the GitHub Actions output
+- Uploaded as downloadable artifacts
+- Can be integrated into automated quality gates
+
+This provides continuous monitoring of reconstruction performance and helps identify regressions or improvements in the codebase.

scripts/README.md

@@ -0,0 +1,68 @@
+# Analysis Scripts
+
+This directory contains analysis scripts for EICrecon reconstruction quality assessment.
+
+## compare_reconstruction_truth.py
+
+This script compares reconstructed particle properties against Monte Carlo truth for collections where truth-reconstruction associations exist. It calculates and prints quantitative comparisons in terms of reconstructed/truth ratios to assess reconstruction performance.
+
+### Features
+
+- Reads EDM4hep ROOT files produced by EICrecon
+- Analyzes particle-level quantities (momentum, energy) using `ReconstructedParticleAssociations`
+- Analyzes track-level quantities using `CentralCKFTrackAssociations`
+- Supports both regular and truth-seeded reconstruction
+- Provides statistical analysis including mean, median, RMS, and resolution metrics
+- Filters outliers to provide meaningful statistics
+
+### Usage
+
+```bash
+python compare_reconstruction_truth.py <input_files...> [--output output.txt]
+```
+
+### Example
+
+```bash
+# Analyze a single file
+python compare_reconstruction_truth.py rec_e_1GeV_20GeV_craterlake.edm4eic.root
+
+# Analyze multiple files with output to file
+python compare_reconstruction_truth.py *.edm4eic.root --output analysis_results.txt
+```
+
+### Output
+
+The script prints analysis results for each collection type:
+
+1. **Reconstructed Particles**: Momentum and energy ratios
+2. **Central CKF Tracks**: Track parameter ratios
+3. **Truth-Seeded Tracks**: Performance with perfect seeding
+
+For each quantity, the following statistics are reported:
+- Mean ratio ± standard deviation
+- Median ratio
+- RMS
+- Resolution (σ/μ)
+- Number of matched particles/tracks
+- 25th-75th percentile range
+
+### Dependencies
+
+- `uproot`: For reading ROOT files
+- `awkward`: For handling jagged arrays
+- `numpy`: For numerical calculations
+
+### Integration with GitHub Actions
+
+This script is automatically run in the EICrecon GitHub Actions workflow via the `reconstruction-quality-analysis` job, which:
+
+1. Downloads reconstruction output files from `eicrecon-gun` and `eicrecon-dis` jobs
+2. Runs the analysis on various detector configurations
+3. Uploads the results as artifacts
+4. Displays a summary in the workflow output
+
+The analysis covers:
+- Particle gun events (pions and electrons)
+- Deep inelastic scattering events
+- Multiple detector configurations (craterlake, craterlake_tracking_only, craterlake_18x275)