Skip to content

Commit 890c6cd

Browse files
ArthurCRodriguesArthurCRodrigues
authored
New pipeline documentation (#215)
1 parent d7abdbe commit 890c6cd

10 files changed

Lines changed: 922 additions & 0 deletions

File tree

‎docs/features/grading_engine.md‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# Grading Engine
2+
3+
## Overview
4+
5+
The grading engine is the core subsystem behind the [Grade pipeline step](../pipeline/04-grade.md). It takes a `CriteriaTree` (the rubric) and a student submission, executes every test function in the tree, and produces a `ResultTree` — a scored mirror of the criteria tree with actual results, reports, and aggregated scores at every level.
6+
7+
This document covers the engine's internal mechanics: how it traverses the tree, executes tests, resolves files, handles weights, and calculates scores.
8+
9+
---
10+
11+
## Tree Traversal
12+
13+
The `GraderService.grade_from_tree()` method is the entry point. It processes the criteria tree top-down:
14+
15+
```
16+
CriteriaTree
17+
├── base (CategoryNode) ──▶ process_category() ──▶ CategoryResultNode
18+
├── bonus (CategoryNode) ──▶ process_category() ──▶ CategoryResultNode
19+
└── penalty (CategoryNode) ──▶ process_category() ──▶ CategoryResultNode
20+
```
21+
22+
Each `process_category()` call delegates to `__process_holder()`, a generic method that handles both `CategoryNode` and `SubjectNode` since they share the same structure (subjects + tests + optional `subjects_weight`).
23+
24+
For each holder node:
25+
1. Process all child **subjects** recursively → list of `SubjectResultNode`
26+
2. Process all child **tests** → list of `TestResultNode`
27+
3. Balance weights for each group
28+
4. Return the corresponding result node
29+
30+
---
31+
32+
## Test Execution
33+
34+
When the engine reaches a `TestNode`, it calls `process_test()`:
35+
36+
```python
37+
def process_test(self, test: TestNode) -> TestResultNode:
38+
file_target = self.get_file_target(test)
39+
test_params = test.parameters or {}
40+
if self._submission_language:
41+
test_params['__submission_language__'] = self._submission_language
42+
43+
test_result = test.test_function.execute(
44+
files=file_target, sandbox=self._sandbox, **test_params
45+
)
46+
return TestResultNode(
47+
name=test.name,
48+
test_node=test,
49+
score=test_result.score,
50+
report=test_result.report,
51+
parameters=test_result.parameters,
52+
)
53+
```
54+
55+
Key details:
56+
- **File targeting**: If the `TestNode` specifies a `file_target`, only matching submission files are passed to the test function. Otherwise, `None` is passed and the test operates on the sandbox or all files.
57+
- **Language injection**: The submission language is injected as `__submission_language__` into the test parameters, enabling multi-language command resolution inside test functions (e.g., choosing `python3 main.py` vs `java Main`).
58+
- **Sandbox**: The sandbox container (if any) is passed to every test function. Templates that require sandbox execution (like `input_output`) use it to run student code.
59+
60+
Each `TestFunction.execute()` returns a `TestResult` with:
61+
- `score` (0–100)
62+
- `report` (human-readable explanation)
63+
- `parameters` (optional, echoed back for transparency)
64+
65+
---
66+
67+
## Weight Balancing
68+
69+
Weights determine how much each node contributes to its parent's score. The engine enforces that sibling weights always sum to 100 at every level.
70+
71+
### Sibling Balancing
72+
73+
The `__balance_nodes()` method normalizes weights:
74+
75+
```python
76+
def __balance_nodes(self, nodes, factor):
77+
total_weight = sum(node.weight for node in nodes) * factor
78+
if total_weight == 0:
79+
equal_weight = 100.0 / len(nodes)
80+
for node in nodes:
81+
node.weight = equal_weight
82+
elif total_weight != 100:
83+
scale_factor = 100.0 / total_weight
84+
for node in nodes:
85+
node.weight *= scale_factor
86+
```
87+
88+
- If all weights are zero, they're distributed equally.
89+
- If they don't sum to 100 (after applying the factor), they're scaled proportionally.
90+
91+
### Subject/Test Split
92+
93+
When a node contains both subjects and direct tests, the `subjects_weight` field determines the split:
94+
95+
```
96+
subjects_weight = 70
97+
├── Subjects group gets factor = 0.70
98+
└── Tests group gets factor = 0.30
99+
```
100+
101+
Each group's weights are balanced independently within their respective factor. This means subjects compete with subjects, and tests compete with tests, with the `subjects_weight` controlling the ratio between the two groups.
102+
103+
If a node has only subjects or only tests, the factor is 1.0 (no split needed).
104+
105+
---
106+
107+
## Score Calculation
108+
109+
After the entire tree is processed, `ResultTree.calculate_final_score()` aggregates scores bottom-up through the `RootResultNode`:
110+
111+
```
112+
final_score = base_score + bonus_contribution - penalty_deduction
113+
```
114+
115+
At each level, the score is a weighted average:
116+
117+
```python
118+
# For a subject or category:
119+
score = sum(child.score * child.weight / 100 for child in children)
120+
```
121+
122+
The `RootResultNode.calculate_score()` combines the three categories:
123+
- **Base**: The primary score (0–100 scale, weighted by `base.weight`)
124+
- **Bonus**: Added on top (e.g., weight=10 means up to 10 extra points)
125+
- **Penalty**: Subtracted (e.g., weight=-20 means up to 20 points deducted)
126+
127+
---
128+
129+
## File Targeting
130+
131+
The `get_file_target()` method resolves which submission files a test should receive:
132+
133+
```python
134+
def get_file_target(self, test_node: TestNode):
135+
if not test_node.file_target or not self.__submission_files:
136+
return None
137+
target_files = []
138+
for file_name in self.__submission_files:
139+
if file_name in test_node.file_target:
140+
target_files.append(self.__submission_files[file_name])
141+
return target_files
142+
```
143+
144+
This allows tests to operate on specific files (e.g., a `has_tag` test targeting only `index.html`) rather than the entire submission. If no `file_target` is specified, the test receives `None` and is expected to work with the sandbox or handle files internally.
145+
146+
---
147+
148+
## AI Executor Integration
149+
150+
Some templates use an `AiExecutor` for batch AI-powered test execution. After the tree traversal completes, the engine checks if the first test has an attached executor and calls `executor.stop()` to flush any pending AI batch requests:
151+
152+
```python
153+
first_test = self.__find_first_test(criteria_tree.base)
154+
if first_test and hasattr(first_test, "test_function"):
155+
test_func = first_test.test_function
156+
if hasattr(test_func, "executor") and test_func.executor:
157+
test_func.executor.stop()
158+
```
159+
160+
This is a post-processing step that ensures all AI-batched results are resolved before the `ResultTree` is returned.
161+
162+
---
163+
164+
## Result Tree Structure
165+
166+
The output `ResultTree` mirrors the `CriteriaTree` but with actual scores:
167+
168+
```
169+
ResultTree
170+
├── root: RootResultNode
171+
│ ├── base: CategoryResultNode
172+
│ │ ├── subjects: [SubjectResultNode, ...]
173+
│ │ │ ├── tests: [TestResultNode, ...]
174+
│ │ │ └── subjects: [SubjectResultNode, ...]
175+
│ │ └── tests: [TestResultNode, ...]
176+
│ ├── bonus: CategoryResultNode (optional)
177+
│ └── penalty: CategoryResultNode (optional)
178+
├── template_name: str (optional)
179+
└── metadata: dict
180+
```
181+
182+
Each `TestResultNode` contains:
183+
- `name` — Test identifier
184+
- `score` — Achieved score (0–100)
185+
- `report` — Human-readable result explanation
186+
- `parameters` — Test parameters used
187+
- `test_node` — Reference back to the original `TestNode`
188+
189+
The `ResultTree` also provides utility methods:
190+
- `get_all_test_results()` — Flat list of all test result nodes
191+
- `get_failed_tests()` — Tests with score < 100
192+
- `get_passed_tests()` — Tests with score = 100
193+
- `to_dict()` — Full serialization with summary statistics
194+
195+
---
196+
197+
## Source Files
198+
199+
| File | Contents |
200+
|------|----------|
201+
| `autograder/services/grader_service.py` | `GraderService` — tree traversal, test execution, weight balancing |
202+
| `autograder/models/result_tree.py` | `ResultTree`, `RootResultNode`, `CategoryResultNode`, `SubjectResultNode`, `TestResultNode` |
203+
| `autograder/models/criteria_tree.py` | `CriteriaTree`, `CategoryNode`, `SubjectNode`, `TestNode` |
204+
| `autograder/models/dataclass/grade_step_result.py` | `GradeStepResult` — wrapper for final score + result tree |
205+
| `autograder/models/dataclass/test_result.py` | `TestResult` — individual test execution output |
206+
| `autograder/services/command_resolver.py` | `CommandResolver` — multi-language command resolution |

‎docs/index.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,12 +33,26 @@ Welcome to the Autograder documentation. This index organizes all available docu
3333
|----------|-------------|
3434
| [Deliberate Code Execution](features/deliberate_code_execution.md) | Execute code without grading (DCE feature) |
3535
| [Focus Feature](features/focus_feature.md) | Focus-based feedback highlighting high-impact improvements |
36+
| [Grading Engine](features/grading_engine.md) | Deep dive on the tree-based grading engine (traversal, weights, scoring) |
3637
| [Setup Config](features/setup_config_feature.md) | Preflight checks, required files, and setup commands |
3738
| [Setup Config Quick Start](guides/SETUP_CONFIG_QUICK_START.md) | Quick guide for setup configuration |
3839
| [Multi-Language Support](features/multi_language_support.md) | Python, Java, Node.js, and C++ support |
3940
| [Pipeline Execution Tracking](architecture/pipeline_execution_tracking.md) | Step-by-step pipeline execution details |
4041
| [GitHub Action](guides/github_module.md) | GitHub Classroom integration |
4142

43+
## Pipeline
44+
45+
| Document | Description |
46+
|----------|-------------|
47+
| [Pipeline Overview](pipeline/README.md) | Architecture, PipelineExecution, assembly rules, step dependency table |
48+
| [Load Template Step](pipeline/01-load-template.md) | Loads the grading template with test functions |
49+
| [Build Tree Step](pipeline/02-build-tree.md) | Constructs the CriteriaTree from JSON config |
50+
| [Pre-Flight Step](pipeline/03-pre-flight.md) | Validates files, creates sandbox, runs setup commands |
51+
| [Grade Step](pipeline/04-grade.md) | Executes tests and produces the scored ResultTree |
52+
| [Focus Step](pipeline/05-focus.md) | Ranks tests by impact on the final score |
53+
| [Feedback Step](pipeline/06-feedback.md) | Generates student-facing feedback reports |
54+
| [Export Step](pipeline/07-export.md) | Sends scores to external systems |
55+
4256
## Architecture & Data Structures
4357

4458
| Document | Description |

‎docs/pipeline/01-load-template.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Step 1: Load Template
2+
3+
## Purpose
4+
5+
The Load Template step is the entry point of the pipeline. It loads the grading template that defines which test functions are available for the assignment. Without a template, no tests can be matched or executed.
6+
7+
## How It Works
8+
9+
The step uses the `TemplateLibraryService` singleton to load a template by name. There are two paths:
10+
11+
1. **Built-in template** — Loaded from the template registry by identifier (e.g., `"input_output"`, `"web_dev"`, `"api_testing"`).
12+
2. **Custom template** — User-provided template object. This path is planned but not yet implemented; it will require sandboxed loading for security.
13+
14+
The loaded `Template` object is stored in the step result's `data` field, making it available to all subsequent steps.
15+
16+
## Dependencies
17+
18+
None. This is the first step in the pipeline.
19+
20+
## Input
21+
22+
| Source | Data |
23+
|--------|------|
24+
| Constructor | `template_name: str` — identifier of the template to load |
25+
| Constructor | `custom_template` (optional) — user-provided template object |
26+
27+
## Output
28+
29+
| Field | Type | Description |
30+
|-------|------|-------------|
31+
| `data` | `Template` | The loaded template instance with all its test functions |
32+
| `status` | `StepStatus.SUCCESS` | On successful load |
33+
34+
## What a Template Contains
35+
36+
A `Template` provides:
37+
- **`template_name`** — Display name (e.g., "Input/Output Testing")
38+
- **`template_description`** — What the template is designed for
39+
- **`requires_sandbox`** — Whether test execution needs an isolated container (e.g., `True` for `input_output`, `False` for `web_dev`)
40+
- **`tests`** — Dictionary of `TestFunction` instances keyed by name
41+
- **`get_test(name)`** — Retrieves a specific test function by name
42+
43+
Available built-in templates:
44+
45+
| Identifier | Name | Requires Sandbox | Use Case |
46+
|------------|------|-----------------|----------|
47+
| `input_output` | Input/Output Testing | Yes | Command-line programs with stdin/stdout |
48+
| `web_dev` | Web Development | No | HTML/CSS/JS file validation |
49+
| `api_testing` | API Testing | Yes | HTTP endpoint validation |
50+
51+
## Failure Scenarios
52+
53+
- Template name not found in the registry → `StepStatus.FAIL` with error message listing available templates.
54+
- Custom template loading attempted → `NotImplementedError` (feature not yet implemented).
55+
56+
## Source
57+
58+
`autograder/steps/load_template_step.py` → `TemplateLoaderStep`
59+
60+
`autograder/services/template_library_service.py` → `TemplateLibraryService`

‎docs/pipeline/02-build-tree.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Step 2: Build Tree
2+
3+
## Purpose
4+
5+
The Build Tree step transforms the raw JSON grading criteria into a structured `CriteriaTree` — the hierarchical rubric that drives the entire grading process. It also validates the configuration and links each test entry to its actual `TestFunction` from the loaded template.
6+
7+
## How It Works
8+
9+
1. **Parse configuration** — The raw JSON dictionary is validated and parsed into a `CriteriaConfig` Pydantic model, which enforces structural rules (e.g., a category must have either tests or subjects, subjects with both must declare `subjects_weight`).
10+
2. **Build tree nodes** — The `CriteriaTreeService` recursively walks the config and creates `CategoryNode`, `SubjectNode`, and `TestNode` objects.
11+
3. **Match test functions** — For each test entry in the config, the service looks up the corresponding `TestFunction` from the template loaded in the previous step. If a test name doesn't match any function in the template, the step fails with a `ValueError`.
12+
4. **Balance weights** — Sibling subject weights are normalized to sum to 100 if they don't already.
13+
14+
The resulting `CriteriaTree` is stored in the step result and used by the Grade step to execute tests.
15+
16+
## Dependencies
17+
18+
| Step | What It Needs |
19+
|------|---------------|
20+
| **Load Template** | The `Template` object to look up test functions by name |
21+
22+
## Input
23+
24+
| Source | Data |
25+
|--------|------|
26+
| Constructor | `criteria_json: dict` — raw criteria configuration dictionary |
27+
| Pipeline | `StepName.LOAD_TEMPLATE` → `Template` |
28+
29+
## Output
30+
31+
| Field | Type | Description |
32+
|-------|------|-------------|
33+
| `data` | `CriteriaTree` | The fully built criteria tree with embedded test functions |
34+
| `status` | `StepStatus.SUCCESS` | On successful build |
35+
36+
## CriteriaTree Structure
37+
38+
```
39+
CriteriaTree
40+
├── base: CategoryNode (required)
41+
│ ├── subjects: [SubjectNode, ...]
42+
│ │ ├── tests: [TestNode, ...]
43+
│ │ └── subjects: [SubjectNode, ...] (nested)
44+
│ └── tests: [TestNode, ...]
45+
├── bonus: CategoryNode (optional)
46+
└── penalty: CategoryNode (optional)
47+
```
48+
49+
Each `TestNode` holds:
50+
- `name` — Test identifier
51+
- `test_function` — Reference to the actual `TestFunction` instance from the template
52+
- `parameters` — Keyword arguments to pass to the test function at execution time
53+
- `file_target` — Which submission files this test operates on (optional)
54+
55+
## Weight Balancing
56+
57+
If sibling subjects don't sum to 100, the service scales them proportionally:
58+
59+
```
60+
Subject A: weight=30, Subject B: weight=70 → kept as-is (sum=100)
61+
Subject A: weight=3, Subject B: weight=7 → scaled to 30 and 70
62+
```
63+
64+
When a category or subject contains both `subjects` and `tests`, the `subjects_weight` field determines how the total weight is split between the two groups. For example, `subjects_weight=70` means 70% of the score comes from subjects and 30% from direct tests.
65+
66+
## Failure Scenarios
67+
68+
- Invalid criteria JSON structure (missing required fields, wrong types) → Pydantic validation error.
69+
- Test name not found in the template → `ValueError: Couldn't find test {name}`.
70+
- Category has neither tests nor subjects → validation error.
71+
72+
## Source
73+
74+
`autograder/steps/build_tree_step.py` → `BuildTreeStep`
75+
76+
`autograder/services/criteria_tree_service.py` → `CriteriaTreeService`
77+
78+
`autograder/models/config/` → `CriteriaConfig`, `CategoryConfig`, `SubjectConfig`

0 commit comments

Comments
 (0)