Skip to content

Commit a49921b

Browse files
junzisjunzis
committed
docs(pipe): refresh PipeDecoder guide for v3.3 validation stack
The previous guide pre-dated the phantom-rejection, bootstrap cluster, and motion-check work; it also had a typo in the usage example. Rewrites the overview, adds new constructor options (max_speed_kt, motion_margin_km), documents the four cross-checks with a stats counter table, and adds a BDS coverage matrix so readers can see which registers get a dedicated check, which participate in 5,0/6,0 disambiguation scoring, and which are only scrubbed indirectly when a DF20 altitude mismatch fires.
1 parent 5692fdc commit a49921b

1 file changed

Lines changed: 125 additions & 35 deletions

File tree

‎docs/pipe.md‎

Lines changed: 125 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,21 @@
11
# PipeDecoder
22

3-
`PipeDecoder` is pyModeS' new stateful streaming decoder. It processes
4-
one message at a time and maintains per-ICAO state across calls, so:
5-
6-
- **CPR pairs** — an even/odd pair of airborne position frames
7-
produced ≤10 seconds apart is resolved to absolute lat/lon without
8-
needing an external reference.
9-
- **BDS 5,0/6,0 disambiguation** — when a Comm-B message plausibly
10-
matches both registers, prior observations of groundspeed / track /
11-
heading are used to pick the better candidate (Phase 3 scoring).
12-
- **DF20/21 ICAO verification** — DF17/18 messages populate a
13-
trusted-ICAO set; subsequent DF20/21 messages whose CRC-derived
14-
ICAO matches a trusted one are flagged with `icao_verified=True`.
3+
`PipeDecoder` is pyModeS' stateful streaming decoder. It processes one
4+
message at a time and maintains per-ICAO state across calls, so:
5+
6+
- **CPR pair resolution** — an even/odd pair of airborne-position
7+
frames produced ≤ `pair_window` seconds apart is resolved to absolute
8+
lat/lon without needing an external reference.
9+
- **BDS 5,0 / 6,0 disambiguation** — when a Comm-B message plausibly
10+
matches both registers, prior observations of groundspeed, track,
11+
and heading score the candidates and pick the better fit.
12+
- **DF20/21 ICAO verification** — CRC-valid DF17/18 frames populate a
13+
trusted-ICAO set; a later DF20/21 whose CRC-derived ICAO matches one
14+
in the set is flagged with `icao_verified=True`.
15+
- **Phantom rejection** — CRC alone doesn't catch every FRUIT frame
16+
that happens to land with a plausible ICAO. Several cross-checks
17+
layered on top of CRC use per-ICAO anchors to drop phantoms before
18+
they pollute state (see [Validation](#validation) below).
1519

1620
## Basic usage
1721

@@ -22,7 +26,7 @@ pipe = PipeDecoder(surface_ref="EHAM", pair_window=10.0, eviction_ttl=300.0)
2226

2327
for raw_msg, timestamp in stream:
2428
result = pipe.decode(raw_msg, timestamp=timestamp)
25-
pirnt(result)
29+
print(result)
2630
...
2731

2832
print(pipe.stats) # counters
@@ -32,37 +36,116 @@ pipe.reset() # clear all state
3236
## Constructor options
3337

3438
- `surface_ref` — airport code or `(lat, lon)` for surface CPR
35-
resolution (single-message path). Required for surface positions
36-
to return lat/lon. Not needed for airborne.
37-
- `full_dict` — if `True`, every decoded result is populated with
38-
every key from the canonical schema (missing fields = `None`).
39+
resolution (single-message path). Required for surface positions to
40+
return lat/lon. Not needed for airborne.
41+
- `full_dict` — if `True`, every decoded result is populated with every
42+
key from the canonical schema (missing fields = `None`).
3943
- `pair_window` — maximum age gap (seconds) between an even and odd
40-
CPR frame for them to count as a pair. Default 10 seconds.
44+
CPR frame for them to count as a pair. Default `10.0`.
4145
- `eviction_ttl` — per-ICAO state and pending CPR frames older than
4246
this are dropped lazily at the start of the next `decode()` call
43-
with a timestamp. Default 300 seconds (5 minutes).
47+
with a timestamp. Default `300.0` (5 minutes).
48+
- `max_speed_kt` — ceiling for the per-ICAO motion check (see
49+
[Validation](#validation)). Default `1500` — ~2× typical airliner
50+
cruise; loose enough to accept fast business jets and wind-boosted
51+
ground speeds, tight enough that a phantom hundreds of km away can't
52+
masquerade as a continuation of the real track.
53+
- `motion_margin_km` — slack added to the motion envelope to absorb
54+
CPR quantisation + clock jitter. Default `2.0` km.
4455

4556
## State lifecycle
4657

47-
Per-ICAO state is built incrementally from tracked fields in
48-
decoded results:
58+
Per-ICAO state is built incrementally from tracked fields in decoded
59+
results:
4960

5061
- BDS 0,9 velocity → `groundspeed`, `track`, `heading`
5162
- BDS 0,9 sub 3/4 → `airspeed` + `airspeed_type` routes to `ias` or `tas`
5263
- BDS 5,0 → `groundspeed`, `track`, `tas`
5364
- BDS 6,0 → `heading`, `ias`, `mach`
5465

55-
These values are then passed as `known=` to subsequent decodes of
56-
the same ICAO, enabling Phase 3 disambiguation.
57-
58-
State entries carry a `_last_seen` timestamp. On each `decode()`
59-
call with a timestamp, entries older than `eviction_ttl` are dropped.
66+
These values are then passed as `known=` to subsequent decodes of the
67+
same ICAO, enabling BDS 5,0 / 6,0 disambiguation. When `groundspeed`
68+
and `altitude` are known but `ias`, `mach`, or `tas` aren't yet
69+
observed, they're derived via the ISA atmosphere model so BDS 6,0
70+
scoring still has a reference field.
71+
72+
State entries carry a `_last_seen` timestamp. On each `decode()` call
73+
with a timestamp, entries older than `eviction_ttl` are dropped.
74+
75+
## Validation
76+
77+
On top of CRC, `PipeDecoder` runs four plausibility cross-checks
78+
against per-ICAO anchors updated only from CRC-valid frames that
79+
passed their own check. A frame that fails is kept (header fields
80+
intact) so the caller can see it, but its position / velocity fields
81+
are scrubbed, the anchor is not updated, and state is not mutated.
82+
83+
| Check | Frames | Against | Stats counter |
84+
|-------|--------|---------|---------------|
85+
| Altitude | DF20 (header AC-code) | ADS-B altitude anchor | `altitude_mismatch` |
86+
| Altitude | DF17/18 BDS 0,5 | ADS-B altitude anchor | `altitude_mismatch` |
87+
| Velocity | DF17/18 TC=19 | ADS-B velocity anchor; abs VR > 10 000 fpm | `velocity_mismatch` |
88+
| Velocity | DF20/21 BDS 5,0 | ADS-B velocity anchor | `velocity_mismatch` |
89+
| Heading | DF20/21 BDS 6,0 | ADS-B track anchor (wider tol.) | `velocity_mismatch` |
90+
91+
### Position bootstrap
92+
93+
The first few resolved positions for a new ICAO don't yet have an
94+
anchor to cross-check against — so they're held back. The decoder
95+
collects up to 5 candidate positions into `_bootstrap`, runs a cluster
96+
analysis to pick a consistent seed, and only then promotes them into
97+
the rolling position history used for the motion check. While held,
98+
`latitude` / `longitude` are suppressed in the returned result; once
99+
the cluster locks, **both halves** of each resolved CPR pair are
100+
retro-filled, so batch callers who keep their result list around see
101+
the positions on their early samples.
102+
103+
If no consistent cluster forms, the bootstrap buffer resets and
104+
candidates start over — counted as `bootstrap_reset`.
105+
106+
### Motion check
107+
108+
Post-bootstrap, each candidate position is compared against the most
109+
recent accepted anchor. The allowed distance is
110+
`max_speed_kt * dt + motion_margin_km`. Positions that exceed it are
111+
rejected (`position_rejected`) — the anchor still rotates through the
112+
history regardless of accept/reject, so real tracks eventually
113+
out-vote lingering phantoms.
114+
115+
### BDS coverage
116+
117+
What happens per register when it's offered to `PipeDecoder`:
118+
119+
| BDS | Dedicated check | Disambiguation scoring | Indirect scrub on DF20 altitude mismatch |
120+
|-----|-----------------|------------------------|------------------------------------------|
121+
| 0,5 airborne position | altitude | — | — |
122+
| 0,9 velocity (TC=19) | gs / track / abs VR | — | — |
123+
| 1,0, 1,7 data-link capability | — | — | yes (`supported_bds`) |
124+
| 2,0 aircraft identification | — | — | yes (`callsign`) |
125+
| 4,0 selected vertical intention | — | — | yes (MCP/FMS alt, VNAV, etc.) |
126+
| 4,4, 4,5 meteorological | — | — | yes (wind, temperature, turbulence, …) |
127+
| 5,0 track & turn | gs / true_track | yes | yes |
128+
| 6,0 heading & speed | magnetic_heading | yes | yes |
129+
130+
- **Dedicated check** — a per-ICAO cross-check runs on this register;
131+
failing frames are scrubbed and counted in
132+
`altitude_mismatch` / `velocity_mismatch`.
133+
- **Disambiguation scoring** — when the raw payload plausibly matches
134+
multiple registers, prior state scores candidates via
135+
`_SCORE_FIELDS_BDS50` / `_BDS60` (implemented only for 5,0 and 6,0).
136+
- **Indirect scrub** — when a DF20 fails the altitude check, any
137+
inferred fields from the listed registers are wiped regardless of
138+
whether *this* register would have caught it.
139+
140+
A phantom DF20 whose inferred payload lands in 1,0 / 2,0 / 4,0 / 4,4 /
141+
4,5 with a plausible AC-code altitude and no prior anchor passes
142+
silently — the dedicated checks don't cover those registers.
60143

61144
## Thread safety
62145

63-
`PipeDecoder` is **not thread-safe** by default. Every `decode()` call mutates
64-
internal state without locking. Wrap the instance with a lock if
65-
multiple threads feed it concurrently:
146+
`PipeDecoder` is **not thread-safe** by default. Every `decode()` call
147+
mutates internal state without locking. Wrap the instance with a lock
148+
if multiple threads feed it concurrently:
66149

67150
```python
68151
import threading
@@ -76,9 +159,8 @@ def decode_one(msg: str, ts: float):
76159
return pipe.decode(msg, timestamp=ts)
77160
```
78161

79-
However, in most use cases, for single-producer pipelines (one reader thread
80-
draining a socket) no locking is needed, just don't share the decoder across
81-
threads.
162+
For single-producer pipelines (one reader thread draining a socket) no
163+
locking is needed — just don't share the decoder across threads.
82164

83165
## Stats
84166

@@ -88,6 +170,14 @@ threads.
88170
- `decoded` — messages that parsed successfully
89171
- `crc_fail` — messages whose decoded `crc_valid` was `False`
90172
- `pending_pairs` — CPR frames currently held waiting for their pair
91-
92-
The trusted ICAO set, per-ICAO state, and pending CPR frames are all
93-
cleared by `reset()`.
173+
- `altitude_mismatch` — frames rejected by the altitude cross-check
174+
- `velocity_mismatch` — frames rejected by a velocity / heading check
175+
- `position_rejected` — post-bootstrap positions rejected by the
176+
motion check
177+
- `bootstrap_held` — candidate positions added to the bootstrap buffer
178+
(some may end up promoted, others discarded on reset)
179+
- `bootstrap_reset` — bootstrap buffers that failed to cluster and
180+
restarted
181+
182+
The trusted ICAO set, per-ICAO state, pending CPR frames, anchors,
183+
bootstrap buffers, and position history are all cleared by `reset()`.

0 commit comments

Comments
 (0)