Explainable AI (XAI) Toolkit Expansion for donkeycar

[YashTandon05]

Explainable AI (XAI) Toolkit

Overview

Stock Donkeycar's deep-learning autopilot gives you a steering/throttle number and nothing else — you can't tell how sure the model was, or why it steered the way it did, without guessing. This fork adds an explainability and uncertainty toolkit on top of the existing pilot, built from four independent pieces:

1. Confidence (MC-Dropout) — "do the model's internal sub-networks agree with each other?" A live 0-100% signal.
2. Novelty / out-of-distribution (OOD) detection — "does this camera frame look like anything the model was trained on?" Also live, 0-100%.
3. TTA stability — "if I nudge the lighting slightly, does the model's answer wobble?" Also live, 0-100%.
4. Confidence-aware throttle scaling — an optional part that slows the car down (and can bring it to a full stop) when any of the three signals above indicates trouble, so uncertainty isn't just a number nobody acts on.

Alongside the live signals, there's an offline analysis viewer: point it at a recorded drive and a model, and it produces a frame-by-frame, browser-based playback showing where the model was looking and why it was or wasn't confident, using four different visual attribution techniques (Grad-CAM, Grad-CAM++, Integrated Gradients, and pixel-gradient saliency).

All three live signals — and therefore the throttle scaling built on top of them — share one dependency: a calibration file (<model>.calib.json) that turns each raw internal number into a 0-100% score that's meaningful for your specific model and your specific training data. Without it there is no percentage to display or act on, so the dashboard panels stay hidden and throttle scaling passes throttle through untouched. The offline viewer is the exception: it runs without a calibration too, it just falls back to plotting raw variance instead of the 0-100% lines. If your model was trained with any AUGMENTATIONS turned on, calibration needs to know about that too, or the novelty signal will misfire — covered in detail in §2.8.

This document's Section 1 is a usage walkthrough: what to turn on, in what order, and what you'll see. Section 2 is a technical explanation of how each signal actually works.

---

1. Usage

1.1 The pipeline, at a glance

record data  →  train  →  calibrate  →  drive live (dashboard)
                                     ↘  analyse offline (viewer)

Calibration sits in the middle because every percentage downstream — the live dashboard panels, throttle scaling, and the offline viewer's confidence/novelty/stability graph — depends on it existing. (The offline viewer still runs without one; it just plots raw variance instead.)

1.2 Recording and training

No changes here — record a tub and train exactly as stock Donkeycar describes (donkey train). The XAI toolkit only reads the model file afterwards; it doesn't change how training itself works. One thing to know: MC-Dropout confidence (and, by extension, most of this toolkit) currently only supports the linear model type, since it relies on the Dropout layers already present in that architecture staying active at inference time — see §2.2 for why.

Recording an autopilot drive (for offline review, not for training). By default Donkeycar only records frames while you're driving in 'user' (manual) mode — switching into autopilot stops recording, even if the recording toggle is on. If you want to feed an actual autonomous run into this toolkit's offline viewer (§1.6), for example to see what the model was looking at right before a near-miss, set the stock Donkeycar option in myconfig.py:

RECORD_DURING_AI = True

This is stock Donkeycar behaviour and this toolkit does not change it: autopilot frames go into the same tub as your manual frames, so the standard warning still applies — don't train on a tub recorded this way without filtering it first. The XAI columns (pilot/confidence, pilot/novelty, …) are written alongside every frame, but only autopilot frames actually carry values; manual frames leave them empty, because the signals run under Donkeycar's run_pilot condition. The offline tool handles that mix explicitly rather than guessing — see the note on unscored frames in §1.6.

1.3 Calibration

Calibration replays a tub of "known good" driving through the model once and records how the raw signals behave on data the model actually learned from — this becomes the yardstick every live percentage is measured against. Nothing lights up on the dashboard until this has run once for a given model.

Option A — automatic, at the end of training. In myconfig.py:

XAI_CONFIDENCE_AUTO_CALIBRATE = True

This adds one extra replay pass over your training tubs right after donkey train finishes, and saves <model>.calib.json next to the model automatically. Off by default because it adds time to every training run.

Option B — manual, from the command line (needed if you trained before turning on auto-calibrate, or want to recalibrate later):

python -m donkeycar.parts.mc_calibrate --tub data/mytub --model models/mypilot.h5

usage: mc_calibrate [-h] --tub TUB [TUB ...] --model MODEL [--config CONFIG]
                     [--limit LIMIT] [--passes PASSES] [--alpha ALPHA]
                     [--out OUT] [--augment | --no-augment]

  --tub TUB [TUB ...]  one or more tub paths to replay
  --model MODEL        path to the .h5 model
  --config CONFIG      path to config.py (defaults to ./config.py, falling
                        back to the bundled complete template)
  --limit LIMIT        only use the first N frames
  --passes PASSES      override XAI_CONFIDENCE_PASSES
  --alpha ALPHA        override XAI_CONFIDENCE_ALPHA
  --out OUT            output calibration path (default <model>.calib.json)
  --augment            widen the novelty baselines with augmented frames
  --no-augment         fit the novelty baselines on clean frames only

Point --tub at the same data the model was actually trained on — calibration is meant to answer "what does normal look like to this model," and calibrating against a different drive (e.g. a test/inference tub) silently defeats the whole signal. The offline GUI launcher (§1.6) tries to auto-detect this for you from Donkeycar's own training record; the CLI here has no such lookup, so it's on you to point --tub correctly.

One calibration run covers all three signals at once — confidence, novelty and TTA stability are always all written into the file, whether or not you currently have them switched on for driving. So turning XAI_TTA_ENABLED (or any other signal) on later just works; you don't need to recalibrate to "add" it.

When to recalibrate:

• After retraining or swapping the model (new weights mean the old numbers no longer describe it).
• After changing AUGMENTATIONS in a meaningful way (see §2.8).
• After changing an XAI_*_INTERVAL value. Calibration replays at whatever cadence the interval was set to at calibration time (see §2.10); tightening or relaxing an interval afterward means the live update rhythm no longer matches what the anchors were fit against.

On config loading: all three entry points (mc_calibrate, gradcam_uncertainty, the GUI launcher) resolve --config the same way — your ./config.py if you're in a car directory, otherwise the bundled template — and in every case they also apply the sibling myconfig.py on top. That last part matters: Donkeycar's own load_config() finds myconfig.py by string-replacing "config.py" in the path, which silently does nothing when the base file is named anything else (such as the bundled cfg_complete.py), so your real overrides would be dropped and the whole calibration built against template defaults.

On image size: calibration and offline analysis both read the input size directly from the model file, so a model trained at a different resolution than your current config still works — you'll just see a warning saying which size was used, and IMAGE_W/IMAGE_H are taken from the model. If that warning looks wrong, point --config at the config.py the model was actually trained with. Because the model file is treated as the authority there, resolution is deliberately left out of the preprocessing-sidecar comparison described in §2.9 — the crop geometry and transformation list, which a model file cannot self-describe, are still checked strictly.

1.4 Driving live with the signals on

Each signal is an independent switch in myconfig.py — turn on only what you want:

XAI_CONFIDENCE_ENABLED = True   # MC-Dropout confidence
XAI_NOVELTY_ENABLED = True      # feature-space novelty / OOD
XAI_TTA_ENABLED = True          # test-time-augmentation stability

Once enabled and calibrated, driving with the web dashboard open shows a panel per signal:

Panel heading (as shown)	Meaning	Green (safe)	Amber	Red
"Do the model's 'sub-networks' agree?"	Confidence	≥ 65%	25-65%	< 25%
"Does this look familiar?"	Novelty (OOD)	≤ 25%	25-65%	> 65%
"Is the answer stable?"	TTA stability	≥ 65%	25-65%	< 25%

(Novelty's color scale is intentionally flipped from the other two — a high number there means "unfamiliar," which is bad, whereas a high number for confidence/stability is good.) Each panel has a "What does this mean? ▾" toggle with a plain-English explanation, and the confidence panel is explicit that it's "[r]elative to this model's own baseline, not a calibrated probability" — see §2.3 for what that means concretely.

If a signal is enabled but not yet calibrated, its panel simply never appears — instead, on connecting to the dashboard you'll see a dismissible amber banner headed "Calibration warnings: these signals won't show a percentage, or may be reading against the wrong baseline," listing exactly which signal(s) are affected and the exact mc_calibrate command to fix each. Go run calibration (§1.3) and reconnect.

Panels for calibrated signals appear as soon as the dashboard connects, including on a reload or a second browser: the server sends each new client the current value up front rather than waiting for the number to change.

Pi / performance tuning: each signal has its own XAI_*_INTERVAL (seconds; 0.0 = every frame, no rate-limiting at all). Out of the box these are already relaxed rather than 0.0 — confidence and novelty default to 0.3 (~3 updates/sec), TTA to 0.5 (~2 updates/sec) since it's the priciest signal (it runs XAI_TTA_SAMPLES extra forward passes per update). Confidence never costs steering responsiveness either way — the car still steers on a cheap single-pass inference every frame regardless of its interval, only the confidence number updates less often. Novelty and TTA are pure observers, so a held frame between updates costs zero forward passes. Raise any of these further if a Raspberry Pi is still struggling to keep up with DRIVE_LOOP_HZ — TTA is the first knob to turn. See §1.5 for how a longer interval trades off against throttle-scaling reaction time.

1.5 Confidence-aware throttle scaling

An optional fourth switch that actually acts on the three signals above instead of just displaying them:

XAI_THROTTLE_SCALING_ENABLED = True

With this on, whichever of confidence/novelty/stability is enabled gets combined: if any signal crosses its "reduced" threshold, throttle is scaled down (linearly, down to XAI_THROTTLE_MIN_SCALE, default 0.4 = never below 40% of the commanded throttle from a single signal); if any signal is past its "critical" threshold continuously for XAI_THROTTLE_STOP_DURATION seconds (default 1.0), throttle is forced to zero. It only ever touches throttle — steering is never modified. Turning this on without enabling at least one of the three signals just logs a warning and does nothing (there's nothing to scale on).

It runs after AI_LAUNCH_*, on purpose. Donkeycar's stock AI-launch boost (AI_LAUNCH_DURATION, AI_LAUNCH_THROTTLE) replaces pilot/throttle with a fixed value rather than modulating it, so throttle scaling is wired in after the launch boost in the vehicle loop, not before — otherwise a critical signal (including the forced stop) would have no effect for the first AI_LAUNCH_DURATION seconds after switching into autopilot, exactly the window the car is least predictable in. With this ordering, whichever value pilot/throttle currently holds — launch-boosted or not — still gets scaled or stopped if a signal says so.

Reaction-time tradeoff with the interval knobs (§1.4): novelty and TTA are only ever as fresh as their XAI_*_INTERVAL — a sudden problem won't be seen until the next scheduled update, up to that many seconds later, and XAI_THROTTLE_STOP_DURATION then still needs to see it stay critical continuously before forcing a stop. At the defaults, that's roughly up to 0.3 + 1.0 = 1.3s for novelty-triggered or 0.5 + 1.0 = 1.5s for TTA-triggered stops, worst case. That's fine if you're using these signals for dashboard/monitoring, but if you're relying on XAI_THROTTLE_SCALING_ENABLED as a real-time safety net, consider lowering XAI_NOVELTY_INTERVAL/XAI_TTA_INTERVAL back toward 0.0 so the car reacts faster, at the cost of more compute per frame.

1.6 Offline analysis — the GUI launcher (recommended)

The easiest way to review a drive is the browser-based launcher, which wraps the CLI tool below so you never need to remember its flags. From your car's directory (donkey createcar installs this script for you):

python run_gradcam_analysis.py

This starts a local web server and prints a URL to open. You'll see a "Run Explainability Analysis" form:

Field	What to put
Tub path	e.g. data/mytub (autocompletes from tubs found under data/)
Model path	e.g. models/mypilot.h5 (autocompletes from models/) — once selected, a hint below the field shows what pipeline this model was actually trained with (e.g. Trained with: CROP, CANNY), read straight from the model's own .preprocessing.json sidecar (§2.9)
Transformations override (optional)	leave blank to use this launcher's own config. By default, the run is refused if this launcher's TRANSFORMATIONS/POST_TRANSFORMATIONS don't match what the selected model was actually trained with — since a mismatch would silently compute every heat map on pixels the model never saw. Only fill this in to deliberately analyse a model under a different pipeline than it trained with (an intentional what-if); doing so replaces POST_TRANSFORMATIONS for this run only and skips that check, since you're now doing it on purpose
Frames to analyse	top-K most uncertain (default, analyse a fixed number of the hardest frames), percentile ≥ (analyse the worst N% of the drive), or all frames (only for short drives)
Limit / range of frames considered	optional — restrict to e.g. 500 or 1000-2000 instead of the whole tub
Auto-calibrate this model first if it has no calibration yet	auto-checked once you pick a model that has no saved calibration yet, unchecked if it already does — saves a manual mc_calibrate step. Override it by hand any time; your choice is never clobbered afterward
Calibration tub (training data)	only shown while auto-calibrate is checked. Auto-fills from Donkeycar's own training record for that model if it can find one; type over it if you want to force a specific tub
Export every camera frame	check this if you want the output folder to be self-contained (e.g. to copy off a remote GPU box) — otherwise only the analysed frames' images are saved

Click Run Analysis and a progress readout tracks the current stage (variance → novelty → Grad-CAM). If auto-calibration had to fall back to using the analysis tub itself (because no training-tub record could be found), you'll see an amber warning and the page will not auto-continue — you have to click Continue to viewer explicitly, so a possibly-wrong calibration baseline can't slip by unnoticed. Otherwise it jumps to the viewer automatically when done.

Unscored frames. A tub recorded with RECORD_DURING_AI (§1.2) contains manual frames that carry no XAI values, because the signals only run while the autopilot does. Where a tub already has logged values the tool reuses them rather than recomputing, and frames without one are treated as unknown, not as zero — they stay in the timeline (drawn as a gap in the graph, shown as variance n/a), but they're excluded from "top-K most uncertain" and "percentile ≥" selection, and a log line reports how many were skipped. Zero would mean lowest possible variance, i.e. maximum confidence, which would rank exactly the frames nobody measured as the most certain in the drive. If you want those frames analysed anyway, use all frames, which ignores the ranking entirely.

The viewer, once open, shows the current frame plus a chosen overlay layer:

• Where the model was unsure (uncertainty) — the default view
• Where the model looked (Grad-CAM)
• Where the model looked, sharper (Grad-CAM++)
• Have I seen this before? (novelty)
• Exact pixels that mattered (saliency)
• Exact pixels that mattered, cleaner (integrated gradients)
• Just the camera
• What the model actually sees (transformed) — the frame after TRANSFORMATIONS/POST_TRANSFORMATIONS (crop, edge detection, colour conversion). This is the actual input every heat map above was computed against; only shown when the pipeline changes something (a no-op pipeline would make this identical to "Just the camera")

A short plain-language explanation for whichever layer is selected is always shown underneath the dropdown — no separate "advanced mode" to find it in. When a transformation changes the frame's geometry (see §2.9 — this is now the normal case for CROP), a "draw on model input" checkbox appears next to the layer dropdown: unticked (default) draws every heat map projected back onto the raw camera frame, so the scene stays recognisable; ticked draws them directly on the transformed frame instead, with no projection needed since that's the exact input the maps were computed from. Playback controls: Space/arrow keys or the play button at 2/5/10 fps, a scrub bar, and click-to-jump on the confidence graph below the image. The graph plots confidence/novelty/TTA-stability as separate, independently toggleable lines (checkboxes above the graph — whichever signals you actually calibrated/enabled), with the frames that got a full image analysis marked along the top edge. A "What do confidence / novelty / stability / variance mean?" link expands a glossary covering all of these terms in plain language, always available with one click.

1.7 Offline analysis — CLI (advanced / scripted use)

Equivalent to the launcher's form, useful for automation or a headless box:

python -m donkeycar.parts.gradcam_uncertainty --tub data/mytub --model models/mypilot.h5 --top-k 50

usage: gradcam_uncertainty [-h] --tub TUB --model MODEL [--config CONFIG]
                            [--out OUT] [--passes PASSES] [--top-k TOP_K]
                            [--percentile PERCENTILE] [--ig-steps IG_STEPS]
                            [--all] [--limit LIMIT] [--start START]
                            [--export-frames]

  --out OUT             output dir (default <tub>/gradcam_analysis)
  --passes PASSES       MC-Dropout passes (default cfg or 15)
  --top-k TOP_K         analyse the K most uncertain frames (default)
  --percentile PERCENTILE
                        instead analyse frames >= this variance percentile
  --ig-steps IG_STEPS   Integrated Gradients Riemann steps (default cfg
                        XAI_IG_STEPS or 32)
  --all                 analyse every frame (short drives only)
  --limit LIMIT         only consider N records
  --start START         skip this many records before considering any frames
  --export-frames       copy every camera frame into the output dir too

Then view the result (with or without the launcher's form):

python -m donkeycar.parts.uncertainty_viewer --analysis data/mytub/gradcam_analysis

The CLI enforces the same preprocessing-match check as the launcher (§2.9) — it refuses to run if --config's TRANSFORMATIONS/POST_TRANSFORMATIONS don't match the model's saved training pipeline — but has no equivalent of the launcher's override field, so point --config at the actual config the model was trained with rather than trying to work around a mismatch here.

1.8 Config reference

Confidence (MC-Dropout)

XAI_CONFIDENCE_ENABLED = False           # master on/off
XAI_CONFIDENCE_PASSES = 15               # stochastic forward passes per frame
XAI_CONFIDENCE_ALPHA = 0.2               # EMA smoothing of the variance
XAI_CONFIDENCE_INTERVAL = 0.3            # seconds between updates (0 = every frame, no rate-limiting)
XAI_CONFIDENCE_AUTO_CALIBRATE = False    # calibrate automatically after donkey train
XAI_CONFIDENCE_CALIBRATE_LIMIT = None    # cap frames used for auto-calibration

Novelty / OOD detection

XAI_NOVELTY_ENABLED = False
XAI_NOVELTY_ALPHA = 0.2
XAI_NOVELTY_ENCODER = 'mobilenet_v2'     # 'mobilenet_v2' | 'mobilenet'
XAI_NOVELTY_ENCODER_INPUT = 128          # square input size fed to the encoder
XAI_NOVELTY_ENCODER_ALPHA = 1.0          # encoder width multiplier (e.g. 0.35 on a Pi)
XAI_NOVELTY_INTERVAL = 0.3                # seconds between updates (0 = every frame, no rate-limiting)
# Offline heat map only -- no effect while driving:
XAI_NOVELTY_SPATIAL_INPUT = 224           # short side; grid ≈ size/32
XAI_NOVELTY_SPATIAL_MAX_FRAMES = 400      # frames sampled to fit the map's baseline
XAI_NOVELTY_SPATIAL_MAX_VECTORS = 4000    # cap on per-location vectors (bounds memory)

TTA stability

XAI_TTA_ENABLED = False
XAI_TTA_SAMPLES = 8       # M, augmented copies per frame
XAI_TTA_ALPHA = 0.2
XAI_TTA_INTERVAL = 0.5    # seconds between updates (0 = every frame, no rate-limiting)
XAI_TTA_STRENGTH = 0.2    # photometric jitter strength (0 = no augmentation)

Throttle scaling

XAI_THROTTLE_SCALING_ENABLED = False
XAI_CONFIDENCE_REDUCED_THRESHOLD = 65.0
XAI_CONFIDENCE_CRITICAL_THRESHOLD = 25.0
XAI_NOVELTY_REDUCED_THRESHOLD = 25.0
XAI_NOVELTY_CRITICAL_THRESHOLD = 65.0
XAI_TTA_REDUCED_THRESHOLD = 65.0
XAI_TTA_CRITICAL_THRESHOLD = 25.0
XAI_THROTTLE_MIN_SCALE = 0.4
XAI_THROTTLE_STOP_DURATION = 1.0   # see the reaction-time note in §1.5

Offline analysis / calibration extras

XAI_IG_STEPS = 32                        # Integrated Gradients Riemann steps
XAI_CALIBRATE_WITH_AUGMENTATIONS = True  # see §2.8
XAI_CALIBRATE_AUG_PASSES = 2
XAI_CALIBRATE_AUG_MAX_SAMPLES = 1500

1.9 Things to watch out for

• MC-Dropout needs the Keras .h5 model, not a .tflite export — TFLite conversion bakes dropout out, so there's nothing left to disagree.
• linear model type only — this is enforced, not just advised. Confidence and TTA read the model's output tensors as plain steering/throttle numbers, which is only true of the linear architecture; on a categorical, behavior, imu or inferred pilot (or with TRAIN_LOCALIZER = True) those outputs are bin vectors or a differently-shaped list. If either signal is switched on for such a model the vehicle logs a warning and runs without them, rather than feeding the car a meaningless average of a softmax. Novelty is unaffected and still works on any model type — it never touches your pilot, only its own encoder.
• A calibrated confidence % is not a probability. It's a percentile rank relative to your own training data's variance distribution — see §2.3 before treating it as "the model is X% likely to be right."
• Novelty saturates at the extremes. Wildly out-of-distribution input (camera covered, solid color, pointed at a wall) reads as ~98%, same as any other very-unfamiliar scene — it tells you that something is unfamiliar, not how unfamiliar in fine detail.
• Recalibration triggers are listed in §1.3 — it's easy to forget after retraining or changing augmentations, and the dashboard's "uncalibrated" banner is there specifically to catch that. The banner checks each signal's data, not just that a calibration file exists, so a calibration written by an older version will still correctly report which signal is missing.
• RECORD_DURING_AI (§1.2) is stock behaviour, unchanged by this toolkit: autopilot frames land in the same tub as manual ones, so the usual "don't train on this" caveat still applies. Such a tub is a mix of scored (autopilot) and unscored (manual) frames; §1.6 covers how the offline tool treats the unscored ones.
• A mismatched TRANSFORMATIONS/POST_TRANSFORMATIONS config now stops things, rather than silently misbehaving. Training writes a <model>.preprocessing.json sidecar recording the exact pipeline it used; both live driving and offline analysis (§1.6/§1.7) read it back and refuse to proceed if the current config doesn't match. Two gaps worth knowing: the plain mc_calibrate CLI (§1.3) does not run this check itself — it's enforced at drive-time and analysis-time, not at calibration-time — so it's still on you to point --tub/--config at the right pair when calibrating by hand; and the sidecar carries no schema version, so a sidecar written by an older build is compared only on the fields both versions happen to share (you'll get a "predates tracking …" warning naming what couldn't be verified).

---

2. Technical deep dive

2.1 Why explainability and uncertainty matter here

A CNN driving policy is a black box mapping camera pixels to a steering angle; by default it gives you the same confident-looking single number whether the road ahead is a textbook straightaway or something the model has never seen. For a small robot that can hit a wall or a person, being able to ask "how sure are you" and "why did you decide that" is the difference between a system you can trust incrementally and one you can only evaluate by watching it crash. Everything in this toolkit is one of two things: a way of estimating uncertainty (confidence, novelty, TTA stability — "should we trust this frame's answer?"), or a way of producing visual attribution (Grad-CAM, Grad-CAM++, Integrated Gradients, saliency — "what part of the image caused this answer?").

2.2 Dropout and Monte Carlo Dropout (confidence)

Dropout, as used during normal training, randomly zeroes out a fraction of a layer's neurons on every training step (Donkeycar's default linear architecture has Dropout(0.2) after several convolutional and dense layers — see donkeycar/parts/keras.py). This forces the network to not rely too heavily on any single neuron (since it might be zeroed out next step), which is a well-known technique for reducing overfitting. Normally, once training finishes, dropout is switched off for inference — every neuron participates, and you get one deterministic prediction.

Monte Carlo Dropout (Gal & Ghahramani, 2016) is the trick of leaving dropout switched on at inference time and running the same input through the network multiple times. Since a different random set of neurons is zeroed out on each pass, each pass is effectively asking a slightly different, randomly-thinned "sub-network" for its opinion. If the input is something the model is confident about, most sub-networks should agree on roughly the same steering angle regardless of which neurons got dropped — the underlying feature is redundantly represented. If the input is ambiguous or unfamiliar, different sub-networks latch onto different (possibly spurious) cues and disagree more. Confidence in this toolkit is literally the inverse of that disagreement: run N stochastic passes, measure the variance of the resulting steering angle across them, and treat high variance as low confidence.

Implementation detail worth knowing: running N passes sequentially is too slow for a 20 Hz drive loop (roughly 165 ms for 15 passes on a desktop in early testing). Instead, the same image is replicated N times into one batch and passed through the model in a single batched call — the network still applies an independent random dropout mask to each element of the batch, so this is mathematically the same N independent stochastic passes, just computed in parallel (roughly 29 ms for the same 15 passes). See donkeycar/parts/mc_dropout.py.

The raw variance is smoothed frame-to-frame with an exponential moving average (XAI_CONFIDENCE_ALPHA) so the displayed number doesn't flicker, then converted into the 0-100% shown on the dashboard via calibration (§2.3).

2.3 Calibration mechanics

A raw MC-Dropout variance number (or a raw Mahalanobis distance, for novelty) is meaningless on its own — its scale depends on this specific model's weights, this specific dataset, even the random dropout masks drawn. Calibration (mc_calibrate.py) solves this by replaying a known tub through the model once, collecting the raw variance for every frame, and computing percentiles (e.g. the 50th/80th/97th) of that distribution. A monotonic piecewise-linear map is then built from "raw variance value" to "0-100% score," anchored at those percentiles, and saved to <model>.calib.json. At drive time, a new raw variance is looked up against this same map to produce the displayed percentage.

The important consequence: a confidence score is a statement about where this frame's variance falls relative to your own training data's variance distribution — not a calibrated probability of correctness. This is also why the dashboard explicitly disclaims it as "relative to this model's own baseline." It also means the percentile anchors are inherently relative, not absolute: by construction, roughly 20% of any tub replayed against an 80th-percentile-derived threshold — even the original training data itself — will read above that threshold. A "low confidence" reading doesn't necessarily mean something is objectively rare; it means it's rarer than most of what this model saw during calibration.

Novelty and TTA stability are calibrated the same way, with their own percentile anchors, just walking in the opposite direction (ascending for novelty — higher raw distance is worse — vs. descending for confidence and TTA stability, where higher is better).

2.4 Feature-space novelty / out-of-distribution detection

The goal here is different from confidence: instead of "do sub-networks of this model agree," it's "does this frame look anything like what the model was trained on at all" — catching genuine out-of-distribution (OOD) input like the car leaving the track entirely.

The underlying statistic is Mahalanobis distance: instead of plain Euclidean distance from a "typical" feature vector, it's a distance that accounts for how much each dimension is expected to vary — a value 3 standard deviations away from the mean on a dimension that's normally almost constant is treated as far more surprising than the same absolute distance on a dimension that's normally noisy. Concretely: fit a Gaussian to a set of feature vectors extracted from calibration frames (mean + per-dimension variance — a diagonal covariance approximation, i.e. we don't model correlations between dimensions, which is far cheaper to compute and invert than a full covariance matrix), then for a new frame's feature vector, compute how many (variance-normalized) standard deviations away it falls.

The interesting part of this feature is which features that distance is measured in — and getting it wrong the first time was a genuine, real finding. The first implementation measured distance in the driving model's own penultimate layer (dense_2, 50-dimensional). This didn't work: a model trained only to predict steering angle has every incentive to discard anything not needed for that task — texture, color, semantic content — so grass, carpet and open track all collapse to nearly the same dense_2 vector, because they all mean "drive straight." Measuring against real training data, grass scored as more familiar than a legitimate on-track frame in some cases (a "grass is 90%+ confident" result that was genuinely reproduced, not a hypothetical). The fix: extract features from a generic, frozen ImageNet-pretrained encoder (MobileNetV2 by default, XAI_NOVELTY_ENCODER) instead of the task-specific steering model. A network trained on ImageNet's thousand object categories has to keep general visual information (texture, color, shape) around, so grass and track genuinely land in different regions of that feature space — the same math, applied to a feature space that hasn't collapsed away the information needed to tell them apart, correctly separated grass/carpet (scoring as clearly unfamiliar) from track frames.

The live detector (donkeycar/parts/novelty.py) runs one extra deterministic forward pass through this encoder per update, computes the Mahalanobis distance against the calibrated Gaussian, smooths it with an EMA, and maps it to 0-100% the same way as confidence (just ascending instead of descending). The offline viewer's per-location novelty overlay uses the very same encoder weights, just without the final averaging step, so the heat map and the percentage beside it live in the same feature space and answer the same question — a large improvement on the earlier version, where the map came from the steering model's conv2d_5 and the percentage from the encoder, i.e. two unrelated spaces.

They are not, however, the same measurement, and it's worth being precise about that: the live pooled encoder squashes the frame to a 128×128 square (fine, since every location is averaged together anyway), while the offline map runs at a larger aspect-preserving input (XAI_NOVELTY_SPATIAL_INPUT, default 224 → roughly a 7×12 grid on a 16:9 frame) so the heat map isn't stretched when drawn back over the frame. Different input geometry means different feature vectors, which is why calibration fits and stores them as two separate blocks (novelty_ood and novelty_ood_spatial) with their own Gaussians and anchors. Expect the map and the percentage to agree in character — the same regions look unfamiliar — not to agree numerically.

2.5 Test-Time Augmentation (TTA) stability

A third, independent question: if the same real-world scene were lit very slightly differently, would the model give a different answer? TTA stability answers this directly rather than by proxy: it takes the current frame, generates XAI_TTA_SAMPLES copies with small random photometric-only jitter (brightness/contrast/gamma/noise — deliberately not geometric changes like crop or rotation, since those would actually change what the correct steering angle is), batches them through the model in one deterministic forward pass (dropout off, unlike MC-Dropout), and measures the variance of the resulting steering angle across that batch. High variance means the model's answer is fragile to lighting noise that shouldn't matter; low variance means it's robust to it.

This is a genuinely different signal from MC-Dropout confidence, not a duplicate of it — MC-Dropout asks "do many random internal sub-networks of this one exact image agree," while TTA asks "does the same sub-network agree with itself across slightly different lightings of the same scene." Measuring both against the same real footage showed a correlation of only about 0.19 between their raw variances — confirming they pick up on different kinds of trouble.

2.6 Confidence-aware throttle scaling

ThrottleScaler takes whichever of the three 0-100% signals are enabled and, for each one independently, computes a scale factor between the configured min-scale and 1.0: full throttle above the signal's "reduced" threshold, linearly interpolated down to XAI_THROTTLE_MIN_SCALE between "reduced" and "critical," and held at the minimum below "critical." Confidence and TTA stability are descending (high value = good, so the scale shrinks as the value drops); novelty is ascending (high value = bad, so the scale shrinks as the value rises) — both use the same underlying interpolation, just mirrored.

The final throttle scale is the minimum across every enabled signal's scale — i.e. whichever signal currently looks worst wins, a direct implementation of "slow down if any signal indicates trouble," rather than averaging signals together (which could let one bad signal be diluted by two fine ones). If any signal has been continuously in its critical range for XAI_THROTTLE_STOP_DURATION seconds, throttle is forced to exactly zero; the timer is shared across signals, so a car that goes from "novelty-critical" straight into "confidence-critical" without recovering in between still counts as one continuous critical episode, not two resets. Steering is never touched by this part.

2.7 Visual attribution: four ways to ask "why"

The offline viewer computes four different attribution maps per analysed frame, because they answer subtly different questions and none of them is uniquely "correct":

Grad-CAM — computes the gradient of the (summed) steering output with respect to the activations of a late convolutional layer (conv2d_5), uses those gradients to weight each channel of that layer's feature map, sums the weighted channels, and applies a ReLU (keeping only the parts that positively influenced the output). Because it operates on a late-stage conv layer, the result is naturally coarse — a small grid (e.g. 8×13) upsampled back to image size — showing roughly where in the image mattered, not exact pixels. In this toolkit, Grad-CAM is combined with MC-Dropout: it's computed once per dropout sub-network in the same batch, and the mean map across sub-networks becomes the "attention" overlay while the pixel-wise variance across those same maps becomes the "uncertainty" overlay — literally, "where did the model look" and "where did different sub-networks disagree about where to look."

Grad-CAM++ — a refinement of Grad-CAM's channel weighting that accounts for a feature appearing in multiple locations at once (plain Grad-CAM's simple gradient-averaging weight can under-represent that case). The textbook derivation assumes a softmax classification output; since this is a regression output (a single steering value), this toolkit uses a practical approximation of the pixel-wise weighting derived from first- and second-order gradient terms (alpha = g² / (2g² + (ΣA)g³)) rather than the classification-specific formula. In practice it tends to produce sharper, more spatially precise blobs than plain Grad-CAM.

Integrated Gradients (IG) — instead of looking at gradients at the actual input alone (which can saturate — a very confidently-classified pixel can have a near-zero local gradient even though it clearly mattered), IG integrates the gradient along a straight-line path from a neutral baseline (a black image) to the real input, in XAI_IG_STEPS discrete steps (a Riemann-sum approximation of the integral), and multiplies the averaged path gradient by (input - baseline). This satisfies a completeness property that plain gradients don't: the attributions sum up to exactly the difference between the model's output on the real image and on the baseline. Because it operates directly on input pixels rather than a coarse conv layer, its maps are full resolution.

Vanilla gradient saliency — the simplest and cheapest: one backward pass, the gradient of the output directly with respect to every input pixel. (One implementation wrinkle: a saturating output activation would clip gradients right where they matter most, so the output layers are linearized first. For the linear architecture they already are, so this is detected and skipped entirely; where it is needed — a categorical model's softmax, via donkey makemovie --salient — it's applied to a reloaded copy, leaving the pilot you're still driving with untouched.) Like IG, it's full-resolution, but with no baseline/integration step, so it's noisier and more speckled.

Why look at all four: the two conv-layer-based methods (Grad-CAM, Grad-CAM++) give smooth, coarse, easy-to-read localization; the two pixel-gradient methods (saliency, IG) give precise-but-noisier, exact per-pixel attribution. On a real test frame containing a pedestrian, Grad-CAM/++ showed two smooth blobs on the ground near the pedestrian, while saliency showed many small, scattered points across the pedestrian, nearby pillars, and background architecture — a genuinely different, complementary answer, not a redundant one. Relying on only one technique risks mistaking "coarse but clean" for "the whole picture," or "precise but noisy" for "definitely irrelevant."

2.8 How augmentations feed into calibration

Calibration's job is to define "what does normal training data look like" — which becomes the novelty detector's baseline. If a model is trained with, say, AUGMENTATIONS = ['BRIGHTNESS', 'BLUR'] (so it has learned to drive correctly through varying exposure and softer focus), but calibration is only ever shown clean, un-augmented frames, the novelty baseline never learns that those conditions are "normal" — so the very conditions the model was made robust to would get flagged as unfamiliar and throttle down live, exactly backwards from the intent.

This was measured directly, not just reasoned about: fitting the novelty baseline on clean frames only, then scoring augmented-but-otherwise-normal frames against it, 79% crossed the "reduced" novelty threshold (vs. 38% for clean frames) and 50% crossed "critical" (vs. 10%). A model trained to handle varying lighting would have been throttled down by its own safety feature the moment it encountered it.

The fix, in mc_calibrate.py: when AUGMENTATIONS is non-empty and XAI_CALIBRATE_WITH_AUGMENTATIONS is on (the default), calibration pushes a strided subset of frames (capped by XAI_CALIBRATE_AUG_MAX_SAMPLES) through the same ImageAugmentation pipeline used in training, XAI_CALIBRATE_AUG_PASSES times each (re-randomized every pass), and pools those augmented features into the novelty baseline alongside the clean ones. The resulting calib.json carries a small "augmentation" provenance block recording whether this happened and with what settings, so it's inspectable after the fact.

This is scoped to novelty only — not confidence or TTA. Both of those carry an exponential moving average over a time-ordered sequence of frames; splicing randomly-augmented frames into that sequence would corrupt the smoothing (a sudden synthetic brightness jump between two real consecutive frames isn't something either signal is designed to interpret). Novelty's calibration, by contrast, is a plain unordered percentile fit over a feature distribution, so mixing in augmented samples is safe and (per the measurement above) necessary.

2.9 How transformations affect the signals

TRANSFORMATIONS (crop, trapezoidal mask, colour-space conversion, high-pass filters) are different from augmentations in one way that matters enormously here: they apply at both training and inference. The model is only ever shown transformed pixels, so anything measuring the model's behaviour must be measured on transformed pixels too, or it's describing a situation that never actually occurs while driving.

That creates a split, and the toolkit handles the two halves differently:

• Confidence, TTA stability, Grad-CAM, Grad-CAM++, saliency and Integrated Gradients all get the transformed frame. Every one of them is a question about your model — how much its sub-networks disagree, how stable it is, which pixels drove its output — so they have to see the model's real input. Calibration replays tub frames through the same TRANSFORMATIONS pipeline training uses, in the same order (transform → augment → post-transform).
• Novelty gets the raw frame. It's the odd one out because it doesn't use your steering model at all — it measures scene familiarity in a frozen ImageNet encoder (§2.4). A crop or a high-pass filter strips exactly the colour and texture content that encoder relies on, so transforming the input degrades the very thing that makes OOD detection work. "Is this scene familiar?" is a question about the world, not about the model.

The magnitude here is not subtle. Measured on real frames with a modest crop (30 of 108 rows), feeding novelty the transformed frame instead of the raw one moved its score from 4.9% to 97.8% — normal driving reading as maximally unfamiliar. Whichever input you choose, calibration and live driving must use the same one; the toolkit keeps them in step.

Both CROP and TRAPEZE are masking transforms: they blank out pixels outside the region of interest but leave the image at its original dimensions. That makes a mismatch the dangerous case — same dimensions, no shape error, just silently wrong thresholds — so it is caught directly instead: training saves a <model>.preprocessing.json sidecar recording the exact TRANSFORMATIONS/POST_TRANSFORMATIONS/ROI_CROP_* pipeline it was trained with, and both live driving and offline analysis refuse to run if the current config doesn't match it (§1.9). A calibration also records which transformations built it and the dashboard raises a "Calibration stale" warning if that's drifted, independent of this check.

In the offline viewer, what you see by default is the raw camera frame with heat maps drawn on it — since every available transform preserves the frame's geometry, that mapping is direct. You can also switch to the "What the model actually sees (transformed)" layer to see the transformed frame directly, and tick "draw on model input" to see the heat maps drawn on it instead (§1.6).

2.10 Known limitations

• MC-Dropout's calibration percentiles are unseeded. Calibrating the same model against the same tub twice will produce slightly different p50/p80/p97 anchors each time (a real, measured difference, not a bug) — this is expected statistical noise from the random dropout masks, so don't expect bit-identical thresholds across reruns.
• (Fixed) Novelty's anchors used to be fitted on un-smoothed distances but scored on smoothed ones. _build_novelty_block took its p50/p80/p97 from the raw per-frame Mahalanobis distances of the calibration frames, while every consumer — the live detector and the offline timeline alike — feeds the EMA-smoothed distance through those anchors. Smoothing shrinks the spread (measured on a real logged drive: raw σ≈987 vs smoothed σ≈452, raw p97≈5239 vs smoothed p97≈2919), so the top of the scale was much harder to reach than the percentile design implied — replaying a drive through anchors fitted the old way, 4% of frames crossed the 65% "critical" mark scored against raw distances, 0% did scored the way the code actually scores them. calibrate_from_tub now folds the calibration replay's own clean, time-ordered distances through the same EMA the live detector uses (_replay_ema, matching FeatureNoveltyDetector.run()'s formula exactly) and fits novelty_ood's anchors on that sequence instead, via a new ood_anchor_distances parameter (_build_novelty_block, mc_calibrate.py). novelty_ood_spatial (the offline heat map) is untouched — it has no smoothing or interval concept, so raw per-location distances were already the right thing to fit on.
• (Fixed) Calibration used to always replay at interval = 0, while driving defaults to 0.3/0.5. The EMA constant is per update, not per second, so a rate-limited live signal used to be smoothed over a ~6× longer real-time window than the calibration that defined its thresholds (median frame spacing ≈0.05s at 20 Hz vs an effective ≈0.3-0.5s live update spacing — confirmed on a real drive: only 15% of frames actually trigger a fresh EMA fold at the default XAI_CONFIDENCE_INTERVAL=0.3). MCDropoutConfidence/FeatureNoveltyDetector/TTAStabilityDetector.run() all now accept an optional now= override; calibrate_from_tub reads each frame's own recorded _timestamp_ms and passes it through, so confidence and TTA replay through their live Part objects at the tub's actual recorded pace rather than however fast the replay loop happens to run, and novelty's distance sequence is folded through the tub's own timestamps via _replay_ema the same way. Every other caller — the live drive loop, gradcam_uncertainty.py's replay — doesn't pass now= and is unaffected. Tubs recorded before timestamp logging existed fall back to the old behaviour (measure every frame) with a warning, since replaying at the process's own speed has no relationship to how fast the frames were actually driven.
• A generic ImageNet encoder for novelty runs on CPU today. A Hailo-NPU-accelerated backend is a planned follow-up (to match the rest of the pipeline running on-device on hardware that has one), but isn't implemented in this toolkit yet — CPUEncoderExtractor (donkeycar/parts/ood.py) is currently the only backend.
• Novelty scores saturate rather than scale gracefully at the extreme end (§1.9) — useful for a binary "is this track or not" read, less useful for graded ranking between different kinds of out-of-distribution input.