Skip to content

smart-swarm.md

Latest commit

 

History

History
612 lines (463 loc) · 27.5 KB

smart-swarm.md

File metadata and controls

612 lines (463 loc) · 27.5 KB
Raw

Smart Swarm Guide

Mission Type: 2 (SMART_SWARM)
Primary UI Surface: Swarm Design page
Runtime Model: live leader-follower formation with saved follow chains and in-flight reassignment

Overview

Smart Swarm is the live, cooperative formation mode in MDS. Each drone uses a saved swarm assignment file:

  • real hardware: swarm.json

  • SITL: swarm_sitl.json

  • hw_id identifies the physical drone

  • follow identifies the drone it should follow by hardware ID

  • offset_x, offset_y, offset_z define the relative formation offset

  • frame controls whether offsets are interpreted in ned or body

Unlike the time-synchronized Swarm Trajectory mode, Smart Swarm is designed for live clustered operations where operators may change leaders, offsets, frames, or relay roles while drones are airborne.

Identity rule:

  • Mission Config slot reassignment changes show-slot ownership only
  • Smart Swarm follow chains stay hw_id-anchored
  • true spare replacement belongs in Fleet Enrollment, which is where follow references are rewritten for the new hardware identity

The current Smart Swarm inner loop is a hybrid model:

  • leader telemetry arrives as global lat/lon/alt
  • when available, leader local LOCAL_POSITION_NED also rides along the dedicated swarm-state stream
  • the follower runtime converts that into a shared local NED frame
  • offsets are applied in either ned or leader-body coordinates
  • PX4 receives local offboard VelocityNedYaw setpoints

That is intentional. Smart Swarm is a relative-control problem, so local offboard control is the primary path; global telemetry is only the outer reference.

First SITL Run

For a clean first Smart Swarm demo in SITL:

  1. launch the fleet: 
    bash multiple_sitl/create_dockers.sh 5
  2. start the dashboard: 
    bash app/linux_dashboard_start.sh --sitl
  3. open Overview and wait until the target drones show READY with live telemetry
  4. open Swarm Design and verify the saved SITL assignments from swarm_sitl.json
  5. use Formation Analysis to preview the intended cluster
  6. use Smart Swarm Runtime to review Formation Preview and the live readiness snapshot for either the selected drone or selected cluster
  7. start Smart Swarm for the intended scope
  8. if you want the full branch acceptance run from the command line, use: 
    python3 tools/validate_smart_swarm_runtime.py
  9. if you plan to chain Drone Show or Swarm Trajectory validation immediately afterward on the same SITL fleet, recreate the containers or manually restage the aircraft onto the next mode's launch geometry before the next run

The shipped 5-drone SITL demo layout currently contains two clusters and mixed ned / body offsets, so formation settle time is not instantaneous after takeoff.

Validator note:

  • the Smart Swarm validator restores the selected saved swarm assignments after a successful reassignment drill, so a temporary runtime reassignment does not become the new saved SITL baseline
  • it does not magically put landed aircraft back onto Drone Show staging slots; cross-mode launch validation still needs a deliberate launch-geometry reset between mission families

Operator Model

Design save workflow

The Swarm Design page separates local assignment saves from fleet-wide Git write-back during the current compatibility phase:

  • Save writes the current swarm.json assignment set on the GCS only.
  • Commit & Push saves, commits, and pushes the change so nodes can receive it through the current sync path.
  • A failed commit or push is reported as local-save success plus Git write-back failure, never as complete success.
  • Progress distinguishes the local save, commit, and remote push.

Offset fields accept signed meter values. On mobile, use the normal text keyboard if the numeric keyboard does not expose a minus key.

Smart Swarm now has two clean command scopes:

1. Single-drone commands

Use the normal drone action controls when you want to affect only one aircraft.

Examples:

  • send RTL to one drone
  • send LAND to one drone
  • send a new mission to one drone

These commands stay scoped to that drone. They do not automatically cancel Smart Swarm on unrelated drones.

Important caveat:

  • if the addressed drone is a leader or relay leader, its followers may still react through normal leader-loss logic
  • that can mean continued following, upstream reassignment, or self-hold, depending on topology and live health
  • so command scope stays local, but topology side effects can still propagate through the follow chain

2. Normal Dashboard start

The normal field flow is intentionally short: take off using MDS, QGroundControl, or the RC, let the aircraft stabilize, then open Overview and select Smart Swarm beside Drone Show. Review the brief leader/offset/status summary, click Start Smart Swarm, then Confirm start. Opening the card never sends a command. MDS resolves the saved executable cluster and starts the complete dependency-closed cluster. The takeoff does not have to come from MDS.

MDS still performs authoritative node-side airborne, PX4, link, and configuration checks. These are not UI bypasses. Dashboard telemetry age is advisory, not a second start gate. Start normally includes every displayed saved role, independently of the dashboard's generic selection checkboxes. If a follower is unavailable, Start without… offers one explicit confirmation for a dependency-safe partial formation; this is labelled partial and never silently called a full swarm. An unavailable leader cannot be bypassed because its descendants would be orphaned.

3. Advanced Swarm runtime commands

Use the Smart Swarm Runtime panel on the Swarm Design page when the intent is live Smart Swarm control:

  • Start Smart Swarm
  • Stop Swarm (Hold)
  • Land Swarm
  • RTL Swarm

The runtime panel supports:

  • Selected Drone
  • Selected Cluster

Specific cluster selections in Formation Analysis also drive the cluster-scoped runtime target. The All executable clusters option is analysis-only and does not issue one command across the full fleet.

This keeps swarm intent explicit instead of overloading the generic command sender with swarm-only controls, and it preserves mixed-mission operations when only part of the fleet is flying Smart Swarm.

The dashboard Smart Swarm card uses the same tracked session contract as the advanced runtime; it does not use the generic mission scheduling form. The advanced page remains the place for multiple clusters and recovery controls. Start requires a complete saved follow chain, so choose Selected Cluster when the selected drone is a follower; leader-only starts and per-drone recovery remain available.

These runtime commands now publish into the same shared command lifecycle stream as Command Control and per-drone airborne overrides. That means the backend-backed live/recent command monitor can recover command context after refresh/navigation instead of keeping Smart Swarm runtime actions as toast-only events.

Mixed-mission leader rule:

  • a Smart Swarm leader can be selected alone and reassigned into Standard Drone Show or Custom CSV Drone Show without cancelling Smart Swarm on unrelated followers
  • the addressed leader interrupts its own Smart Swarm runtime script before the new show mission starts; follower drones keep their current Smart Swarm mission and continue tracking the leader as long as usable leader telemetry is still published
  • prefer Stop Swarm (Hold), Land Swarm, or RTL Swarm when you want to recover a live formation; normal cancellation also performs a bounded Offboard-to-HOLD handoff before its controller exits, but the dedicated recovery controls communicate the operator's intent more clearly
  • if the leader changes, update the saved/runtime follow chain deliberately in Swarm Design instead of relying on implicit mission side effects

Formation preview and live readiness

The Smart Swarm Runtime panel intentionally separates:

  • saved formation preview
  • live readiness snapshot

The preview shows the saved follow chain, roles, and offsets for the current runtime target. The live readiness snapshot is based on current telemetry/readiness state for the targeted drones.

Important operator rule:

  • formation plots are not live flight views
  • every targeted drone must have current telemetry and authoritative armed/airborne evidence; MSL altitude is never treated as height above launch
  • runtime start performs the same final airborne admission again at command dispatch time
  • if a target drone is not ready, fix that on Overview or Mission Config before start
  • a multi-drone Take Off is not physically atomic after dispatch: every node repeats its own final readiness and ground checks, so one aircraft can still reject while another accepts. For the first field validation, launch and confirm each aircraft individually, let both settle in Hold, and only then start the exact Smart Swarm cluster.

Runtime ownership and leader changes

Each start carries a command/session ID and saved-topology revision. Both roles acknowledge that session before the follower engages. Runtime status separates configured role from active role and reports Starting, Active, Partial, Pilot takeover, or Stopped. Missing runtime evidence is Degraded or Unconfirmed, never proof that the aircraft stopped. If one required role rejects startup, another role's process launch is not shown as a fully active formation. A retry after a lost HTTP response recovers the same command ID.

The follower watches PX4 mode, armed state, and landed state. RC/QGC RTL, Land, or manual takeover stops follower setpoints and preserves the selected recovery mode; MDS never countercommands it with HOLD. This watcher does not elect leaders. Existing upstream_or_hold remains the single leader-loss policy: after its bounded loss window it follows the cycle-safe upstream leader, or self-promotes and holds. User-edited leader changes use the same transition path. Runtime assignment files carry session ownership and become inactive on shutdown, landing, takeover, and restart.

Slot Reassignment vs Spare Replacement

These scenarios are intentionally not the same workflow.

Slot reassignment

Example:

  • H5 is reassigned from P5 to P6
  • H6 is reassigned from P6 to P5

Use:

  • Mission Config

Effect:

  • Drone Show / Swarm Trajectory slot ownership changes
  • launch plots and Drone {pos_id}.csv mapping change
  • Smart Swarm follow chains do not change automatically

Why:

  • Smart Swarm topology is about which physical aircraft follows which physical leader, so follow remains hw_id-anchored

Spare replacement

Example:

  • failed fleet member H12
  • spare airframe H101 must take over slot P12

Use:

  • Fleet Enrollment → Replace existing slot

Effect:

  • slot P12 is preserved
  • the new physical drone takes over that fleet slot
  • replacement rewrites the affected Smart Swarm hw_id / follow references so the spared-in aircraft becomes the new physical leader target where needed

Same airframe, new companion image

Use:

  • Fleet Enrollment → Recover existing node

Effect:

  • same hw_id
  • same pos_id
  • refreshed IP / companion metadata only

Runtime Behavior

Leader transport and latency model

Smart Swarm now uses a dedicated leader-state contract instead of the old generic drone-state HTTP poll path:

  • primary transport: leader drone WS /ws/swarm-state
  • fallback transport: leader drone GET /api/v1/swarm/state
  • assignment source of truth: GCS swarm config routes
  • follower-side live assignment cache: persisted runtime assignment file

The dedicated swarm-state payload exists so Smart Swarm timing can evolve without coupling itself to the broader operator dashboard snapshot. The transport now carries:

  • millisecond telemetry timestamps
  • monotonic stream sequence numbers
  • leader global position/velocity
  • leader local NED position/velocity when available
  • yaw and yaw-rate

Follower freshness is no longer keyed off a coarse second-resolution update_time. Runtime freshness now depends on the dedicated stream contract, with HTTP fallback only when the realtime stream is unavailable.

The current transport timing knobs are centralized in params.py, including:

  • SMART_SWARM_CONTROL_RATE_HZ
  • SMART_SWARM_HTTP_FALLBACK_RATE_HZ
  • SMART_SWARM_CONFIG_REFRESH_INTERVAL_SEC
  • SMART_SWARM_SOURCE_MAX_AGE_SEC
  • SMART_SWARM_OWN_STATE_MAX_AGE_SEC
  • SMART_SWARM_HARD_STALE_TIMEOUT_SEC
  • SMART_SWARM_LEADER_STATE_TIMEOUT_SEC
  • SMART_SWARM_GCS_CONFIG_TIMEOUT_SEC
  • SMART_SWARM_GCS_NOTIFY_TIMEOUT_SEC
  • SMART_SWARM_USE_REALTIME_STREAM
  • SMART_SWARM_ENABLE_HTTP_FALLBACK
  • SMART_SWARM_STATE_STREAM_RATE_HZ
  • SMART_SWARM_STREAM_CONNECT_TIMEOUT_SEC
  • SMART_SWARM_STREAM_BACKOFF_INITIAL_SEC
  • SMART_SWARM_STREAM_BACKOFF_MAX_SEC
  • SMART_SWARM_STREAM_PREDICT_GRACE_SEC
  • SMART_SWARM_USE_LOCAL_NED_WHEN_VALID
  • SMART_SWARM_POSITION_GAIN
  • SMART_SWARM_KV
  • SMART_SWARM_LEADER_VELOCITY_FEEDFORWARD
  • SMART_SWARM_MAX_HORIZONTAL_SPEED_M_S
  • SMART_SWARM_MAX_VERTICAL_SPEED_M_S
  • SMART_SWARM_MAX_ACCELERATION_M_S2
  • SMART_SWARM_MAX_JERK_M_S3
  • SMART_SWARM_MAX_COMMAND_DT_SEC
  • SMART_SWARM_MAX_YAW_RATE_DEG_S
  • SMART_SWARM_CAPTURE_*
  • SMART_SWARM_TRACKING_*
  • SMART_SWARM_MAX_LEADER_UNREACHABLE_ATTEMPTS
  • SMART_SWARM_LEADER_ELECTION_COOLDOWN_SEC
  • SMART_SWARM_LEADER_LOSS_STRATEGY

That keeps Smart Swarm motion, freshness, and failover policy in one place instead of scattering literals across runtime tasks. The former generic PD and low-pass-filter settings are not a second configuration path.

Tracking proof workflow

If you need to prove that a follower is tracking the leader on the current runtime path, use the dedicated tracking-analysis guide:

That workflow captures:

  • expected follower N/E/D from leader state plus offsets
  • actual follower N/E/D
  • repeated jog-sized leader moves
  • explicit NED offsets and leader jogs (body-frame transforms also have unit tests)
  • JSON, CSV, and plot artifacts for later review

Current transport behavior and next-step roadmap

Current runtime contract:

  • follower-to-leader state is websocket-primary with bounded reconnect/backoff
  • follower-to-leader HTTP fallback is still available and intentionally kept
  • follower-to-GCS assignment refresh remains periodic HTTP polling
  • follower-to-GCS leader-change notify remains HTTP POST
  • stale-data and transport-loss handling share the same failover path

Why this is the correct current design:

  • realtime leader updates no longer wait on second-resolution timestamps
  • the websocket stream cuts avoidable polling delay without removing the known-good HTTP path
  • reconnect, stale-data grace, and failover remain explicit instead of hidden inside transport-specific side effects
  • command scope and assignment scope stay loosely coupled

Why it is still not the final large-scale architecture:

  • GCS-side live telemetry views are still a mix of polling and streams
  • assignment refresh is still HTTP polling rather than push-driven
  • real-world degraded networks can still benefit from transport metrics, confidence scoring, and richer outage analytics

Recommended next industrial path:

  1. keep the websocket + HTTP dual path
  2. validate degraded-network behavior under induced delay/loss/partitions
  3. add richer operator-visible leader-state confidence and transport metrics
  4. only then consider payload compression or more aggressive transport optimization

Real-world network considerations to keep in mind:

  • short disconnects must not immediately cause unsafe topology churn
  • stale but reachable links are often more dangerous than hard failures
  • leader election should stay topology-safe even when GCS is slow or briefly unavailable
  • reconnect behavior must avoid synchronized thundering-herd retries
  • operator logs must distinguish transport loss, stale data, and actual flight-mode failover
  • transport changes must not hide command scope; a leader-only override should stay leader-only unless explicit swarm logic says otherwise

Mission start

When Smart Swarm starts, each drone:

  1. loads the local fleet config
  2. refreshes the latest swarm assignment from GCS
  3. decides whether it is a top leader or follower
  4. starts follower tasks only if it is configured as a follower

This avoids stale local swarm_sitl.json assignments at startup.

Dynamic reassignment

During flight, the runtime periodically refreshes assignments from GCS. Supported live changes include:

  • changing the followed leader
  • changing offsets
  • switching between ned and body
  • switching a drone between leader and follower roles

When a drone transitions back into follower mode, the runtime now explicitly re-establishes offboard control and restarts any missing follower tasks instead of assuming the previous follower runtime is still healthy.

Session-scoped failover writeback uses the node's own callback capability and a compare-before-save topology revision; it cannot edit another node or overwrite newer operator changes. NED offsets compose along skipped ancestor links (for example, +6 m plus +6 m stays +12 m from the upstream leader). Body-frame ancestor headings cannot be inferred reliably after that source is lost, so that recovery holds instead of guessing. A failed writeback keeps the local Hold assignment until the operator changes that slot or starts a new session; an unchanged saved assignment cannot silently restart following.

Follower control behavior

Follower control now uses one stateful motion pipeline:

  • leader and own motion must pass identity, validity, finite-value, and freshness checks before use
  • the follower may enter a formation from a large valid separation; the controller reports an acquiring phase and closes the error with bounded speed, acceleration, and jerk
  • leader-velocity feedforward is included in the target velocity command
  • leader body-frame offsets include yaw-rate-induced offset velocity
  • horizontal speed, vertical speed, acceleration, jerk, and yaw rate are shaped from the first command after the zero Offboard seed
  • a delayed event-loop iteration cannot spend the whole scheduling delay as a larger acceleration or yaw budget
  • topology/offset changes and leader jog-sized target changes are filtered and shaped without a binary zero-velocity capture gate
  • a filtered, continuous position deadband reduces corrections driven by small GPS noise; velocity damping and the shared command shaper limit oscillation
  • stale leader confidence scales feedback and feedforward together before the hard failover deadline

That makes the controller better suited for:

  • leaders moving under mission scripts
  • live jog/manual leader motion
  • in-flight role changes and offset edits
  • mixed-quality links where short jitter bursts should not immediately create a topology event

Leader-loss handling

Current default policy: upstream_or_hold

If a follower loses its direct leader:

  • if the failed leader was itself following another leader, the follower can adopt that upstream leader after revision-safe writeback and NED offset composition; fresh motion and bounded recapture are still required
  • if no safe upstream leader exists, the drone self-promotes to an independent leader and enters HOLD
  • if GCS writeback cannot be confirmed, the node holds locally; GCS reporting is not itself flight authority, and a GCS outage alone does not trigger an election while the direct leader stream remains healthy

Leader-loss handling now treats both cases as degraded leader health:

  • outright leader API fetch failures
  • leader transport that still responds while the authoritative motion-source timestamp stops advancing

This is safer than the older global “next numeric hw_id” fallback because it stays within the active follow chain instead of jumping across unrelated drones.

Available policy values in params.py:

  • upstream_or_hold - default, cluster-safe fallback
  • hold - always self-promote and hold
  • next_hw_id - legacy deterministic behavior, kept only for controlled compatibility

Cycle protection is enforced in two places:

  • dashboard assignment validation before save
  • GCS backend validation for canonical PUT /api/v1/config/swarm and PATCH /api/v1/config/swarm/assignments/{hw_id} updates

That prevents live leader changes from silently introducing a loop into the follow chain.

Runtime Guarantees Added In This Audit

  • dedicated leader-state stream at WS /ws/swarm-state with GET /api/v1/swarm/state fallback
  • expected leader identity, producer validity, finite motion values, and millisecond source freshness are checked before a sample enters control
  • follower control waits for both own-state and leader-state lock before sending formation setpoints
  • leader prediction uses the validated PX4 fused position/velocity directly, extrapolates only over a bounded source-age horizon, and never advances the measurement clock. The redundant slow Kalman filter was removed after log replay showed residual forward velocity and a false forward target after the leader stopped. The single downstream controller owns noise smoothing
  • follower commands include leader-velocity feedforward before saturation, reducing steady-state lag against moving leaders
  • body-frame offsets include leader yaw-rate compensation
  • startup and reconfiguration use an explicit acquiring phase rather than requiring manual staging at the offset. No finite distance or target-step threshold rejects formation acquisition; invalid numeric state remains rejected
  • the command sent to PX4 is limited from its first sample by separate horizontal/vertical speed envelopes plus acceleration, jerk, and yaw rate
  • stale-data confidence applies to the complete motion request; invalid own state suspends motion, while valid target jumps and tracking divergence return through bounded acquisition
  • follower re-entry restarts offboard mode cleanly after leader-to-follower transitions
  • failed follower re-entry now retries instead of getting stuck half-switched
  • stale leader telemetry now participates in the same failover path as explicit request failures
  • runtime controls default to Selected Drone; cluster scope is opt-in, and every target requires fresh armed/airborne evidence
  • cluster-scoped start blockers now apply only to the targeted drones instead of unrelated unsaved edits elsewhere in the design page

Files That Matter

Runtime and failover

GCS persistence and live updates

  • swarm.py
    • GET /api/v1/config/swarm
    • PUT /api/v1/config/swarm
    • PATCH /api/v1/config/swarm/assignments/{hw_id}

Frontend control surfaces

Operational Notes

  • Smart Swarm follow links use hw_id, not pos_id.
  • Slot swaps change the show slot, not the follow chain.
  • Start Smart Swarm only after saving the intended assignments.
  • In SITL, the default demo file is swarm_sitl.json; it currently defines 5 drones across two clusters.
  • Use swarm runtime controls when you want either a selected-drone override or an explicit cluster-level intent.
  • Use single-drone controls when you want a scoped override.
  • Followers may start away from their configured offsets. Smart Swarm first reports Acquiring and moves them toward the formation through the shared speed/acceleration/jerk shaper. Operators must still keep the formation inside the configured flight/geofence envelope.
  • RC, jog, Precision Move, and Drone Show actions on a leader do not end its Smart Swarm role session. RTL, Land, Stop Swarm, or a configured terminal mode ends the session. An incompatible mission sent to a follower releases that follower only and reports the change to the cluster.
  • Leader actions borrow the session's MAVSDK server and never terminate it. One OS-held motion lease prevents a follower role change from issuing setpoints while that action is finishing. Role changes retry against the latest assignment; Stop/RTL/Land and pilot takeover retain precedence.
  • Hover, small corrections, acquisition, and turns use the same controller: filtered position error, continuous deadband, smooth bounded feedback, leader velocity feedforward, velocity damping, then speed/acceleration/jerk shaping. The shaper reserves braking room before reaching a speed limit; saturation is normal operation, not a reason to terminate following.
  • Joining formation, Settling into formation, Following, and Following paused describe actual follower reports. An active leader cannot hide a missing follower report. An unexpected control failure holds and remains reported until the operator stops/restarts the session.
  • Deadbands trade small residual position error for quieter hovering. Tune SMART_SWARM_POSITION_DEADBAND_M, SMART_SWARM_VERTICAL_DEADBAND_M, the position-filter time constant, gains, and motion limits together using flight logs. Noisy-hover and lagged-plant tests are regression evidence, not proof against every GNSS disturbance or aircraft dynamic. This is not collision avoidance: large acquisition paths must still be clear, and PX4 geofences and estimator failsafes are unchanged.
  • Do not weaken PX4 estimator, GNSS, arming, or Offboard-loss policy to make a field test pass. Resolve the underlying readiness evidence and review the active aircraft parameter profile deliberately.

Recommended SITL Validation

For each Smart Swarm release, validate at minimum:

  1. takeoff with 4-5 drones
  2. start Smart Swarm on a cluster
  3. change offsets and frame in flight
  4. reassign one follower to a different leader
  5. send a single-drone override to confirm other followers remain in Smart Swarm
  6. run Land Swarm or RTL Swarm
  7. verify all drones disarm cleanly

The reusable validation tool for this flow is:

python3 tools/validate_smart_swarm_runtime.py

For stricter acceptance or degraded-network/failover checks, also run:

python3 tools/validate_smart_swarm_runtime.py \
  --horizontal-tolerance 1.5 \
  --altitude-tolerance 0.6

python3 tools/validate_smart_swarm_runtime.py \
  --simulate-leader-dropout

The leader-dropout drill is SITL-oriented. It pauses the active leader container, validates promotion / continued follower tracking, then confirms the paused leader resumes telemetry after unpause.

Known Next-Step Opportunities

  • richer operator-facing swarm stop/hold state reporting
  • smarter cluster-level leader election policies
  • transport optimization beyond HTTP polling if very large swarms require it
  • UI playback and incident review tied to unified logging