SwAV-HAR: Self-Supervised Learning for Human Activity Recognition

Adaptation of SwAV (Caron et al., NeurIPS 2020) for self-supervised pretraining on inertial sensor time series (accelerometer + gyroscope).

Table of contents

• Project structure
• Architecture overview
• Setup
• Data
• Checkpoints
• How to run
  • Pretraining
  • Evaluation
• Config reference
• Shell scripts
  • Single-run wrappers
  • Multi-dataset orchestrators

---

Project structure

swav-har/
├── train.py                  # SwAV pretraining entry point
├── eval_linear_har.py        # Frozen-backbone linear probe + clustering metrics
├── eval_finetune_har.py      # End-to-end fine-tuning from a pretrained checkpoint
├── eval_scratch_har.py       # Supervised baseline (backbone trained from scratch)
├── eval_spc_har.py           # Samples-per-class sweep (freeze / fine-tuning / scratch)
├── configs/                  # YAML experiment configs
├── scripts/                  # Shell helpers for batch runs and ablations
└── src/
    ├── models/
    │   └── swav_har.py       # SwAVModel: backbone + projection head + prototypes
    ├── backbones/
    │   ├── resnet_se5.py     # ResNet-SE-5 (1D, backbone_dim=64)
    ├── augmentations/
    │   └── ts_augs.py        # Time-series multi-crop augmentations
    ├── datasets/
    │   └── har_dataset.py    # DAGHAR dataset loaders (UCI, MotionSense, HAPT, etc.)
    ├── training/
    │   ├── trainer.py        # SwAVTrainer: training loop, queue, checkpointing
    │   └── swav_loss.py      # SwAV loss + distributed Sinkhorn-Knopp
    ├── optim/
    │   └── lars.py           # LARS optimizer (pure PyTorch)
    └── eval/
        ├── clustering_eval.py  # KMeans and prototype clustering metrics
        └── tsne_viz.py         # tSNE visualization

---

Architecture overview

The SwAV pipeline has four stages, all configured via a single YAML file:

1. Dataset (src/datasets/har_dataset.py)

Loads DAGHAR standardized-view CSVs. Each sample is a [6, 60] tensor (6 IMU channels, 60 timesteps at 20 Hz = 3 seconds). Available DataModules:

Registry name	Dataset	Classes	Activity labels
uci	UCI-HAR	6	Sit, Stand, Walk, Stair-up, Stair-down
motionsense	MotionSense	6	Sit, Stand, Walk, Stair-up, Stair-down, Run
kuhar	KuHar	6	(same as Motion Sense)
realworld_thigh	RealWorld (thigh)	6	(same as Motion Sense)
realworld_waist	RealWorld (waist)	6	(same as Motion Sense)
wisdm	WISDM	6	(same as Motion Sense)
hapt	HAPT	7	Sit, Stand, Walk, Stair-up, Stair-down, Laying, Transition

Each DataModule knows its own n_classes and activity_labels - no need to specify them in the config.

2. Augmentations (src/augmentations/ts_augs.py)

Multi-crop strategy adapted from images to time series:

• 2 global views: full 60-timestep window + heavy augmentations (noise, scaling, temporal permutation).
• 4 small views: random 15-timestep sub-windows + lighter augmentations (noise, scaling only).

Sinkhorn-Knopp codes are computed only from global views. Available augmentations: GaussianNoise, Scaling, GlobalScaling, AxisPermutation, Permutation, RandomCrop. Each can be toggled in the config.

3. Model (src/models/swav_har.py)

SwAVModel wraps three components:

input [B, 6, T] => backbone => features [B, 64]
                => projection head (MLP) => embeddings [B, 128] (L2-normalized)
                => prototype layer => scores [B, K]

• Backbone: extracts features from raw sensor data. Default is ResNetSE5 (output: 64-dim).
• Projection head: Linear(64, 256) => BN => ReLU => Linear(256, 128) when hidden_mlp=256. Maps backbone features to the space where SwAV clustering operates.
• Prototypes: nn.Linear(128, K, bias=False). Weights are normalized to the unit sphere each iteration. K is set via nmb_prototypes in the config.

Views of different temporal sizes are grouped by size and processed through the backbone in a single batch per size group for efficiency.

4. Training loop (src/training/)

Step-based training with:

• LR schedule: linear warmup + cosine decay.
• Prototype normalization: weights projected onto the unit sphere before each forward pass.
• Sinkhorn-Knopp (swav_loss.py): computes soft assignments enforcing equipartition across prototypes. Runs in float64 to avoid overflow. Supports distributed all_reduce.
• Queue: FIFO feature bank enabled after queue_start_step. Stabilizes assignments by augmenting the batch with recent embeddings.
• freeze_prototypes_niters: zeroes prototype gradients for the first N iterations to prevent early collapse.
• Mixed precision: optional FP16 via torch.amp.
• Optimizer: Adam (default) or LARS.

---

Setup

pip install -r requirements.txt

Assumes a base environment with PyTorch >= 2.0, numpy, pandas, scipy, scikit-learn, matplotlib, and PyYAML (provided by the Docker image or Minerva-Dev).

---

Data

Not included in this repository. The code expects the DAGHAR standardized-view CSV format (train.csv / validation.csv / test.csv per dataset). Set daghar_root in the config (or pass --eval_daghar_root) to your local copy.

---

Checkpoints

To reproduce results, run pretraining with train.py and the configs in configs/, or request checkpoints from the authors.

---

How to run

Pretraining

Multi-GPU (torchrun):

CUDA_VISIBLE_DEVICES=0,1,2 torchrun --nproc_per_node=3 \
    train.py --config configs/<config_name>.yaml

To train on a different dataset, use the corresponding config:

python train.py --config configs/<config_name>.yaml

Checkpoints are saved to dump_path defined in the YAML (e.g., checkpoints/<config_name>). A full checkpoint is saved every checkpoint_freq steps.

To change backbone, dataset, augmentation, or prototypes, edit the YAML.

Evaluation

There are four evaluation scripts. All support cross-dataset evaluation: use --eval_dataset and --eval_daghar_root to evaluate on a different dataset than the one used for training.

1. Linear probe (eval_linear_har.py)

Freezes the pretrained backbone, extracts features once, and trains a linear classifier on top. Also reports KMeans clustering metrics, prototype-based clustering metrics, and tSNE plots.

# Evaluate on the same dataset as training
python eval_linear_har.py \
    --config configs/<config_name>.yaml \
    --pretrained checkpoints/<config_name>/checkpoint.pth.tar

# Cross-dataset: train on UCI, evaluate on HAPT
python eval_linear_har.py \
    --config configs/<config_name>.yaml \
    --pretrained checkpoints/<config_name>/checkpoint.pth.tar \
    --eval_dataset hapt

# Cross-dataset with different DAGHAR root
python eval_linear_har.py \
    --config configs/<config_name>.yaml \
    --pretrained checkpoints/<config_name>/checkpoint.pth.tar \
    --eval_dataset hapt \
    --eval_daghar_root /path/to/other/daghar

Key options:

Flag	Default	Description
--feature_space	backbone	Which layer to evaluate: backbone (64-dim) or projection (128-dim, L2-normalized)
--max_steps	25000	Number of training steps for the linear head
--lr	1e-3	Learning rate
--eval_dataset	config's dataset	Dataset name for evaluation (e.g., hapt, motionsense)
--eval_daghar_root	config's root	DAGHAR root path for eval dataset

Outputs in <dump_path>/eval_linear/: results.txt, train_metrics.csv, val_metrics.csv, confusion matrices, tSNE plots.

2. Fine-tuning (eval_finetune_har.py)

Trains the full model end-to-end starting from a pretrained SwAV checkpoint, with two LR groups (--lr for the head, --lr_backbone for the backbone).

python eval_finetune_har.py \
    --config configs/<config_name>.yaml \
    --pretrained checkpoints/<config_name>/checkpoint.pth.tar \
    --epochs 100 --lr 1e-3 --lr_backbone 1e-4

# Cross-dataset: pretrained on UCI, fine-tuning and evaluate on HAPT
python eval_finetune_har.py \
    --config configs/<config_name>.yaml \
    --pretrained checkpoints/<config_name>/checkpoint.pth.tar \
    --eval_dataset hapt

Outputs in <dump_path>/eval_finetune/: results.txt, confusion_matrix_kmeans.txt, tSNE plot, best_classifier.pth.

3. Supervised baseline / scratch (eval_scratch_har.py)

Backbone randomly initialized and trained end-to-end - no --pretrained needed. Serves as the supervised baseline to quantify the value of SwAV pretraining.

python eval_scratch_har.py \
    --config configs/<config_name>.yaml \
    --epochs 100 --lr 1e-3

Outputs in <dump_path>/eval_scratch/: results.txt (test accuracy + KMeans clustering), confusion_matrix_kmeans.txt, tsne_scratch.png, best_classifier.pth.

4. Samples-per-class sweep (eval_spc_har.py)

Measures accuracy as a function of the percentage of labeled data, across all three modes (freeze, fine-tuning, scratch). Each (mode, spc, seed) combination is an independent run.

# Same dataset
python eval_spc_har.py \
    --config configs/<config_name>.yaml \
    --pretrained checkpoints/<config_name>/checkpoint.pth.tar \
    --modes freeze finetune scratch \
    --spc 1 5 10 25 50 100 \
    --n_seeds 5

# Cross-dataset SPC sweep
python eval_spc_har.py \
    --config configs/<config_name>.yaml \
    --pretrained checkpoints/<config_name>/checkpoint.pth.tar \
    --eval_dataset hapt \
    --modes freeze finetune scratch

Key options:

Flag	Default	Description
--spc	1 5 10 25 50 75 100	Percentage of labeled samples per class
--n_seeds	5	Number of random seeds per (mode, spc) pair
--modes	freeze finetune scratch	Which modes to evaluate
--eval_dataset	config's dataset	Dataset for evaluation
--eval_daghar_root	config's root	DAGHAR root for eval dataset

Output: <dump_path>/eval_spc/spc_results.csv (written incrementally after each SPC value).

---

Config reference

Example config (configs/_final_uci.yaml):

---

Shell scripts

Pre-built scripts in scripts/, all thin wrappers around the Python entry points above. They fall into two groups: single-run wrappers, which take one <config> (and usually one <checkpoint>) as positional arguments, and multi-dataset orchestrators, which loop over all 7 DAGHAR datasets, generate a run script under logs/, and launch it in a detached tmux session so long runs survive disconnects.

Single-run wrappers

Support cross-dataset evaluation via the EVAL_DATASET / EVAL_DAGHAR_ROOT environment variables.

Script	Description
run_eval_freeze_har.sh	Frozen-backbone linear probe for one config/checkpoint (wraps eval_linear_har.py).
run_eval_finetune_har.sh	End-to-end fine-tuning for one config/checkpoint (wraps eval_finetune_har.py).
run_eval_scratch_har.sh	Supervised baseline, backbone trained from scratch (wraps eval_scratch_har.py).
run_spc_eval_har.sh	Samples-per-class sweep over a percentage grid (default 1 5 10 25 50 100 200 all) across freeze/fine-tuning/scratch (wraps eval_spc_har.py).
run_spc_count_eval_har.sh	Same sweep over a raw sample-count grid instead (--spc_unit count, default 1 5 10 25 50 100 200 1000); writes to a separate eval_spc_count/ dir.

Multi-dataset orchestrators

Loop over uci motionsense kuhar realworld_thigh realworld_waist wisdm hapt and run in a background tmux session.

Script	tmux session	Description
run_final_pretrain.sh	final_pretrain	Generates configs/_final_<ds>.yaml for each dataset from its configs/_in_domain_<ds>.yaml, applying the best augmentation variant and nmb_prototypes found by the ablation scripts (falling back to uci's settings for datasets without ablation results), then runs SwAV pretraining (train.py) for each dataset in sequence. Skips a dataset if checkpoints/final_<ds>/checkpoint.pth.tar already exists, unless FORCE_TRAIN=true.
run_in_domain_final.sh	in_domain_final	Per-dataset pipeline: pretrain (if no checkpoint yet) => freeze / fine-tuning / scratch evals launched in parallel across 3 GPUs => optional SPC sweep(s). Set SPC_ONLY=true to skip pretrain/evals and only run the SPC sweep(s) against already-existing checkpoints.
run_lodo.sh	lodo	Leave-One-Dataset-Out: for each target dataset, concatenates the train.csv files of the other 6 datasets into a merged CSV, pretrains SwAV on that union, then runs freeze/fine-tuning/scratch (+ optional SPC sweep) on the held-out target.
run_pipeline_final.sh	(chains the sessions above)	Orchestrator: waits for the final_pretrain tmux session to finish, then runs run_in_domain_final.sh followed by SPC_GRID="" bash scripts/run_lodo.sh, blocking on each session's completion before moving to the next. Logs everything to logs/pipeline_final.log.
run_transition_eval.sh	transition_eval	Discovers every checkpoint under checkpoints/{in_domain,final,lodo,cross}_*/ and runs eval_transitions_har.py against HAPT (HAPT_DAGHAR_ROOT) for each, to measure transition-class detection quality across all pretraining protocols.