ArrowSpace Latent Diffusion with Spectral Chart Conditioning — a
spectrally conditioned latent diffusion model for sound generation.
A frozen ArrowSpace graph-wiring prior
This repository is a sound-generation-specific fork of arrowspace-latent-diffusion. The image-generation code has been replaced with 1-D audio-native modules. Unlike ALD-SC, LSD has no falsifiable research claim — it exists for sound production: generating sounds for their own sake, and exploring what we call the sound of the future.
The frozen ArrowSpace prior
$(L_F, \lambda^{\mathrm{ED}})$ is used here as a sound-design tool, not as the object of a claim. The goal is simply to generate compelling, novel audio.
LSD is a sound-generation variant, not the ALD-SC research artifact. There is no falsifiable claim to defend. The objective is to let artists train fast on their own samples and explore what we call the sound of the future. Two knobs encode the artistic requirement that training be interesting, not merely reproducible:
-
SEED = None— non-repeatable training. Each run samples a fresh seed, so every model — and every generated sound — differs. Variation across runs is a feature, not a bug. -
NOISE_INJECT— latent-space noise injected into the EnCodec latent$z$ before decoding during decoder training. 0.0 reproduces the deterministic baseline; larger values raise the early loss and yield different, more varied models. Tune for taste.
# Run the end-to-end notebook
uv run jupyter notebook notebooks/01_sound_generation.ipynbThe basic point of this research programme is to design decoding using three structures, all computed from the training corpus via the ArrowSpace library:
-
The item-space — the 1-D audio latent
$z$ (EnCodec continuous features) carrying local acoustic detail. -
The feature-space graph Laplacian
$L_F$ — its eigenvectors$U_q$ define the smooth semantic subspace; its eigenvalues$\nu_k$ define entropy exchange rates. -
The dispersion network
$\lambda^{\mathrm{ED}}$ — ArrowSpace's per-feature energy-dispersion distribution (arXiv:2606.21535). Not a diagnostic, but a constructive representation of how semantic structure is distributed over the feature graph.
Decoding uses WaveReconstructionBlock propagates information along
See docs/00.md § "The research programme" and
AGENTS.md §1.1 for the full design statement.
Frozen ArrowSpace prior
┌──────────────────────────────────────┐
│ L_F, U_q, Λ_q, λ_ED (from ArrowSpace)│
└──────────────────────────────────────┘
│
audio x ── EnCodec ──► (z, A) ──► c_spec ──► 1-D DiT ──► ẑ
(frozen) │ │ │ │ │
encoder │ │ └─ project: A U_q U_q^T │
│ │ │
│ └─ A = pool(z) (pooled features) │
│ │
│ SpectralSchedule (Barontini clock) │
│ │ ▼
└────────────── GraphDecoder ◄── ClockGated tempo ──► x̂
(WaveReconstructionBlock:
project → gate → lift along U_q)
Each audio clip encodes via frozen EnCodec (24 kHz mono) to:
- z — 1-D latent (EnCodec pre-quantization continuous features, D=128, T=375 for 5s clips)
- A = pool(z) — pooled feature field for spectral chart extraction
-
c_spec
$\in \mathbb{R}^{3q}$ —[ẽ, λ_chart, ν](conditioning vector)
c_spec is derived from z itself (self-consistent decoding): the DiT generates z unconditionally, then the decoder derives c_spec from the generated z. No separate c_spec sampling is needed.
The WaveReconstructionBlock at each resolution:
- Pool feature activations →
$A$ - Project to chart:
$\hat{H} = A \cdot U_q$ (decode along smooth directions) - Gate by dispersion:
$g = \sigma(W \cdot c_{\mathrm{spec}})$ (energy allocation) - Lift back:
$A' = (\hat{H} \odot g) \cdot U_q^\top$ (reconstruct in feature space) - Residual conv update
The ClockGatedGraphDecoder modulates gate strength by
The baseline decoder (BaselineAudioDecoder) has identical channel
widths and upsampling strides, but uses plain ResBlock1d in place of
WaveReconstructionBlock (no
| Component | File | Description |
|---|---|---|
| ArrowSpace adapter | wire_graph.py |
|
| Frozen prior |
arrow_prior.py, build_prior.py
|
|
| 2.5-D encoding target | dual_space.py |
|
| Audio codec | audio_codec.py |
Frozen EnCodecEncoder, BaselineAudioDecoder, AudioVAE
|
| 1-D DiT denoiser | dit.py |
1-D Conv1d patchify + AdaLN + CFG dropout, latent_shape attr |
| Schedules | schedule.py |
Cosine + linear, v-prediction |
| Graph decoder | graph_decoder.py |
1-D WaveReconstructionBlock, GraphDecoder, ClockGatedGraphDecoder
|
| Entropic clock | spectral_schedule.py |
|
| Samplers | sampling.py |
DDIM + Euler with spectral stopping, 1-D latent_shape noise init |
| Losses | losses.py |
|
| Training | trainer.py |
train_audio_decoder() (with noise_std latent augmentation) + train_audio_diffusion() + log_training() (structlog) |
| Data | data.py |
Esc50Dataset, AudioFolderDataset, ToyAudioDataset, MusicSynthDataset (load_audio_clip falls back to soundfile if torchaudio backend is unavailable) |
| Inference contract | inference.py |
LSDModel: generate_sound_bank (A), condition_on_audio (B), synthesize_midi (C) — see docs/03.md
|
| CLI scripts | scripts/ |
build_audio_prior, train_audio_decoder, train_audio_diffusion, sample_audio, eval_audio
|
| Configs | configs/ |
audio_decoder.yaml, audio_diffusion.yaml
|
| Notebook | notebooks/01_sound_generation.ipynb |
End-to-end pipeline with interactive knobs |
160 unit tests, all on CPU. uv run pytest tests/ -v.
| Phase | Scope | State |
|---|---|---|
| Setup | Project rename, audio deps, docs identity | ✅ Complete |
| Phase 1 | 1-D DiT + graph decoder + EnCodec + ESC-50 + notebook | ✅ Complete |
| Phase 2 | Music-specific generation (text/CLAP, genre, long-form) | Future |
| Phase 3 | Advanced ESDM concepts (wave recurrence, entropy clock training) | Future |
- #1 — Sound generation: end-to-end audio synthesis via ALD-SC
- FAD computation (
fadtkremoved due to dependency conflicts; multi-scale STFT used as fallback) - Real-data experiments on ESC-50 with real EnCodec (notebooks 05 and 06 use real EnCodec; notebook 01 is a CPU demo)
This project uses uv for dependency management and requires Python ≥ 3.13 with PyTorch ≥ 2.2. EnCodec and torchaudio provide audio encoding/decoding; Jupyter is included for running notebooks.
git clone https://github.com/tuned-org-uk/latent-sound-diffusion.git
cd latent-sound-diffusion
uv sync# Run the test suite (CPU; 160 tests)
uv run pytest tests/ -v
# Lint and format
uv run ruff check src/ tests/ scripts/
uv run ruff format src/ tests/ scripts/
# Build the ArrowSpace prior from EnCodec features
uv run python scripts/build_audio_prior.py --toy --out prior.pt
# Train a decoder (graph or baseline)
uv run python scripts/train_audio_decoder.py --prior prior.pt --graph --epochs 50
uv run python scripts/train_audio_decoder.py --prior prior.pt --baseline --epochs 50
# Train the 1-D DiT
uv run python scripts/train_audio_diffusion.py --prior prior.pt --toy --epochs 50
# Generate audio
uv run python scripts/sample_audio.py --prior prior.pt --decoder decoder.pt --dit dit.pt --out results/sample.wav
# Evaluate reconstruction (graph vs baseline, λ_ED ablation)
uv run python scripts/eval_audio.py --graph-decoder decoder.pt --baseline-decoder baseline.pt --toy
# Run the end-to-end notebook
uv run jupyter notebook notebooks/01_sound_generation.ipynb| # | Notebook | Description |
|---|---|---|
| 01 | 01_sound_generation.ipynb |
End-to-end sound generation with interactive knobs (SEED, STEPS, USE_C_SPEC, TEMPERATURE, CLIP_INDEX) |
| 02 | 02_long_form_generation.ipynb |
Long-form (5-8s) generation via overlap-and-add with pitch envelope, ADSR, and timbral filtering knobs |
| 03 | 03_bps_modulation_effects.ipynb |
BPS-synchronized modulation (tremolo, vibrato, filter sweep) with pedalboard effects and scipy.signal IIR filters |
| 04 | 04_latent_concatenation.ipynb |
Latent-space concatenation of multiple variations into ~20s output, with per-section BPS modulation and pedalboard effects |
| 05 | 05_full_music_generation.ipynb |
Full music generation with the real frozen EnCodec encoder and synthetic MusicSynthDataset |
| 06 | 06_end_to_end_real_data.ipynb |
End-to-end training on real audio files in data/ using AudioFolderDataset and real EnCodec |
| 07 | 07_production_workflow.ipynb |
Full LSD production workflow — train two models, A/B/C inference, then longform + BPS/effects + latent-concat progression |
To run a notebook server:
# Launch Jupyter Lab (preferred)
uv run jupyter lab notebooks/
# Or classic notebook interface
uv run jupyter notebook notebooks/01_sound_generation.ipynb
# Or execute non-interactively (outputs saved in-place)
uv run jupyter nbconvert --to notebook --execute notebooks/01_sound_generation.ipynbNotebook 01 runs entirely on CPU using a stub encoder and
ToyAudioDataset (no external data or GPU required). Notebooks 05 and 06 use
the real frozen EnCodecEncoder: notebook 05 with synthetic MusicSynthDataset,
and notebook 06 with real audio files in data/ via AudioFolderDataset.
latent-sound-diffusion/
├── pyproject.toml # uv / hatchling project config
├── AGENTS.md # contributor guide (read this first)
├── configs/
│ ├── audio_decoder.yaml # decoder training config
│ └── audio_diffusion.yaml # diffusion training config
├── docs/
│ ├── 00.md # design document — the research programme
│ ├── 01.md # design document — ESDM transfer
│ └── 02.md # design document — audio adaptation
├── notebooks/
│ ├── 01_sound_generation.ipynb # end-to-end notebook (stub encoder)
│ ├── 02_long_form_generation.ipynb # overlap-and-add + effects
│ ├── 03_bps_modulation_effects.ipynb # BPS modulation + pedalboard
│ ├── 04_latent_concatenation.ipynb # latent-space concatenation
│ ├── 05_full_music_generation.ipynb # real EnCodec encoder
│ ├── 06_end_to_end_real_data.ipynb # train on audio files in data/
│ └── 07_production_workflow.ipynb # full production workflow (A/B/C + effects chain)
├── scripts/
│ ├── build_audio_prior.py # build prior from EnCodec features
│ ├── train_audio_decoder.py # decoder training (graph vs baseline)
│ ├── train_audio_diffusion.py # 1-D DiT training
│ ├── sample_audio.py # CLI audio generation
│ └── eval_audio.py # reconstruction + FAD eval
├── src/ald_sc/
│ ├── __init__.py
│ ├── arrow_prior.py # ArrowSpacePrior: frozen spectral prior
│ ├── audio_codec.py # EnCodecEncoder, BaselineAudioDecoder, AudioVAE
│ ├── build_prior.py # build_arrow_prior() from corpus embeddings
│ ├── data.py # Esc50Dataset, AudioFolderDataset, ToyAudioDataset
│ ├── dit.py # MinimalDiT: 1-D patchify + AdaLN + CFG
│ ├── dual_space.py # DualSpaceMatrix M_N (2.5-D encoding target)
│ ├── graph_decoder.py # 1-D WaveReconstructionBlock, GraphDecoder,
│ │ # ClockGatedGraphDecoder
│ ├── losses.py # ALDSCLoss: diff + rec (L1+STFT) + chart + smooth
│ ├── sampling.py # sample_euler(), sample_ddim() + spectral stopping
│ ├── schedule.py # CosineSchedule, LinearSchedule (v-prediction)
│ ├── spectral_schedule.py # Per-mode τ_k, ᾱ_k, heat-death criterion
│ ├── trainer.py # train_audio_decoder(), train_audio_diffusion()
│ └── wire_graph.py # ArrowSpace adapter: L_F + λ_ED
└── tests/ # 16 test files, 160 tests (CPU)
-
Frozen prior.
$L_F$ ,$U_q$ are buffers, never parameters. - Frozen encoder. EnCodec weights are never updated. Only the decoder and DiT are trained.
-
Decoding on the feature-space manifold.
$L_F$ defines reconstruction paths (via$U_q$ );$\lambda^{\mathrm{ED}}$ defines energy allocation. -
Diffusion runs on
$z$ only. No second diffusion process over the spectral chart$s$ . - Corpus-level prior, not per-clip. Do not construct a new graph per audio clip.
- Unconditional generation (Phase 1). No text/genre/tag conditioning. Text/CLAP conditioning is tracked separately for the music-generation follow-up.
See AGENTS.md §6 for the full design constraints.
- Design documents:
docs/00.md,docs/01.md,docs/02.md - Upstream:
arrowspace-latent-diffusion - Diffusion as spectral-geometric projection
entropic-semantic-diffusionpyarrowspace— ArrowSpace library (Rust bindings)- Energy Dispersion Networks
- EnCodec: Défossez et al., 2022
- ESC-50: Piczak, 2015
- Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (CVPR 2022)
- Barontini, Testing the problem of time with cold atoms (PRL 2026)
- Stancevic et al., Entropic Time Schedulers for Generative Diffusion Models (arXiv 2025)
MIT