demo: flip config to segment-then-reference #5701

theosanderson · 2025-12-15T00:18:01Z

resolves #

Screenshot

PR Checklist

All necessary documentation has been adapted.
The implemented feature is covered by appropriate, automated tests.
Any manual testing that has been done is documented (i.e. what exactly was tested?)

🚀 Preview: Add preview label to enable

theosanderson · 2025-12-15T12:02:06Z

@claude Write up a summary of changes in this PR

claude · 2025-12-15T12:02:26Z

Claude finished @theosanderson's task in 1m 55s —— View job

Summary of PR Changes

This PR implements a major architectural restructuring of how reference genomes are organized and managed across the Loculus platform, moving from a "suborganism-first" to a "segment-first" data model. This is a significant refactoring that touches both infrastructure configuration and the website frontend.

Core Architectural Change: Segment-First Reference Genome Structure

Previous Model (Suborganism-First)

The old structure organized reference genomes by "suborganism" (essentially a reference name), with segments nested underneath:

referenceGenomes[suborganism][segment] = { sequence, genes }

New Model (Segment-First)

The new structure organizes by segment first, with multiple references available per segment:

referenceGenomes[segment][reference] = { sequence, insdcAccessionFull, genes }

This is a more flexible model that better represents biological reality - different segments of multi-segmented organisms can use different reference strains.

Key Changes by Component

Kubernetes Configuration (`kubernetes/loculus/`)

values.yaml (+245/-220 lines)

Restructured the entire reference genome configuration to use the new segment-first format
Updated organism configurations to reflect the new data model

values.schema.json (+47/-57 lines)

Updated JSON schema to validate the new segment-first structure
Added new field onlyForReferenceName to metadata to replace onlyForSuborganism

_merged-reference-genomes.tpl (+65/-37 lines)

Complete rewrite of the Helm template that transforms reference genomes into LAPIS configuration
Now handles two modes:
- Single-reference mode: When all segments have only one reference, no prefixing is used
- Multi-reference mode: When segments have multiple references, LAPIS names are prefixed with reference name (e.g., CV-A16-main, CV-A16-VP4)
This template is critical for generating the proper nucleotide sequences and genes for LAPIS

Website Frontend Changes

New Type Definitions (`website/src/types/referencesGenomes.ts`)

Introduced new types: SegmentFirstReferenceGenomes, ReferenceGenomesLightweightSchema
Defined clear type structure for segment-first organization
Added helper types for segment names, reference names, and gene names

New Components & Utilities

SegmentReferenceSelector.tsx (NEW, +155 lines)

Brand new UI component for selecting references on a per-segment basis
Features two display modes:
- Single segment: Simple dropdown selector
- Multi-segment: Tabbed interface where each tab represents a segment
Visual indicators show which segments have references selected
Includes clear button to deselect references

getSegmentAndGeneInfo.tsx (NEW, +57 lines)

New utility function to extract segment and gene information from the segment-first schema
Handles both single-reference and multi-reference modes
Returns structured data about available segments, genes, and whether organism is multi-segmented

isActiveForSelectedReferenceName.tsx (NEW, +17 lines)

Helper to determine if a metadata field should be active based on selected reference
Replaces previous isActiveForSelectedSuborganism logic

stillRequiresReferenceNameSelection.tsx (NEW, +12 lines)

Helper to check if reference selection is still needed for metadata fields
Part of the new reference selection workflow

Updated Components

Search Page Components (Multiple files, ~200 lines changed)

SearchForm.tsx, SearchFullUI.tsx: Integrated new SegmentReferenceSelector
DownloadDialog.tsx, DownloadForm.tsx: Updated to work with segment-first model
SuborganismSelector.tsx: Renamed internal references but maintains backward compatibility
MutationField.tsx: Updated to use new reference selection helpers

Sequence Details Page (Multiple files, ~150 lines changed)

getTableData.ts: Major refactoring to generate table data from segment-first structure
DataTable.tsx, SequenceDataUI.tsx: Updated to display data in new format
SequencesContainer.tsx: Updated to handle segment-first gene and segment info

Configuration & State Management

config.ts (+42/-24 lines): Added getReferenceGenomeLightweightSchema() function to transform heavy reference genome data into lightweight schema for UI
useSearchPageState.ts (+19/-1 lines): Added SegmentReferenceSelections type and state management
useUrlParamState.ts (+20/-3 lines): Enhanced to support new reference selection in URL parameters

Test Updates

All affected test files updated to use segment-first data structures
Test data migrated from suborganism-first to segment-first format
~400 lines of test updates across multiple test files

Renamed/Removed Files

Removed: getSuborganismSegmentAndGeneInfo.tsx and its tests (replaced by getSegmentAndGeneInfo.tsx)
These utilities were specific to the old suborganism-first model

Breaking Changes

This is a breaking change that requires:

Configuration migration: All values.yaml files must be restructured to use segment-first format
API/Schema changes: The metadata field onlyForSuborganism is replaced with onlyForReferenceName
State management: URL parameters and internal state now track references differently

Technical Benefits

More flexible reference management: Each segment can independently use different references
Clearer separation of concerns: Segment structure is separated from reference selection
Better UI/UX: Users can now select references per segment with visual feedback
More accurate biological representation: Multi-segmented organisms can have segment-specific references
Simplified single-reference mode: When only one reference exists, no prefixing is needed (cleaner)

Migration Notes

The PR maintains some backward compatibility through the selectedSuborganism concept in state management, but the underlying data structures have fundamentally changed. Existing deployments will need to:

Update their values.yaml reference genome configurations
Update any metadata fields using onlyForSuborganism to use onlyForReferenceName
Test the new segment reference selection UI

Files Changed Summary

55 files modified: +1,538 insertions, -972 deletions
4 new files added for segment-first functionality
2 files removed (old suborganism-specific utilities)
Heavy concentration of changes in:
- Configuration layer (Kubernetes templates and schemas)
- Type definitions and utilities
- Search and sequence details UI components
- Test files to match new data structures

anna-parker · 2025-12-16T09:21:27Z

Notes: we will need to add a check in prepro (probably also change the config for this) that we cannot have multiple references of the same segment as this is a segment duplication and we dont want to allow that per sample. (At the moment we would only error if there are multiple sequences with the same reference)

anna-parker · 2026-01-05T14:55:51Z

kubernetes/loculus/values.yaml

-            sequence: "[[URL:https://corneliusroemer.github.io/seqs/artefacts/sars-cov-2/ORF9b.fasta]]"
-          - name: "S"
-            sequence: "[[URL:https://corneliusroemer.github.io/seqs/artefacts/sars-cov-2/S.fasta]]"
+      main:


I really like this structure!

we discussed changing main, could singleSegment or genome be a better option?

anna-parker · 2026-01-05T14:58:19Z

kubernetes/loculus/values.yaml

+            "S":
+              sequence: "[[URL:https://corneliusroemer.github.io/seqs/artefacts/sars-cov-2/S.fasta]]"
+      E:
+        singleReference:


this is duplicated? I think it can be removed

👍 Yeah this looks borked

anna-parker · 2026-01-05T15:46:47Z

kubernetes/loculus/values.yaml

          preprocessing:
            args:
-              segment: CV-A16
+              segment: CV-A16-main


I find it a bit confusing to have a segment/reference structure but use reference-segment in LAPIS. I think we should probably use segment-reference and potentially drop the segment if it is main

anna-parker · 2026-01-05T15:59:41Z

kubernetes/loculus/templates/_merged-reference-genomes.tpl

-          {{- $lapisNucleotideSequences = append $lapisNucleotideSequences (dict
-            "name" (printf "%s-%s" $suborganismName $sequence.name)
-            "sequence" $sequence.sequence
+{{/* Extract all unique reference names from the first segment */}}


huh? I think this is also an error, we need to do this for every segment not just the first one... also I dont think we should add the reference as a prefix

The goals of the current PR were just to support the current code/set up in a flipped form. There are a bunch of other changes (such as this) that would be needed to fully complete the feature for some organisms.

anna-parker · 2026-01-05T16:00:54Z

preprocessing/nextclade/src/loculus_preprocessing/prepro.py

    nucleotide_insertions: dict[SegmentName, list[NucleotideInsertion]] = {}
    amino_acid_insertions: dict[GeneName, list[AminoAcidInsertion]] = {}

+    # For minimizer classification, transform segment names by appending "-main"


again we should remove this

anna-parker · 2026-01-05T16:02:50Z

website/src/components/ReviewPage/getSegmentAndGeneDisplayNameMap.tsx

+                mappingEntries.push([geneName, geneName]);
+            }
+        } else {
+            // Multiple references: use {reference}-{segment} format


again lets use {segment}-{reference} in the case there are multiple segments and references

anna-parker · 2026-01-05T16:04:51Z

website/src/components/SearchPage/DownloadDialog/DownloadForm.tsx

-        const { nucleotideSegmentNames, geneNames } = referenceGenomeLightweightSchema[SINGLE_REFERENCE];
+    const segments = Object.keys(referenceGenomeLightweightSchema.segments);
+
+    // Check if single reference mode


this is actually a bug I believe - different segments can have different numbers of references and this will not be handled correctly here if the first segment has one reference but others have multiple

As above, the code here is not intended to handle different segments with different numbers of references (but yeah, we'd want to extend to that)

no problem! I have started using this as a base for making the flip and I am adding these comments here - if you prefer I can start on a fully fresh branch but I was actually planning to make a PR with my changes off this one - which would you prefer?

I don't have strong feelings on best approach. I guess the concern about using this as a base would be that it is pretty rough and ready, but obvs it does help with speed!

sounds good! I might play around with this branch a bit more by adding my suggestions to see if I can get it working a bit better and then potentially split my changes into smaller, more managable PRs: #5799

anna-parker · 2026-01-05T16:05:43Z

website/src/components/SearchPage/ReferenceNameSelector.tsx

@@ -0,0 +1,2 @@
+// Re-export for backward compatibility
+export { SuborganismSelector as ReferenceNameSelector } from './SuborganismSelector.tsx';


I think we can just delete this actually

theosanderson added preview Triggers a deployment to argocd and removed preview Triggers a deployment to argocd labels Dec 15, 2025

loculus-project deleted a comment from claude bot Dec 15, 2025

theosanderson added preview Triggers a deployment to argocd and removed preview Triggers a deployment to argocd labels Dec 15, 2025

theosanderson changed the title ~~wip~~ demo: flip config to segment-then-reference Dec 15, 2025

theosanderson marked this pull request as draft December 15, 2025 17:08

corneliusroemer removed the preview Triggers a deployment to argocd label Jan 3, 2026

anna-parker reviewed Jan 5, 2026

View reviewed changes

theosanderson added 10 commits January 5, 2026 17:09

restructure

dccbc09

update

b0cfb66

fixup

e12b6dd

update

44fbcf8

fixup

ab00ab2

up

cef9fb8

fixup

bd24df4

fixup

9379f1d

upd

c5a74ee

up

8498009

theosanderson added 3 commits January 5, 2026 17:09

ok

76c0de8

ok

d2c2cee

u

75e0d94

anna-parker force-pushed the restructure branch from 7020ab0 to 75e0d94 Compare January 5, 2026 16:10

		@@ -0,0 +1,2 @@
		// Re-export for backward compatibility
		export { SuborganismSelector as ReferenceNameSelector } from './SuborganismSelector.tsx';

demo: flip config to segment-then-reference #5701

Are you sure you want to change the base?

demo: flip config to segment-then-reference #5701

Uh oh!

Conversation

theosanderson commented Dec 15, 2025 • edited by loculus-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Screenshot

PR Checklist

Uh oh!

theosanderson commented Dec 15, 2025

Uh oh!

claude bot commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of PR Changes

Core Architectural Change: Segment-First Reference Genome Structure

Previous Model (Suborganism-First)

New Model (Segment-First)

Key Changes by Component

Kubernetes Configuration (kubernetes/loculus/)

Website Frontend Changes

New Type Definitions (website/src/types/referencesGenomes.ts)

New Components & Utilities

Updated Components

Test Updates

Renamed/Removed Files

Breaking Changes

Technical Benefits

Migration Notes

Files Changed Summary

Uh oh!

anna-parker commented Dec 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

theosanderson commented Dec 15, 2025 •

edited by loculus-bot

Loading

claude bot commented Dec 15, 2025 •

edited

Loading

Kubernetes Configuration (`kubernetes/loculus/`)

New Type Definitions (`website/src/types/referencesGenomes.ts`)