feat: implement analysis step and refactor pipeline to data-staging pattern#35
Open
kenibrewer wants to merge 43 commits intodevfrom
Open
feat: implement analysis step and refactor pipeline to data-staging pattern#35kenibrewer wants to merge 43 commits intodevfrom
kenibrewer wants to merge 43 commits intodevfrom
Conversation
Design for implementing the CellProfiler analysis step while refactoring the pipeline to use the data-staging pattern from nf-pooled-cellpainting. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
12-task plan covering full pipeline refactor to data-staging pattern plus new CellProfiler analysis module. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add stub mode to nf-test - Add stub block to CYTOTABLE module - Regenerate snapshots for new process structure Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ysis Assay development now always runs (as a QC step), not just in assay_development mode. Analysis + CYTOTABLE only run when cellprofiler_mode is 'analysis'. Test profile updated accordingly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Strip columns not defined in schema_input.json to eliminate nf-schema validation warnings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- assaydevelopment.cppipe from #30 - analysis.cppipe from cpg0000-jump-pilot (CPJUMP1_analysis_without_batchfile) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Allows --image-directory CLI flag to override the image path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Must match the -g flag passed on the CellProfiler CLI. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
….cppipe Match channel names from our samplesheet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adapted Base image location to work with --image-directory CLI flag. Channel names already match our samplesheet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bump to 8 CPUs, 24GB memory to accommodate CellProfiler analysis. Note: Docker Desktop must also be configured with sufficient memory (recommend >= 16GB in Settings > Resources). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MultiQC module updated to v1.33 with single tuple input. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
groupTuple() doesn't guarantee ordering, so the metadata JSON in the script block could vary between runs, breaking -resume caching. Sort by filename at the Nextflow level (for cache key stability) and in Python scripts (belt-and-suspenders). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Member
|
Warning Newer version of the nf-core template is available. Your pipeline is using an old version of the nf-core template: 3.5.1. For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation. |
|
…rflow Groovy's JsonOutput.toJson causes StackOverflowError on Nextflow meta maps in CI (smaller JVM stack). Extract only needed fields into plain maps before serializing. Also update illumination correction module test for new input signature and regenerate snapshots. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Extract sortGroupedImages closure to eliminate 3x duplicated sort block - Filter assay dev images to target site BEFORE groupTuple (avoids grouping all sites only to discard them) - Restore ch_multiqc_custom_config in MultiQC input (was silently dropped) - Single-pass channel collection in generate_illumination_apply_csv.py - Narrow glob pattern for .npy files (*_Illum*.npy) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace f-string escaped braces in error message to avoid false positive Jinja detection - Add design/plan docs to template_strings ignore list Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Design specs and plans are local development artifacts, not pipeline deliverables. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All CellProfiler modules must report the real version even in stub mode for proper version tracking. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Keep both docs/superpowers/ and .superset/ entries. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merge origin/TEMPLATE into feature branch. Resolves conflicts: - .nf-core.yml: update to v3.5.1, keep our lint config - modules/nf-core/multiqc: take template version (v1.32, old input sig) - subworkflows/local/utils_nfcore_cellpainting_pipeline: take template version with help params, keep our samplesheet parsing - Revert MULTIQC call to simple positional arguments - modules.json, ro-crate-metadata.json: take template versions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contains commands, architecture overview, key conventions, and template sync instructions for AI coding assistants. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Real run md5 unchanged. Stub md5 changed due to empty touch vs echo. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix nextflow_schema.json: correct ${projectDir] to ${projectDir},
fix analysis_cppipe path, add .jinja extension to illumination default
- Add enum validation for cellprofiler_mode (assay_development|analysis)
- Remove stop_after param (declared but never used)
- Remove conda directives from assaydevelopment.nf and analysis.nf
(no environment.yml in moduleDir)
- Remove unused CYTOTABLE import from main.nf
- Wire up PIPELINE_COMPLETION in main.nf
- Complete publishDir configs for all CellProfiler steps
- Add module-level stub tests for assaydevelopment, analysis, cytotable
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add AGENTS.md and CLAUDE.md to template_strings ignore list - Sync multiqc module to nf-core/modules@dfa7f1a (v1.32, matches modules.json tracking) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Restore multiqc module files to match af27af1be706 from nf-core/modules, which is what the TEMPLATE branch's modules.json tracks. Remove extra files (conda-lock, custom_prefix.config, tags.yml) that don't exist at that sha. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
maxulysse
reviewed
Apr 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implement the CellProfiler analysis step and refactor the entire pipeline to use the data-staging pattern from nf-pooled-cellpainting. Also syncs with nf-core TEMPLATE v3.5.1.
Feature changes
load_data.csvat runtime via Pythonbin/scripts reading metadata JSON (replaces pre-generated CSV subworkflows)cellprofiler_assaydevelopment_siteparameter for site selectioncellprofiler_modeenum validation (assay_development|analysis)PIPELINE_COMPLETIONsubworkflowstop_afterparameter.cppipepipeline files for assay development and analysispublishDirconfigurations for all processesFiles changed by feature work
New files:
bin/generate_illumination_calc_csv.py— CSV generator for illumination correction (reads metadata JSON)bin/generate_illumination_apply_csv.py— CSV generator for assay dev + analysis (paired orig/illum columns)assets/cellprofiler/analysis.cppipe— CellProfiler analysis pipeline (from ErinWeisbart/nf-core-test-datasets)assets/cellprofiler/assaydevelopment.cppipe— CellProfiler assay dev pipeline (from Add Assay Development to workflow #30)AGENTS.md/CLAUDE.md— development instructions for AI coding assistantsmodules/local/cellprofiler/analysis/tests/main.nf.test— stub testmodules/local/cellprofiler/assaydevelopment/tests/main.nf.test— stub testRewritten:
workflows/cellpainting.nf— inline channel grouping/joining, analysis + CYTOTABLE wiring, deterministic sortingmodules/local/cellprofiler/illuminationcorrection/main.nf— metadata JSON input, generates own CSVmodules/local/cellprofiler/assaydevelopment.nf— real CellProfiler command, data-staging patternmodules/local/cellprofiler/analysis.nf— full rewrite from samtools stub to per-site analysismodules/local/cytotable/main.nf— accepts directory input, added stub blockmain.nf— pass new params, wire PIPELINE_COMPLETION, remove unused importUpdated:
conf/modules.config— publishDir for all CellProfiler steps and cytotableconf/test.config— default to analysis modenextflow.config— addcellprofiler_assaydevelopment_site, removestop_afternextflow_schema.json— fix malformed defaults, add enum, add site param, remove stop_after.nf-core.yml— template_strings ignore list.gitignore— adddocs/superpowers/assets/samplesheet.csv— remove unused columnsmodules/local/cellprofiler/illuminationcorrection/tests/main.nf.test— updated input signaturemodules/local/cytotable/tests/main.nf.test— added stub testDeleted:
subworkflows/local/cellprofiler_load_data_csv/(replaced by inline channel ops + bin scripts)subworkflows/local/cellprofiler_load_data_csv_with_illum/(replaced by inline channel ops + bin scripts)Files changed by nf-core TEMPLATE sync (v3.3.2 → v3.5.1)
.devcontainer/devcontainer.json,.devcontainer/setup.sh.github/actions/nf-test/action.yml.github/workflows/(awsfulltest, awstest, clean-up, download_pipeline, fix_linting, linting, linting_comment, nf-test, release-announcements, template-version-comment).gitpod.yml(removed).pre-commit-config.yaml,.prettierignoreREADME.md,docs/usage.mdmodules.json,ro-crate-metadata.jsonmodules/nf-core/multiqc/(environment.yml, main.nf, tests/)subworkflows/nf-core/(utils_nextflow_pipeline, utils_nfcore_pipeline, utils_nfschema_plugin)subworkflows/local/utils_nfcore_cellpainting_pipeline/main.nf(added help params)tests/.nftignore,tests/default.nf.testCloses
Test plan
🤖 Generated with Claude Code