Asimov

Workflow orchestration for complex scientific analyses

A general-purpose framework for managing automated analysis pipelines on HPC clusters

Battle-tested in production by LIGO, Virgo, and KAGRA collaborations

Documentation · Installation · Quick Start · Contributing

What is Asimov?

Asimov is a general-purpose workflow orchestration platform designed to automate complex scientific analyses on high-throughput computing (HTC) clusters.

At its core, Asimov provides:

Pipeline-agnostic architecture - Coordinate multiple analysis tools through a unified interface
Intelligent job management - Submit, monitor, and recover jobs automatically
Configuration management - Version-controlled, reproducible analysis workflows
Results organization - Structured handling of outputs with validation

Designed for flexibility: While gravitational wave analysis is the primary use case, Asimov's architecture supports any scientific workflow involving:

Long-running computational jobs
Multiple analysis pipelines or tools
Systematic parameter studies
HTC cluster environments (HTCondor, Slurm)
Complex dependencies between analyses

Why Asimov?

Many scientific analyses involve running computationally intensive pipelines on HPC clusters—processes that can take days to weeks and involve hundreds of coordinated jobs. Common challenges include:

Manual configuration for each pipeline and analysis
Constant monitoring of job status across clusters
Error-prone restart and recovery procedures
Scattered results across different filesystems
Difficult reproducibility and collaboration
Pipeline lock-in - custom scripts tied to specific tools

Asimov solves this by providing:

✅ Pipeline-agnostic interface - One framework for multiple analysis tools ✅ Automated job orchestration - Submit, monitor, recover automatically ✅ Version-controlled configuration - Reproducible, auditable workflows ✅ Intelligent error handling - Detect and recover from failures ✅ Results management - Organized, validated outputs ✅ Scalability - From single analyses to large-scale campaigns

🚀 Key Features

🤖 Intelligent Job Management

Asimov interacts with high-throughput computing systems (HTCondor, Slurm) to submit jobs, monitor their progress, detect failures, and automatically initiate recovery or post-processing tasks.

🔌 Pipeline-Agnostic Architecture

Define analyses once, deploy across multiple pipelines. Asimov's extensible plugin system allows you to add support for new analysis tools without modifying the core framework. Current implementations include gravitational wave parameter estimation pipelines (LALInference, Bilby, BayesWave, RIFT), with support for custom pipelines.

📋 Centralized Configuration

All ongoing, completed, and scheduled analyses are recorded in version-controlled ledgers, making it easy to find jobs, configurations, and results. Every analysis is tracked and reproducible.

📊 Reporting & Monitoring

Generate both machine-readable and human-friendly reports of all monitored jobs, with automatic collation of log files, diagnostic outputs, and status summaries.

🗂️ Results Management

Special tools help manage analysis outputs, ensure data integrity, and organize results for publication and archival.

🔄 Workflow Automation

Set up continuous monitoring that automatically:

Starts jobs when dependencies are met
Initiates post-processing when jobs complete
Handles errors and resubmissions
Generates reports and summaries

🌊 Case Study: Gravitational Waves

Proven at scale in production: Asimov was developed by and is actively used by the LIGO, Virgo, and KAGRA collaborations for gravitational wave parameter estimation.

Real-World Impact

GWTC-2, GWTC-3, and GWTC-4.0 catalogs - Managed parameter estimation for 90+ gravitational wave events
O3 & O4 observing runs - Continuous automated analysis of new detections
Multiple pipelines - Unified interface for LALInference, Bilby, BayesWave, and RIFT
Production-ready - Battle-tested on hundreds of analyses over multiple years

This production use demonstrates Asimov's capability to handle:

Complex Bayesian inference (days to weeks per job)
Hundreds of jobs per event (different parameter configurations)
Catalog-scale workflows (90+ events × multiple pipelines)
Critical scientific output (published in major collaboration papers)

See USERS.md for institutions and research groups using Asimov.

🔬 Beyond Gravitational Waves

While gravitational wave analysis is the primary driver, Asimov's architecture is general-purpose. It's suitable for any scientific domain requiring:

Potential Applications

Astrophysics & Cosmology - Parameter estimation for transient events, population studies, systematic surveys
Climate Science - Model ensembles, parameter sweeps, multi-model comparisons
Particle Physics - Monte Carlo campaigns, detector simulations, systematic studies
Computational Chemistry - Molecular dynamics campaigns, conformational searches
Machine Learning - Hyperparameter sweeps, model training pipelines, ensemble methods
Any domain with:
- Long-running computational jobs (hours to weeks)
- Multiple analysis tools to coordinate
- Need for systematic, reproducible workflows
- HPC cluster infrastructure

Adding Your Pipeline

Asimov provides a plugin architecture for adding new analysis pipelines. If your field has established analysis tools that run on HPC clusters, they can be integrated with Asimov. See Contributing for information on extending Asimov to new domains.

📦 Installing Asimov

Asimov is written in Python and available on PyPI and conda-forge.

Via pip

pip install asimov

Via conda

conda install -c conda-forge asimov

Requirements

Asimov requires git to be installed and configured:

git config --global user.email "[email protected]"
git config --global user.name "Your Name"

Supported Python versions: 3.8+

⚡ Quick Start

Get started with Asimov in minutes. This example uses gravitational wave analysis:

1. Create a new project

mkdir my-analysis-project
cd my-analysis-project
asimov init "My Analysis Project"

2. Configure your workflow

# For gravitational wave analysis, you can use pre-configured templates:
asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/defaults/production-pe.yaml
asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/defaults/production-pe-priors.yaml

# For other domains, you'll create custom configuration files (see docs)

3. Add your analysis target

# Example: GW event
asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/events/gwtc-2-1/GW150914_095045.yaml

4. Define analyses

asimov apply -f https://git.ligo.org/asimov/data/-/raw/main/analyses/production-default.yaml -e GW150914_095045

5. Build and submit jobs

asimov manage build    # Generate pipeline-specific configuration files
asimov manage submit   # Submit to cluster

6. Monitor your jobs

# Check status once
asimov monitor

# Or start continuous monitoring (checks every 15 minutes)
asimov start

Asimov will automatically handle post-processing, restart failed jobs, and organize results when complete!

For non-GW applications: See the documentation for guides on adapting Asimov to your analysis workflows.

🤔 Do I Need Asimov?

You should consider Asimov if you:

Are running long-running analyses on HPC clusters (hours to weeks per job)
Need to coordinate multiple analysis pipelines or tools
Want reproducible, version-controlled analysis configurations
Are managing many jobs with complex dependencies
Need automated monitoring and error recovery
Are conducting systematic studies with parameter variations
Want to scale from exploratory analyses to production campaigns

Common Use Cases:

🌊 Parameter estimation - Bayesian inference, MCMC, nested sampling
📊 Systematic studies - Parameter sweeps, sensitivity analyses
📚 Large-scale campaigns - Catalogs, surveys, ensemble runs
🔬 Multi-tool comparisons - Compare results across different analysis pipelines
👥 Collaborative workflows - Shared, reproducible configurations
🤖 Automated pipelines - Continuous analysis of new data

Gravitational Wave Researchers: Asimov is production-ready with full support for LALInference, Bilby, BayesWave, and RIFT.

Other Scientific Domains: Asimov's architecture is extensible to other fields. See Beyond Gravitational Waves for potential applications.

📖 Example Workflow

Here's what a typical Asimov workflow looks like:

1. Initialize project       → asimov init
                            ↓
2. Configure defaults       → asimov apply (settings & priors)
                            ↓
3. Add events              → asimov apply (event data)
                            ↓
4. Add analyses            → asimov apply (analysis configs)
                            ↓
5. Build configs           → asimov manage build
                            ↓
6. Submit to cluster       → asimov manage submit
                            ↓
7. Monitor progress        → asimov start (automated)
                            ↓
8. [Automatic steps]
   - Job monitoring
   - Error detection & recovery
   - Post-processing (PESummary)
   - Results organization
                            ↓
9. Access results          → Organized in results/ directory

All configurations and job metadata are stored in .asimov/ledger.yml for full reproducibility.

🔧 Extensibility

Asimov's plugin architecture allows new analysis pipelines to be integrated without modifying the core framework.

Currently Supported Pipelines

Gravitational wave parameter estimation pipelines (production-ready):

Pipeline	Description	Status
LALInference	Nested sampling & MCMC parameter estimation	✅ Production
Bilby	Modern Bayesian inference framework	✅ Production
BayesWave	Unmodeled analysis & glitch subtraction	✅ Production
RIFT	Rapid parameter estimation	✅ Production

Adding New Pipelines

To add support for a new analysis tool:

Define the interface - Create a pipeline class implementing Asimov's base interface
Configuration mapping - Define how Asimov configurations map to your tool's inputs
Job management - Specify how to build, submit, and monitor jobs
Post-processing - Define result handling and validation

See the documentation and Contributing Guide for detailed instructions on adding pipeline support.

Future Extensions

Gravitic - Framework for experimental gravitational wave pipelines
Machine learning workflows - Training pipeline management
Custom pipelines - Community contributions for new domains

🗺️ Roadmap

Gravitic Pipelines

Support for pipelines constructed using gravitic, allowing experimental tools to be used without constructing entire new pipelines. This will also enable Asimov to manage the training of machine learning algorithms.

Workflow Replication & Extension

Allow existing workflows to be duplicated (similar to git clone), extended with new jobs that can access completed jobs from the parent workflow, and replicated with modifications for systematic studies.

Enhanced Reporting

Web-based dashboards for real-time job monitoring and analysis status visualization.

Multi-Messenger Astronomy

Improved support for joint electromagnetic and gravitational wave analyses, multi-wavelength campaigns.

🤝 Contributing

We welcome contributions from the community! Whether you're:

🐛 Reporting bugs
💡 Suggesting features
📝 Improving documentation
🔧 Adding pipeline support
🧪 Writing tests

Please see our Contributors' Guide to get started!

Ways to Contribute

Report issues on our Issue Tracker
Submit pull requests with improvements
Improve documentation and examples
Share your use cases and workflows
Add support for new analysis pipelines - especially from other scientific domains!
Contribute examples for non-GW applications

📚 Citation

If you use Asimov in your research, please cite it! This helps us demonstrate impact and secure continued development support.

BibTeX:

@ARTICLE{asimov-paper,
      author = {{Williams}, Daniel and {Veitch}, John and {Chiofalo}, Maria and {Schmidt}, Patricia and {Udall}, Rhiannon and {Vajpeyi}, Avi and {Hoy}, Charlie},
        title = "{Asimov: A framework for coordinating parameter estimation workflows}",
      journal = {The Journal of Open Source Software},
    keywords = {Python, astronomy, gravitational waves, General Relativity and Quantum Cosmology, Physics - Data Analysis, Statistics and Probability},
        year = 2023,
        month = apr,
      volume = {8},
      number = {84},
          eid = {4170},
        pages = {4170},
          doi = {10.21105/joss.04170},
archivePrefix = {arXiv},
      eprint = {2207.01468},
primaryClass = {gr-qc},
}

See CITATION.cff for more citation formats.

Publications using Asimov should also reference the relevant pipeline papers as appropriate for your domain.

👨‍💻 Authors

Asimov is developed by the LIGO, Virgo, and KAGRA collaborations.

Primary Maintainer: Daniel Williams (@transientlunatic)

Development Support: Science and Technology Facilities Council (STFC) and the Institute for Gravitational Research at the University of Glasgow.

Contributors

We're grateful to all our contributors! See the full list in CONTRIBUTORS.md.

📄 License

Asimov is released under the MIT License.

🔗 Links

Documentation: https://asimov.docs.ligo.org/asimov
Issue Tracker: https://git.ligo.org/asimov/asimov/-/issues
Releases: https://git.ligo.org/asimov/asimov/-/releases
PyPI: https://pypi.org/project/asimov/
Conda: https://anaconda.org/conda-forge/asimov

⭐ If Asimov helps your research, please consider starring the repository! ⭐

Made with ❤️ by the scientific computing community

Name		Name	Last commit message	Last commit date
Latest commit History 1,506 Commits
.github		.github
.gitlab		.gitlab
asimov		asimov
conda		conda
docs		docs
sandbox		sandbox
scripts		scripts
tests		tests
.gitlab-ci.yml		.gitlab-ci.yml
CHANGELOG.rst		CHANGELOG.rst
CITATION.cff		CITATION.cff
CODE-OF-CONDUCT.rst		CODE-OF-CONDUCT.rst
CODEOWNERS		CODEOWNERS
CONTRIBUTING.rst		CONTRIBUTING.rst
CONTRIBUTORS.md		CONTRIBUTORS.md
HISTORY.rst		HISTORY.rst
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
REVIEW.rst		REVIEW.rst
USERS.md		USERS.md
logo.svg		logo.svg
pyproject.toml		pyproject.toml

License

etive-io/asimov

Folders and files

Latest commit

History

Repository files navigation