docs: Add comprehensive b2luigi vs Dagster comparison page

[Copilot]

Adds a new documentation page comparing b2luigi and Dagster across capabilities, use-cases, and trade-offs.

Changes

• docs/comparison_dagster.rst — new page covering:

  • Feature-by-feature comparison tables (execution/scheduling, observability, parameters/partitioning, file management, deployment)
  • What b2luigi would need to match Dagster (assets+lineage, persistent run history, Dagit-equivalent UI, built-in cron/sensors, partitions, retry policies, IOManagers, type checking, freshness policies, integration marketplace)
  • What b2luigi uniquely offers that Dagster lacks (native HPC batch submission via LSF/HTCondor/SLURM/gbasf2, automatic parameter-derived output paths, ergonomic parameter sweeps via requires(), no daemon required, Apptainer/Singularity support, XRootD targets, LHC grid computing)
  • How Dagster handles heterogeneous workflows and where it falls short (no native HPC scheduler integration)
  • Common community complaints about Dagster (steep learning curve, paradigm churn ops→assets, heavy footprint, daemon burden, limited HPC support)
  • Summary table and tool-selection recommendation

• docs/index.rst — adds comparison_dagster to the toctree

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• docs.dagster.io
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)
• luigi.readthedocs.io
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

Generate me a comprehensive comparison of dagster and b2luigi. Find their documentation and compare their capabilities. What would be needed for b2luigi to mimik the features of dagster. What can b2luigi do, that dagster can't? How does dagster work with very inhomogenious workflows? What are common complaints about dagster?

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.