Skip to content

Commit 2ce11ff

Browse files
committed
Update for RFC-1159 changes
- Document the collection convention of Daytime AP - Document the new PP collection names. - Un-specify pipelines-<hash>-config-<hash> of PP collection names
1 parent ce66e45 commit 2ce11ff

1 file changed

Lines changed: 36 additions & 22 deletions

File tree

index.rst

Lines changed: 36 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -87,12 +87,18 @@ The naming patterns for collections proposed here are summarized in :ref:`table-
8787
* - [<instrument>/]runs/<target>/<release>/<ticket>/*
8888
- unspecified
8989
- Private intermediates of processing data <target> with <release> on <ticket>.
90-
* - <instrument>/prompt/output-YYYY-MM-DD
90+
* - <instrument>/runs/prompt-YYYYMMDD
9191
- CHAINED
92-
- Prompt Processing outputs for <instrument> on day_obs YYYY-MM-DD.
93-
* - <instrument>/prompt/output-YYYY-MM-DD/<pipeline>/pipelines-<package-hash>-config-<config-hash>
92+
- Daily chain aggregating all Prompt Processing and Daytime AP output RUN collections (no inputs) for <instrument> on day_obs YYYYMMDD.
93+
* - <instrument>/runs/prompt/YYYYMMDD/<pipeline>/<deployment-id>
9494
- RUN
95-
- Prompt Processing outputs of <pipeline> for <instrument> on day_obs YYYY-MM-DD with software and configurations in hashes.
95+
- Prompt Processing outputs of <pipeline> for <instrument> on day_obs YYYYMMDD; ``<deployment-id>`` is a string identifying the software environment and configuration.
96+
* - <instrument>/runs/daytimeAP-YYYYMMDD
97+
- CHAINED
98+
- Daily chain of Daytime AP outputs for <instrument> on day_obs YYYYMMDD (used for prompt publication).
99+
* - <instrument>/runs/daytimeAP/YYYYMMDD/*
100+
- RUN
101+
- Daytime AP outputs for <instrument> on day_obs YYYYMMDD.
96102
* - refcats
97103
- CHAINED
98104
- All reference catalogs (distinguished by dataset type).
@@ -251,7 +257,7 @@ All required input data for source injection, including input catalogs, are aggr
251257
Shared/official processing outputs
252258
----------------------------------
253259

254-
Official processing outputs fall into two main categories: Data Release Processing (DRP) and Prompt Processing.
260+
Official processing outputs fall into two main categories: Data Release Processing (DRP) and Alert Production (AP).
255261

256262
DRP-style processing
257263
^^^^^^^^^^^^^^^^^^^^
@@ -299,38 +305,46 @@ that references (at the level of individual datasets) the ``first`` and ``second
299305
It is also worth noting that in general the full DAG does not maintain the usual collection invariant of having only one dataset with a particular dataset type and data ID (e.g. two calexps with the same data ID, from two differently-configured runs, could each contribute to different, non-conflicting coadd patches in downstream runs).
300306

301307

302-
Prompt Processing
303-
^^^^^^^^^^^^^^^^^
308+
AP-style processing
309+
^^^^^^^^^^^^^^^^^^^
310+
311+
Alert Production pipeline is run in Prompt Processing and Daytime AP campaigns.
304312

305313
Prompt Processing runs are automated processing runs that execute on incoming observational data in near real time.
306314
Unlike DRP-style processing, Prompt Processing operates continuously and autonomously: for each observation day, the service automatically processes all data matching configured criteria (such as specific survey programs) as the data arrives.
307-
While Prompt Processing runs within a specific software release environment (deployed as a container image), the collection naming convention uses hash-based identifiers rather than explicit release versions and ticket numbers.
315+
While Prompt Processing runs within a specific software release environment (deployed as a container image), the collection naming convention uses a deployment identifier rather than explicit release versions and ticket numbers.
308316
This approach reflects the operational nature of the service: a single deployment may process data across multiple nights, and different releases may be deployed mid-observation in response to changing conditions or emergencies.
309-
In this context, the precise software and configuration state, captured by the hash values, provides more suitable provenance than a nominal release label or ticket number would.
310317

311318
Prompt Processing output ``RUN`` collections follow the naming pattern:
312319

313-
``<instrument>/prompt/output-YYYY-MM-DD/<pipeline>/pipelines-<package-hash>-config-<config-hash>``
320+
``<instrument>/runs/prompt/YYYYMMDD/<pipeline>/<deployment-id>``
314321

315322
where:
316323

317-
* ``YYYY-MM-DD`` represents the observation day (``day_obs``)
318-
* ``<pipeline>`` identifies the specific pipeline executed
319-
* ``<package-hash>`` is a hash of the complete software environment, including all Python package versions in the active Science Pipelines installation and the Prompt Processing code itself (pipeline definitions, configuration files, and source code)
320-
* ``<config-hash>`` is a hash of the runtime configuration, including the APDB configuration and other pipeline-specific settings
324+
* ``YYYYMMDD`` represents the observation day (``day_obs``)
325+
* ``<pipeline>`` identifies the specific pipeline executed (e.g. ``ApPipe``, ``SingleFrame``)
326+
* ``<deployment-id>`` is a string identifying the software environment and configuration; it is unique per deployment
327+
328+
An example Prompt Processing ``RUN`` collection name is ``LSSTCam/runs/prompt/20240315/ApPipe/pipelines-a1b2c3d-config-e4f5g6h``.
329+
330+
Daytime AP (the daytime Alert Production catchup processing) runs after each night via BPS and processes data that was not handled in real time.
331+
Its output ``RUN`` collections are named under the prefix ``<instrument>/runs/daytimeAP/YYYYMMDD/``.
332+
333+
For each observation day, a ``CHAINED`` collection aggregates all Prompt Processing and Daytime AP output ``RUN`` collections (outputs only):
334+
335+
``<instrument>/runs/prompt-YYYYMMDD``
321336

322-
This naming scheme ensures that each Prompt Processing run produces a uniquely identifiable output collection with full provenance traceability.
323-
The embedded hash values guarantee that any change to either the software environment or the runtime configuration will result in a new, distinct collection name, enabling precise reproducibility and change tracking.
337+
Daytime AP output ``RUN`` collections are prepended to this chain when the Daytime AP run completes, so the chain always reflects the most recent results.
338+
An example daily chain is ``LSSTCam/runs/prompt-20240315``.
324339

325-
An example Prompt Processing ``RUN`` collection name is ``LSSTCam/prompt/output-2024-03-15/ApPipe/pipelines-a1b2c3d-config-e4f5g6h``.
340+
An additional chain containing only Daytime AP outputs is also created for use by the Prompt Publication Service:
326341

327-
For each observation day, a ``CHAINED`` collection is created to aggregate all output ``RUN`` collection from Prompt Processing.
342+
``<instrument>/runs/daytimeAP-YYYYMMDD``
328343

329-
An example Prompt Processing daily chain is ``LSSTCam/prompt/output-2024-03-15``.
344+
General users (anything other than prompt publication) should use the ``<instrument>/runs/prompt-YYYYMMDD`` chain.
330345

331-
These collection naming conventions are specific to real-time Prompt Processing.
332-
Collection conventions for Daytime AP Catchup processing are currently under
333-
discussion in RFC-1159.
346+
These naming conventions were adopted in RFC-1159 and took effect starting ``day_obs`` 2026-04-09.
347+
Collections created before that date retain the old naming scheme (``<instrument>/prompt/output-YYYY-MM-DD``).
334348

335349
.. _collections-developer-processing-outputs:
336350

0 commit comments

Comments
 (0)