You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- Document the collection convention of Daytime AP
- Document the new PP collection names.
- Un-specify pipelines-<hash>-config-<hash> of PP collection names
- Prompt Processing outputs of <pipeline> for <instrument> on day_obs YYYY-MM-DD with software and configurations in hashes.
95
+
- Prompt Processing outputs of <pipeline> for <instrument> on day_obs YYYYMMDD; ``<deployment-id>`` is a string identifying the software environment and configuration.
96
+
* - <instrument>/runs/daytimeAP-YYYYMMDD
97
+
- CHAINED
98
+
- Daily chain of Daytime AP outputs for <instrument> on day_obs YYYYMMDD (used for prompt publication).
99
+
* - <instrument>/runs/daytimeAP/YYYYMMDD/*
100
+
- RUN
101
+
- Daytime AP outputs for <instrument> on day_obs YYYYMMDD.
96
102
* - refcats
97
103
- CHAINED
98
104
- All reference catalogs (distinguished by dataset type).
@@ -251,7 +257,7 @@ All required input data for source injection, including input catalogs, are aggr
251
257
Shared/official processing outputs
252
258
----------------------------------
253
259
254
-
Official processing outputs fall into two main categories: Data Release Processing (DRP) and Prompt Processing.
260
+
Official processing outputs fall into two main categories: Data Release Processing (DRP) and Alert Production (AP).
255
261
256
262
DRP-style processing
257
263
^^^^^^^^^^^^^^^^^^^^
@@ -299,38 +305,46 @@ that references (at the level of individual datasets) the ``first`` and ``second
299
305
It is also worth noting that in general the full DAG does not maintain the usual collection invariant of having only one dataset with a particular dataset type and data ID (e.g. two calexps with the same data ID, from two differently-configured runs, could each contribute to different, non-conflicting coadd patches in downstream runs).
300
306
301
307
302
-
Prompt Processing
303
-
^^^^^^^^^^^^^^^^^
308
+
AP-style processing
309
+
^^^^^^^^^^^^^^^^^^^
310
+
311
+
Alert Production pipeline is run in Prompt Processing and Daytime AP campaigns.
304
312
305
313
Prompt Processing runs are automated processing runs that execute on incoming observational data in near real time.
306
314
Unlike DRP-style processing, Prompt Processing operates continuously and autonomously: for each observation day, the service automatically processes all data matching configured criteria (such as specific survey programs) as the data arrives.
307
-
While Prompt Processing runs within a specific software release environment (deployed as a container image), the collection naming convention uses hash-based identifiers rather than explicit release versions and ticket numbers.
315
+
While Prompt Processing runs within a specific software release environment (deployed as a container image), the collection naming convention uses a deployment identifier rather than explicit release versions and ticket numbers.
308
316
This approach reflects the operational nature of the service: a single deployment may process data across multiple nights, and different releases may be deployed mid-observation in response to changing conditions or emergencies.
309
-
In this context, the precise software and configuration state, captured by the hash values, provides more suitable provenance than a nominal release label or ticket number would.
310
317
311
318
Prompt Processing output ``RUN`` collections follow the naming pattern:
* ``YYYY-MM-DD`` represents the observation day (``day_obs``)
318
-
* ``<pipeline>`` identifies the specific pipeline executed
319
-
* ``<package-hash>`` is a hash of the complete software environment, including all Python package versions in the active Science Pipelines installation and the Prompt Processing code itself (pipeline definitions, configuration files, and source code)
320
-
* ``<config-hash>`` is a hash of the runtime configuration, including the APDB configuration and other pipeline-specific settings
324
+
* ``YYYYMMDD`` represents the observation day (``day_obs``)
325
+
* ``<pipeline>`` identifies the specific pipeline executed (e.g. ``ApPipe``, ``SingleFrame``)
326
+
* ``<deployment-id>`` is a string identifying the software environment and configuration; it is unique per deployment
327
+
328
+
An example Prompt Processing ``RUN`` collection name is ``LSSTCam/runs/prompt/20240315/ApPipe/pipelines-a1b2c3d-config-e4f5g6h``.
329
+
330
+
Daytime AP (the daytime Alert Production catchup processing) runs after each night via BPS and processes data that was not handled in real time.
331
+
Its output ``RUN`` collections are named under the prefix ``<instrument>/runs/daytimeAP/YYYYMMDD/``.
332
+
333
+
For each observation day, a ``CHAINED`` collection aggregates all Prompt Processing and Daytime AP output ``RUN`` collections (outputs only):
334
+
335
+
``<instrument>/runs/prompt-YYYYMMDD``
321
336
322
-
This naming scheme ensures that each Prompt Processing run produces a uniquely identifiable output collection with full provenance traceability.
323
-
The embedded hash values guarantee that any change to either the software environment or the runtime configuration will result in a new, distinct collection name, enabling precise reproducibility and change tracking.
337
+
Daytime AP output ``RUN`` collections are prepended to this chain when the Daytime AP run completes, so the chain always reflects the most recent results.
338
+
An example daily chain is ``LSSTCam/runs/prompt-20240315``.
324
339
325
-
An example Prompt Processing ``RUN`` collection name is ``LSSTCam/prompt/output-2024-03-15/ApPipe/pipelines-a1b2c3d-config-e4f5g6h``.
340
+
An additional chain containing only Daytime AP outputs is also created for use by the Prompt Publication Service:
326
341
327
-
For each observation day, a ``CHAINED`` collection is created to aggregate all output ``RUN`` collection from Prompt Processing.
342
+
``<instrument>/runs/daytimeAP-YYYYMMDD``
328
343
329
-
An example Prompt Processing daily chain is ``LSSTCam/prompt/output-2024-03-15``.
344
+
General users (anything other than prompt publication) should use the ``<instrument>/runs/prompt-YYYYMMDD`` chain.
330
345
331
-
These collection naming conventions are specific to real-time Prompt Processing.
332
-
Collection conventions for Daytime AP Catchup processing are currently under
333
-
discussion in RFC-1159.
346
+
These naming conventions were adopted in RFC-1159 and took effect starting ``day_obs`` 2026-04-09.
347
+
Collections created before that date retain the old naming scheme (``<instrument>/prompt/output-YYYY-MM-DD``).
0 commit comments