Enhance transformers to preserve structured metadata and alarms in outputs; update documentation and examples accordingly.

MatInGit · MatInGit · commit eb6aa48ec706 · 2026-02-18T15:45:54.000Z
diff --git a/docs/source/interfaces.rst b/docs/source/interfaces.rst
@@ -91,6 +91,8 @@ Alarm Behavior
   ``NO_ALARM=0``, ``HIHI=3``, ``HIGH=4``, ``LOLO=5``, ``LOW=6``.
 - Explicit ``alarm`` payload overrides computed alarm.
 - Non-scalars do not compute alarms, but explicit ``alarm`` payloads are accepted.
+- ``p4p`` client attempts a structured put first; if the target rejects it, it
+  retries with a value-only put.
 
 Model Alarm Override
 ~~~~~~~~~~~~~~~~~~~~
@@ -99,6 +101,11 @@ Models can publish structured output with explicit alarm fields (for example
 ``{"PV": {"value": 1.0, "alarm": {...}}}``), and ``ModelObserver`` preserves
 that structure when publishing downstream.
 
+In ``examples/base/local/deployment_config_p4p_alarm.yaml`` this passes through
+an ``output_transformer`` direct-symbol mapping
+(``ML:LOCAL:TEST_S -> ML:LOCAL:TEST_S``), which preserves ``alarm`` and other
+non-``value`` fields.
+
 See example:
 
 - ``examples/base/local/deployment_config_p4p_alarm.yaml``
diff --git a/docs/source/transformers.rst b/docs/source/transformers.rst
@@ -5,15 +5,168 @@ This page explains how to use transformers in your project.
 
 Overview
 --------
-Transformers are used to transform data from one format to another...
+Transformers convert message payloads from one shape to another before model
+evaluation or interface publication.
+
+All transformer modules follow the
+:class:`~poly_lithic.src.transformers.BaseTransformer.BaseTransformer` contract
+and expose:
+
+- ``handler(pv_name, value)`` to receive incoming struct data
+- ``transform()`` to produce output values
 
 Available Transformers
 ----------------------
+
 - :class:`~poly_lithic.src.transformers.BaseTransformers.SimpleTransformer`
 - :class:`~poly_lithic.src.transformers.BaseTransformers.CAImageTransfomer`
 - :class:`~poly_lithic.src.transformers.BaseTransformers.PassThroughTransformer`
 - :class:`~poly_lithic.src.transformers.CompoundTransformer.CompoundTransformer`
 
-Examples
---------
-Here are some examples of how to use transformers...
+Transformer Configs
+-------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 24 46 30
+
+   * - Module
+     - Description
+     - YAML reference
+   * - ``SimpleTransformer``
+     - Scalar/array formula transform from input symbols to output variables
+     - See sample below
+   * - ``CAImageTransformer``
+     - Reconstruct image arrays from flattened channel + X + Y inputs
+     - See sample below
+   * - ``PassThroughTransformer``
+     - Relabel and forward values without numeric transformation
+     - See sample below
+   * - ``CompoundTransformer``
+     - Run multiple transformers in one module
+     - See sample below
+
+Metadata Propagation
+--------------------
+
+Transformer outputs may be either plain values or structured payloads with
+``value`` and additional fields:
+
+- ``PassThroughTransformer`` preserves non-``value`` fields (for example
+  ``alarm``, ``timestamp``, ``metadata``) when they are present in input
+  structs.
+- ``SimpleTransformer`` preserves non-``value`` fields only for direct-symbol
+  formulas (for example ``OUT = IN``). Computed formulas emit value-only output.
+- ``TransformerObserver`` forwards structured outputs unchanged and wraps plain
+  outputs as ``{"value": ...}``.
+
+This makes direct pass-through paths suitable for carrying alarm payloads across
+transformer stages.
+
+``SimpleTransformer`` Sample Configuration
+------------------------------------------
+
+.. code-block:: yaml
+
+   modules:
+     input_transformer:
+       name: "input_transformer"
+       type: "transformer.SimpleTransformer"
+       pub: "model_input"
+       sub:
+         - "system_input"
+       module_args: None
+       config:
+         symbols:
+           - "LUME:MLFLOW:TEST_B"
+           - "LUME:MLFLOW:TEST_A"
+         variables:
+           x2:
+             formula: "LUME:MLFLOW:TEST_B"
+           x1:
+             formula: "LUME:MLFLOW:TEST_A"
+
+``CAImageTransformer`` Sample Configuration
+-------------------------------------------
+
+.. code-block:: yaml
+
+   modules:
+     image_transformer:
+       name: "image_transformer"
+       type: "transformer.CAImageTransformer"
+       pub: "model_input"
+       sub:
+         - "update"
+       module_args: None
+       config:
+         variables:
+           img_1:
+             img_ch: "MY_TEST_CA"
+             img_x_ch: "MY_TEST_CA_X"
+             img_y_ch: "MY_TEST_CA_Y"
+           img_2:
+             img_ch: "MY_TEST_C2"
+             img_x_ch: "MY_TEST_CA_X2"
+             img_y_ch: "MY_TEST_CA_Y2"
+
+``PassThroughTransformer`` Sample Configuration
+-----------------------------------------------
+
+.. code-block:: yaml
+
+   modules:
+     output_transformer:
+       name: "output_transformer"
+       type: "transformer.PassThroughTransformer"
+       pub: "system_output"
+       sub:
+         - "model_output"
+       module_args: None
+       config:
+         variables:
+           LUME:MLFLOW:TEST_IMAGE: "y_img"
+
+``CompoundTransformer`` Sample Configuration
+--------------------------------------------
+
+.. caution::
+
+   This module may be deprecated in the future because the pub-sub model can
+   replace most compound-transformer use cases.
+
+.. code-block:: yaml
+
+   modules:
+     compound_transformer:
+       name: "compound_transformer"
+       type: "transformer.CompoundTransformer"
+       pub: "model_input"
+       sub:
+         - "update"
+       module_args: None
+       config:
+         transformers:
+           transformer_1:
+             type: "SimpleTransformer"
+             config:
+               symbols:
+                 - "MY_TEST_A"
+                 - "MY_TEST_B"
+               variables:
+                 x2:
+                   formula: "MY_TEST_A*2"
+                 x1:
+                   formula: "MY_TEST_B+MY_TEST_A"
+           transformer_2:
+             type: "CAImageTransformer"
+             config:
+               variables:
+                 img_1:
+                   img_ch: "MY_TEST_CA"
+                   img_x_ch: "MY_TEST_CA_X"
+                   img_y_ch: "MY_TEST_CA_Y"
+                 img_2:
+                   img_ch: "MY_TEST_C2"
+                   img_x_ch: "MY_TEST_CA_X2"
+                   img_y_ch: "MY_TEST_CA_Y2"
diff --git a/examples/base/local/deployment_config_p4p_alarm.yaml b/examples/base/local/deployment_config_p4p_alarm.yaml
@@ -1,6 +1,6 @@
 deployment:
   type: "continuous"
-  rate: .1 #seconds
+  rate: .001 #seconds
 
 modules:
   p4p_server:
@@ -10,7 +10,7 @@ modules:
       - "in_interface"
     sub:
       - "get_all"
-      - "model_out"
+      - "out_transformer"
     module_args: None
     config:
       EPICS_PVA_NAME_SERVERS: "localhost:5075"
@@ -125,3 +125,18 @@ modules:
       variables:
         max:
           type: "scalar"
+
+  output_transformer:
+    name: "output_transformer"
+    type: "transformer.SimpleTransformer"
+    pub: "out_transformer"
+    sub: "model_out"
+    module_args: None
+    config:
+      symbols:
+        - "ML:LOCAL:TEST_S"
+      variables:
+        # Direct-symbol mapping intentionally preserves non-value fields
+        # (e.g. alarm/timestamp/metadata) from model_out.
+        ML:LOCAL:TEST_S:
+          formula: "ML:LOCAL:TEST_S"
diff --git a/poly_lithic/src/transformers/BaseTransformers.py b/poly_lithic/src/transformers/BaseTransformers.py
@@ -31,13 +31,21 @@ def __init__(self, config):
         for key, value in self.pv_mapping.items():
             self.__validate_formulas(value['formula'])
         self.latest_input = {symbol: None for symbol in self.input_list}
+        self.latest_input_struct = {symbol: None for symbol in self.input_list}
         self.latest_transformed = {key: 0 for key in self.pv_mapping.keys()}
         self.updated = False
         self.handler_time = None
         self.formulas = {}
         self.lambdified_formulas = {}
+        self.direct_formula_inputs = {}
+        self.renamed_symbol_lookup = {
+            symbol.replace(':', '_'): symbol for symbol in self.input_list
+        }
         for key, value in self.pv_mapping.items():
             self.formulas[key] = sp.sympify(value['formula'].replace(':', '_'))
+            self.direct_formula_inputs[key] = self.__get_direct_input_symbol(
+                self.formulas[key]
+            )
             input_list_renamed = [
                 symbol.replace(':', '_') for symbol in self.input_list
             ]
@@ -53,11 +61,17 @@ def __validate_formulas(self, formula: str):
         except Exception as e:
             raise Exception(f'Invalid formula: {formula}: {e}')
 
+    def __get_direct_input_symbol(self, formula_expr):
+        if isinstance(formula_expr, sp.Symbol):
+            return self.renamed_symbol_lookup.get(str(formula_expr))
+        return None
+
     def handler(self, pv_name, value):
         # logger.debug(f"SimpleTransformer handler for {pv_name} with value {value}")
 
         # chek if pv_name is in sel.input_list
         if pv_name in self.input_list:
+            self.latest_input_struct[pv_name] = dict(value)
             # assert value is float
             try:
                 if isinstance(value['value'], (float, int, np.float32)):
@@ -161,6 +175,24 @@ def transform(self):
                 raise e
 
         for key, value in transformed.items():
+            direct_input = self.direct_formula_inputs.get(key)
+            input_struct = (
+                self.latest_input_struct.get(direct_input)
+                if direct_input is not None
+                else None
+            )
+            if isinstance(input_struct, dict):
+                passthrough_fields = {
+                    field_name: field_value
+                    for field_name, field_value in input_struct.items()
+                    if field_name != 'value'
+                }
+                if passthrough_fields:
+                    self.latest_transformed[key] = {
+                        'value': value,
+                        **passthrough_fields,
+                    }
+                    continue
             self.latest_transformed[key] = value
         self.updated = True
 
@@ -239,12 +271,14 @@ def __init__(self, config):
         # config is a dictionary of output:intput pairs
         pv_mapping = config['variables']
         self.latest_input = {}
+        self.latest_input_struct = {}
         self.latest_transformed = {}
         self.updated = False
         self.input_list = list(pv_mapping.values())
 
         for key, value in pv_mapping.items():
             self.latest_input[value] = None
+            self.latest_input_struct[value] = None
             self.latest_transformed[key] = None
         self.pv_mapping = pv_mapping
 
@@ -253,6 +287,7 @@ def __init__(self, config):
     def handler(self, pv_name, value):
         time_start = time.time()
         logger.debug(f'PassThroughTransformer handler for {pv_name}')
+        self.latest_input_struct[pv_name] = dict(value)
         self.latest_input[pv_name] = value['value']
         if all([value is not None for value in self.latest_input.values()]):
             self.transform()
@@ -263,10 +298,31 @@ def handler(self, pv_name, value):
     def transform(self):
         logger.debug('Transforming')
         for key, value in self.pv_mapping.items():
-            self.latest_transformed[key] = self.latest_input[value]
+            input_value = self.latest_input[value]
+            input_struct = self.latest_input_struct.get(value)
+            if isinstance(input_struct, dict):
+                passthrough_fields = {
+                    field_name: field_value
+                    for field_name, field_value in input_struct.items()
+                    if field_name != 'value'
+                }
+                if passthrough_fields:
+                    self.latest_transformed[key] = {
+                        'value': input_value,
+                        **passthrough_fields,
+                    }
+                else:
+                    self.latest_transformed[key] = input_value
+            else:
+                self.latest_transformed[key] = input_value
 
-            if isinstance(self.latest_input[value], np.ndarray):
-                if self.latest_input[value].shape != self.latest_transformed[key].shape:
+            transformed_value = (
+                self.latest_transformed[key]['value']
+                if isinstance(self.latest_transformed[key], dict)
+                else self.latest_transformed[key]
+            )
+            if isinstance(input_value, np.ndarray):
+                if input_value.shape != transformed_value.shape:
                     logger.error(f'Shape mismatch between input and output for {key}')
         self.updated = True
 
diff --git a/poly_lithic/src/utils/messaging.py b/poly_lithic/src/utils/messaging.py
@@ -261,7 +261,10 @@ def update(self, message: Message) -> Message | list[Message]:
             values = self.transformer.latest_transformed
             message_dict = {}
             for key, value in values.items():
-                message_dict[key] = {'value': value}
+                if isinstance(value, dict) and 'value' in value:
+                    message_dict[key] = value
+                else:
+                    message_dict[key] = {'value': value}
 
             self.transformer.updated = False
             return Message(topic=self.topic, source=str(self), value=message_dict)
diff --git a/readme.md b/readme.md
@@ -327,6 +327,9 @@ return {
 ```
 
 This is supported by ``ModelObserver`` and passed through to interfaces.
+In ``examples/base/local/deployment_config_p4p_alarm.yaml`` this now goes through an
+``output_transformer`` direct-symbol mapping (``ML:LOCAL:TEST_S -> ML:LOCAL:TEST_S``),
+which preserves ``alarm`` and other non-``value`` fields.
 See runnable example:
 
 - config: ``examples/base/local/deployment_config_p4p_alarm.yaml``
diff --git a/tests/test_messaging.py b/tests/test_messaging.py
diff --git a/tests/test_transformers.py b/tests/test_transformers.py
