Add 'Source-Agnostic Processing' feature to the homepage with detailed description #111

Author: ziagham

docs/reference/flowsynx/expressions.mdx

@@ -1,203 +1,265 @@
 ---
 sidebar_position: 3
 title: Expressions Syntax
-description: Learn how to use expressions in FlowSynx workflows to reference outputs, variables, and nested data dynamically during workflow execution.
+description: FlowSynx expression system for dynamic access to variables, task outputs, nested data, arithmetic, conditionals, boolean logic, and functional aggregation.
 ---
 
 # FlowSynx Expression Syntax
 
-FlowSynx supports a flexible **expression system** that allows you to reference **workflow outputs**, **variables**, and **nested properties** dynamically within workflow definitions.  
-Expressions are resolved at runtime by the `ExpressionParser` engine.
+FlowSynx provides a powerful expression system for dynamically resolving **workflow variables**, **task outputs**, **nested properties**, **arithmetic**, **boolean logic**, **conditional (ternary) operations**, and **functional aggregations** at runtime.  
+All expressions are evaluated by the `ExpressionParser`.
 
-## Overview
-
-Expressions are enclosed within the syntax:
+## Core Delimiter
 
+Wrap any resolvable content in:
 ```
 $[...]
 ```
 
-Anything inside `$[...]` is evaluated by the expression parser.
+If the entire value is a single `$[...]` block, the resolved object is returned directly (e.g. an array or JSON object). If embedded inside a larger string, the resolved value is converted to a string and concatenated.
+
+## Data Sources
+
+| Source | Syntax | Description |
+| ------ | ------ | ----------- |
+| Variables | `$[Variables('Key')]` | Workflow-level variable lookup (case-insensitive). |
+| Outputs | `$[Outputs('TaskName')]` | Output of a previously executed task (case-insensitive). |
 
-You can use this syntax anywhere in your workflow — for example in **task parameters**, **paths**, or **input data** fields.
+Keys with spaces or special characters should use single quotes: `$[Variables('My Custom Key')]`.
 
-## Expression Sources
+### Outputs as PluginContext
 
-FlowSynx supports two main data sources inside expressions:
+The [`Outputs`](command:_github.copilot.openSymbolFromReferences?%5B%22%22%2C%5B%7B%22uri%22%3A%7B%22scheme%22%3A%22file%22%2C%22authority%22%3A%22%22%2C%22path%22%3A%22%2Fc%3A%2Fgithub%2Fflowsynx%2Fdocs%2Fdocs%2Freference%2Fflowsynx%2Fexpressions.mdx%22%2C%22query%22%3A%22%22%2C%22fragment%22%3A%22%22%7D%2C%22pos%22%3A%7B%22line%22%3A25%2C%22character%22%3A2%7D%7D%5D%2C%2203b219c3-6667-4007-919c-1aa2e2101390%22%5D "Go to definition") object provides access to the following properties:
 
-| Source | Syntax Example | Description |
-| ------- | --------------- | ----------- |
-| **Variables** | `$[Variables('InputPath')]` | References workflow-level variables defined under `"Variables"` in the workflow JSON. |
-| **Outputs** | `$[Outputs('TaskName')]` | References the output of a previously executed task. |
+| Property         | Description                                                                 |
+| ---------------- | --------------------------------------------------------------------------- |
+| `Id`             | The unique identifier of the plugin context.                               |
+| `SourceType`     | The type of source that generated the output.                               |
+| `Format`         | The format of the output, if applicable.                                    |
+| `Metadata`       | A dictionary containing metadata as key-value pairs.                       |
+| `RawData`        | The raw binary data of the output, if available.                            |
+| `Content`        | The string content of the output, if available.                             |
+| `StructuredData` | A list of dictionaries representing structured data, if applicable.         |
 
-Both can be used in nested or concatenated contexts.
+Keys with spaces or special characters should use single quotes: `$[Variables('My Custom Key')]`.
 
-## Basic Examples
+#### Example
 
-### Accessing Workflow Variables
+##### Accessing Metadata
+You can retrieve specific metadata values using dot notation:
 
-```jsonc
-"Parameters": {
-  "Path": "$[Variables('InputPath')]"
-}
+```
+$[Outputs('TaskName').Metadata['Key']]
 ```
 
-If your workflow defines:
+##### Accessing Content
+To retrieve the content of the output:
 
-```jsonc
-"Variables": {
-  "InputPath": "s3://bucket/input"
-}
+```
+$[Outputs('TaskName').Content]
 ```
 
-The parser will replace `$[Variables('InputPath')]` with:
+##### Accessing Structured Data
+You can traverse the StructuredData property to access specific elements:
 
 ```
-s3://bucket/input
+$[Outputs('TaskName').StructuredData[0]['FieldName']]
 ```
 
-### Accessing Task Outputs
+##### Combining Properties
+You can combine multiple properties in expressions:
 
-```jsonc
-"Parameters": {
-  "Data": "$[Outputs('ExtractData')]"
-}
+```
+$[Outputs('TaskName').Id + ' - ' + Outputs('TaskName').SourceType]
+```
+
+#### Notes on PluginContext
+- Metadata keys are case-insensitive.
+- RawData is a binary array and cannot be directly used in string concatenation. You may need to convert it to a string or process it separately.
+- StructuredData is a list of dictionaries, allowing traversal and dynamic access to nested fields.
+
+#### Updated Examples
+Here are some updated examples reflecting the PluginContext structure:
+
+```
+$[Outputs('Task').Id]
+$[Outputs('Task').Metadata['Category']]
+$[Outputs('Task').StructuredData[0]['Name']]
+$[Contains(Outputs('Task').Metadata, 'urgent')]
+$[Outputs('Task').Content == 'success' ? 'completed' : 'retry']
 ```
 
-This retrieves the **entire output** of a task named `ExtractData`.
+#### Error Handling for PluginContext
+- Accessing a non-existent property or key in Metadata or StructuredData will throw a FlowSynxException with the code ExpressionParserKeyNotFound.
+- Ensure that RawData and StructuredData are not null before accessing their contents.
 
-## Nested Property Access
+This update ensures that the Outputs object is documented accurately as a PluginContext type, reflecting its properties and usage in expressions. 
 
-You can access **nested fields** or **array elements** inside objects and arrays using `.` (dot notation) or `[]` (indexers):
+## Nested Access
+
+Use dot notation and indexers:
 
 | Example | Meaning |
-| -------- | -------- |
-| `$[Outputs('ExtractData').Records[0].Id]` | Accesses the `Id` property of the first record in the `Records` array. |
-| `$[Outputs('Transform').Result.Data.Value]` | Accesses a deeply nested property. |
+| ------- | ------- |
+| `$[Outputs('Extract').Id]` | Record's `Id`. |
+| `$[Outputs('Transform').Content]` | Property traversal. |
+| `$[Outputs('Task').Metadata[Variables('Index')]]` | Dynamic index. |
 
-These can be combined with variable lookups:
+Arrays are supported for `IList` and `JArray`. Properties are resolved case-insensitively (`JObject`, `JToken`, or CLR objects).
 
-```jsonc
-"Parameters": {
-  "Path": "$[Variables('BasePath')]/$[Outputs('Task1').FileName]"
-}
-```
+## Literal Values
 
-If:
-```jsonc
-"Variables": { "BasePath": "/data/output" },
-"Outputs": { "Task1": { "FileName": "report.csv" } }
-```
+Inside expressions you can use:
+- Quoted strings: `'text'` or `"text"`
+- Numbers: `42`, `3.14`
+- Booleans: `true`, `false`
+- Null: `null`
+
+Example: `$['static-value']`
+
+## Functional Methods
+
+Built-in case-insensitive aggregation and search functions:
+
+| Function | Signature | Behavior |
+| -------- | --------- | -------- |
+| `Min` | `Min(arg1, arg2, collection, ...)` | Returns smallest numeric value (flattens enumerables). |
+| `Max` | `Max(...)` | Returns largest numeric value. |
+| `Sum` | `Sum(...)` | Sums all numeric values. |
+| `Avg` | `Avg(...)` | Arithmetic mean (0 if no numbers). |
+| `Count` | `Count(collection)` or `Count(arg1,arg2,...)` | If single enumerable → element count; otherwise number of arguments. |
+| `Contains` | `Contains(container, value)` | Case-insensitive search (strings, enumerables, or direct value comparison). |
+
+Arguments can be:
+- Literals
+- Variable/output references
+- Nested expressions
+- Other functional calls
+
+Examples:
 
-Result:
 ```
-/data/output/report.csv
+$[Sum(Variables('Prep').Values)] 
+$[Avg(Variables('A').Numbers, Outputs('B').Content, 10)] 
+$[Count(Variables('ListTask').Items)] 
+$[Contains(Outputs('Meta').Metadata, 'critical')] 
+$[Min(Variables('A').Values, Variables('B').Values)]
 ```
 
-## Expression Evaluation Rules
+## Arithmetic Expressions
 
-The `ExpressionParser` engine follows these rules:
+You can perform arithmetic inside `$[...]` using operators: `+ - * / %` and parentheses.
 
-1. **Single Expression**  
-   - If the expression is *entirely* a `$[...]` block, the value is returned directly as an object (not just a string).  
-   - Example: `$[Outputs('Task1')]` returns the full object produced by `Task1`.
+```
+$[(Variables('Calc').Base + Variables('Offset')) * 0.5]
+```
+
+Resolution flow:
+1. Embedded `$[...]` segments inside the arithmetic are first resolved.
+2. `Variables(...)` / `Outputs(...)` inside arithmetic are replaced with their string value (missing values become `0` in arithmetic context).
+3. Expression is evaluated; numeric result returned as `double`.
+4. If evaluation fails, it falls back to literal/lookup interpretation.
 
-2. **String Concatenation**  
-   - If the expression is embedded within a larger string, the resolved value is converted to a string and concatenated.  
-   - Example: `"File: $[Outputs('Task1').FileName]"` → `"File: result.csv"`.
+## Boolean Logic & Comparisons
 
-3. **Nested Expressions**  
-   - Expressions can reference other expressions recursively.  
-   - Example: `$[Variables(Outputs('MetaTask').VariableKey)]`.
+Supported operators: `== != > < >= <= && || !`
 
-4. **Case Insensitivity**  
-   - Keys and property names are resolved case-insensitively (`Outputs('task1')` = `Outputs('Task1')`).
+Examples:
 
-5. **Unbalanced Brackets or Parentheses**  
-   - The parser detects and throws descriptive errors for unbalanced `[]` or `()` characters.
+```
+$[Variables('Mode') == 'fast'] 
+$[Variables('Stats').Errors > 0 && Variables('Stats').Warnings == 0] 
+$[!Contains(Variables('Tags').All,'deprecated')]
+```
 
-6. **Quoted Keys**  
-   - You can wrap keys in single quotes to include spaces or special characters:  
-     `$[Variables('My Custom Key')]`.
+Comparison semantics:
+- Equality is case-insensitive string comparison for `==` / `!=`.
+- Relational operators (`> < >= <=`) apply only if both sides convert to numbers.
 
-## Supported Operations
+## Conditional (Ternary) Expressions
 
-### 1. Variable Access
 ```
-$[Variables('key')]
+$[Variables('Score').Value >= 90 ? 'excellent' : 'standard'] 
+$[Contains(Outputs('Meta').Metadata, 'urgent') ? Variables('UrgentPath') : Variables('NormalPath')]
 ```
 
-### 2. Output Access
+Format: `condition ? whenTrue : whenFalse`  
+Condition can include logical or comparison operators.
+
+## Recursive & Dynamic Keys
+
+You can build dynamic keys or nested expressions:
+
 ```
-$[Outputs('taskName')]
+$[Variables(Outputs('Meta').Id)] 
+$[Outputs(Variables('DynamicTask')).StructuredData]
 ```
 
-### 3. Nested Field Access
+Resolution is recursive: inner expressions resolve before outer structures.
+
+## String Concatenation Example
+
 ```
-$[Outputs('task').data.results[1].value]
+"Path": "$[Variables('Base')]/$[Outputs('Export').Id]"
 ```
 
-### 4. Combined or Concatenated Strings
+If:
+
 ```
-"Path": "logs/$[Variables('RunId')]/$[Outputs('Extract').FileName]"
+Variables: { "Base": "/data/out" } Outputs: { "Export": { "Id": "report.csv" } }
 ```
 
-### 5. Recursive Expressions
+Result:
+
 ```
-$[Variables(Outputs('Metadata').DynamicKey)]
+/data/out/report.csv
 ```
 
-## Error Handling
+## Return Value Behavior
 
-When an expression cannot be resolved, the parser throws a `FlowSynxException` with specific error codes.
+1. Whole expression only: `$[Outputs('Task')]` → returns the raw object (array, complex type, etc.).
+2. Embedded in text: `"File=$[Outputs('Task').Id]"` → string interpolation.
+3. Functions/Arithmetic: return numeric or boolean values directly if standalone.
 
-| Error Code | Condition | Example |
-| ----------- | ---------- | -------- |
-| `ExpressionParserKeyNotFound` | Key or source (`Variables` / `Outputs`) is missing or invalid. | `$[Outputs('UnknownTask')]` |
-| — | Unbalanced brackets or parentheses. | `$[Variables('Key'` |
+## Error Handling
 
-## Example Use Case
+Parser throws `FlowSynxException` (code: `ExpressionParserKeyNotFound`) for:
+- Missing `Variables('key')` or `Outputs('key')`.
+- Unbalanced brackets `[...]` or parentheses `(...)`.
+- Invalid functional syntax (e.g. `Contains()` with argument count ≠ 2).
+- Malformed conditional expressions.
 
-Here’s a real-world usage in a FlowSynx workflow task:
+In arithmetic-only contexts, missing lookups are coerced to `0` (to avoid failure). Direct lookup outside arithmetic still throws.
 
-```jsonc
-{
-  "Name": "SaveData",
-  "Type": "FlowSynx.FileSystem",
-  "Dependencies": ["ToCSV"],
-  "Parameters": {
-    "Operation": "write",
-    "Path": "$[Variables('OutputDirectory')]/$[Outputs('ToCSV').FileName]",
-    "Data": "$[Outputs('ToCSV').Content]",
-    "Overwrite": true
-  }
-}
-```
+## Examples
 
-If:
-```jsonc
-"Variables": { "OutputDirectory": "/mnt/results" },
-"Outputs": { "ToCSV": { "FileName": "report.csv", "Content": "data" } }
 ```
-
-Then the evaluated parameters become:
-```jsonc
-{
-  "Operation": "write",
-  "Path": "/mnt/results/report.csv",
-  "Data": "data",
-  "Overwrite": true
-}
+$[Sum(Outputs('Collect').Content, 10, Variables('Bonus'))] 
+$[Outputs('Metrics').Metadata[0] > 50 ? 'pass' : 'fail'] 
+$[Contains(Outputs('Tags').List, Variables('TargetTag')) && Variables('Stats').Count >= 5] 
+$[Min(Outputs('A').Content, Outputs('B').Content)] 
+$[Variables(Outputs('Resolver').Id)]
 ```
 
+
+## Best Practices
+
+- Use single quotes for keys with spaces: `$[Variables('My Key')]`.
+- Keep arithmetic numeric; non-numeric tokens cause literal fallback.
+- Prefer explicit parentheses for complex logic.
+- Avoid over-nesting; consider storing intermediate values in variables.
+
 ## Summary
 
-FlowSynx Expressions allow:
-- Dynamic parameter substitution.
-- Flexible referencing of workflow data.
-- Safe and validated parsing with descriptive errors.
-- Recursive resolution for advanced workflow logic.
+FlowSynx Expressions support:
+- Dynamic access to variables and task outputs
+- Nested property and array traversal
+- Functional aggregation (Min/Max/Sum/Avg/Count/Contains)
+- Arithmetic and numeric evaluation
+- Boolean logic and conditional branching
+- Recursive and dynamic key resolution
+- Strong error signaling for invalid syntax
 
 :::info
-You can use `$[Variables(...)]` and `$[Outputs(...)]` in *any* string field inside a FlowSynx workflow — the parser will automatically handle context-aware replacements.
+Expressions can be used in any string field in workflow definitions. Mix multiple `$[...]` segments freely; each is resolved independently and merged into the final string.
 :::