YAML File "templating"

[jimtng]

I am working on adding enhancements to our YAML configuration file syntax.

1. Variable substitutions / interpolation
2. File inclusion with variables
3. "Package" files that define a combined entity: things + items that make up a device
4. Full support for YAML Anchors
5. Full support for YAML insertion operator

This is heavily inspired by

• https://esphome.io/components/substitutions.html
• https://esphome.io/components/packages.html
• https://community.openhab.org/t/another-things-and-items-file-generator/129848

I added the include with vars feature there too

This post will be updated with more details and examples as I go. It is far from complete / comprehensive, I've just got the basic structure working last night.

Example:

main.yaml:

version: 2
#sssds

# Enhancement sections: variables and packages
variables:
  default_bridge: mqtt:broker:mosquitto
  housename: The House of Quark
  name: default # This can be overridden with !include { vars: { name: ... } }
  loungeroom: &LOUNGEROOM Lounge Room # This is a YAML Anchor - it only works in this file
  # whereas vars are passed to the included file

  VARNAME: &VARNAME vars
  lounge: lounge
  room: room

  thingid: default-thing-id # this doesn't make sense but is just an example


packages:
  livingroom-light1: !include
    file: esphome-light.inc.yaml
    vars:
      thingid: livingroom-light1
      name: Living_Room_Light_1
      label: Living Room Light 1
      location: LivingRoom # This is the name of semantic location group

  livingroom-light2: !include
    file: esphome-light.inc.yaml
    vars:
      thingid: livingroom-light2
      name: Living_Room_Light_2
      label: Living Room Light 2
      location: LivingRoom

  # The keys here aren't used, as long as they are unique,
  # but we can use anchors and aliases to avoid repetition if you like
  &BEDROOM_LIGHT bedroom-light: !include
    file: esphome-light.inc.yaml
    vars:
      thingid: *BEDROOM_LIGHT
      name: Bedroom_Light
      label: Bedroom Light
      location: BedRoom

# Our standard YAML File structure
things:
  mqtt:topic:one:
    label: a ${housename} - One
    bridge: ${default_bridge}
    location: Living Room

  mqtt:topic:two:
    label: b ${housename} - Two
    bridge: ${default_bridge}
    location: 'singlequoted strings are ${notinterpolated}'

  mqtt:topic:three: !include mqtt_template.inc.yaml  # simple include - inherit main variables

  # full include with vars - compact syntax
  mqtt:topic:four: !include { file: mqtt_template.inc.yaml, vars: { name: four, prefix: d, thingid: thing-four } }

  # full include, expanded syntax
  mqtt:topic:five: !include
    file: mqtt_template.inc.yaml
    *VARNAME: # Aliases even work as keys although vscode highlighting gets confused
      name: *LOUNGEROOM #anchors and aliases work everywhere in the same file
      prefix: e
      thingid: thing-five

  mqtt:topic:six: !include
    file: mqtt_template.inc.yaml
    vars:
      name: six
      prefix: f
      # interpolations can be nested up to 10 levels deep (just an arbitrary limit)
      location: ${${lounge}${room}} # => ${loungeroom} => Lounge Room
      thingid: thing-six

mqtt_template.inc.yaml

in this file it starts from the top level (zero indentation), but it gets inserted into whatever level !include was invoked, so it can be as simple as !includeing a plain string, or a full on yaml structure.

label: ${prefix} Included template for ${name}
bridge: ${default_bridge}
location: ${location}
config:
  availabilityTopic: tuya/${thingid}/status
  payloadAvailable: online
  payloadNotAvailable: offline
channels:
  power:
    type: switch
    config:
      stateTopic: tuya/${thingid}/state
      commandTopic: tuya/${thingid}/command
  dimmer:
    type: dimmer
    config:
      stateTopic: tuya/${thingid}/white_brightness_state
      commandTopic: tuya/${thingid}/white_brightness_command

esphome-light.inc.yaml

things:
  mqtt:topic:${thingid}:
    bridge: ${default_bridge}
    label: ${label}
    config:
      availabilityTopic: ${thingid}/status
      payloadAvailable: online
      payloadNotAvailable: offline
    channels:
      power:
        type: switch
        config:
          stateTopic: ${thingid}/light/state
          commandTopic: ${thingid}/light/command
          transformationPattern:
            - JSONPATH:$.state
          formatBeforePublish: '{"state":"%s"}'
      dimmer:
        type: dimmer
        config:
          stateTopic: ${thingid}/light/state
          commandTopic: ${thingid}/light/command
          transformationPattern:
            - JSONPATH:$.brightness
          formatBeforePublish: '{"state":"ON","brightness":"%s"}'
          min: 0
          max: 255
          step: 1

# Create three items for this package
items:
  ${name}: # The main equipment
    type: Group
    label: ${label}
    groups:
      - ${location}
    tags:
      - Lightbulb

  ${name}_Power:
    type: Switch
    label: ${label} Power
    icon: light
    autoupdate: false
    groups:
      - ${name}
    tags: [Switch, Power]
    metadata:
      alexa:
        value: PowerState
      ga:
        value: lightPower
    channels:
      # This part isn't yet done I think?

  ${name}_Dimmer:
    type: Dimmer
    label: ${label} Dimmer
    icon: light
    autoupdate: false
    groups:
      - ${name}
    tags: [Control, Brightness]
    metadata:
      alexa:
        value: Brightness
      ga:
        value: brightness
    channels:
      # This part isn't yet done I think?

Variables

Supported syntax (subject to change):

• ${var} returns the var, or a blank string if var is not defined
• ${var-default_value} - returns default_value only if var is undefined
• ${var:-default_value} - returns default_value if var is undefined, or blank
• ${var-${nested}} - nested vars are supported up to 10 levels deep

Single quoted strings won't be interpolated, e.g.

key: '${this_will_not_be_interpolated}'

Invalid syntax won't be interpolated, simply because the regex pattern won't match

key: ${vars can't have spaces}

todo: add modifiers ${var|uppercase} chainable ${var|uppercase|humanize} with args ${var|substr:1:2} (syntax may change)

At the moment, $var syntax is not supported. The braces are mandatory, because I'm worried it might interfere with things like jsonpath patterns.

I haven't looked into escaping yet, e.g. Plain text \${not_interpolated_because_its_escaped} - this is currently not supported.

Variable Scope / Precedence

The rules for variable precedence may somewhat be different to what you're familiar with.

• Variables declared in the including file (global) has precedence over variables declared inside the included file (local).
• Variables declared in the !include { vars: { xxxx: } } overrides both global and local variables.

This is designed so that you can "include" a file and override its defaults / values and adjust it as needed.

Special Variables

• __FILE__ full absolute path of the current file, e.g. /path/to/file.inc.yaml
• __FILE_NAME__ only the filename portion without the extension or leading path, e.g. file.inc
• __FILE_EXT__ only the extension portion, e.g. yaml
• __PATH__ only the path portion, e.g. /path/to

They can be accessed using the same variable syntax, i.e. ${__FILE_NAME__}

Including Other Files

Two syntaxes are supported for including other files as the "value" of a key

• Simple syntax: keyname: !include <filename>
• Full syntax:

keyname: !include
  file: <filename>
  vars:
    varname: value
    anothervar: anothervalue

# Or alternatively

keyname: !include { file: <filename>, vars: { varname: value, anothervar: anothervalue } }

Variable interpolation is done on the full syntax, e.g.

keyname: !include
  file: "${__FILE_NAME__}.inc.yaml" # needs to be double quoted

Include variable precedence

The variables defined in the vars key for the !include statement takes the highest priority over the toplevel variables.

main.yaml

variables:
  var: toplevel

keyname: !include
  file: subfile.inc.yaml
  vars:
    var: set_by_include

subfile.inc.yaml

variables:
  var: locally_set

subkey: ${var} # => set_by_include

Packages

A Package file looks just like the main file except version: and readOnly: keys aren't needed and shouldn't be in it.
It can contain any number of valid elements (things, items, tags, and anything else that may be be implemented in the future).

Packages are simply merged into the main file into their corresponding top-level elements.

Package combined with variable substitutions allow the use of one package file into many actual instances because the variable substitutions make each inclusion to have unique IDs.

[jimtng]

Todo:

• Variable modifiers ${var|title_case} (humanize, etc etc.)
• support !secret which loads from secrets.yaml - look into secrets registry / provider - as a separate PR
• write tests

Current problems:

• Plain include syntax doesn't support variable interpolation

Need to look into:

• !extend (possibly for later)

Not todo:

• not planning to add loops etc - it will become a full blown templating engine might as well use jinja or something

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/ideas-and-discussion-what-features-do-you-want-in-openhab-5-0/160573/749

[jimtng]

added special variables

[jimtng]

PR created #4818