Yaml config enhancements to support variable substitutions, includes and packages

[jimtng]

Changes in this PR:

• Exclude *.inc.yaml and *.inc.yml from being processed by the YamlModelRepositoryImpl. These are meant to be used as files !included as packages.
• Add a preprocessor for the YAML config file to enhance its syntax to support:

1. Variable definition and substitutions / interpolation
2. File inclusion with variables that can be interpolated within the included file
3. Package files that define a combination of any current and future supported elements (e.g. things, items, custom tags) that make up a device.
4. YAML Anchor & insertion
5. Top-level keys that begin with a dot are treated as hidden keys. These keys are excluded from the final YAML output and serve as anchor points for YAML aliases used elsewhere in the file. This resolve [YAML] Support hidden objects for yaml substitutions #4872

The output of the preprocessor is the same YAML structure and elements that are currently supported without the addition of this PR.

Current flow:

• YAML File -> Yaml Model Repository loader (YamlModelRepositoryImpl)

New flow:

• YAML File -> YamlPreprocessor -> Yaml Model Repository loader (YamlModelRepositoryImpl)

Additional "pseudo elements" which, after being processed, are removed in the generated data output:

• variables
• packages

Example:

main.yaml:

version: 1 # this can be whatever version currently supported, irrelevant to this PR

# Enhancement sections: variables and packages
variables:
  default_bridge: mqtt:broker:mosquitto
  housename: The House of Quark
  name: default # This can be overridden with !include { vars: { name: ... } }
  loungeroom: &LOUNGEROOM Lounge Room # This is a YAML Anchor - it only works in this file
  # whereas vars are passed to the included file
  # variables defined here will override variables defined in the included files

  VARNAME: &VARNAME vars
  # the next two variables are to demonstrate nested var interpolation
  lounge: lounge
  room: room

  thingid: default-thing-id # this doesn't make sense but is just an example


packages:
  livingroom-light1: !include
    file: esphome-light.inc.yaml
    vars:
      # vars defined here override the main vars defined in the `variables` section above
      # but only apply to this included file
      thingid: livingroom-light1
      name: Living_Room_Light_1
      label: Living Room Light 1
      location: LivingRoom # This is the name of semantic location group

  livingroom-light2: !include
    # include the same file, but with different vars
    file: esphome-light.inc.yaml
    vars:
      thingid: livingroom-light2
      name: Living_Room_Light_2
      label: Living Room Light 2
      location: LivingRoom

  # The keys here aren't used, as long as they are unique,
  # but we can use anchors and aliases to avoid repetition if you like
  &BEDROOM_LIGHT bedroom-light: !include
    file: esphome-light.inc.yaml
    vars:
      thingid: *BEDROOM_LIGHT
      name: Bedroom_Light
      label: Bedroom Light
      location: BedRoom

# Our standard YAML File structure
# it can define new things, or when the name matches the one in a package,
# it will combine and override matching properties defined in the package
things:
  mqtt:topic:one:
    label: a ${housename} - One
    bridge: ${default_bridge}
    location: Living Room

  mqtt:topic:two:
    label: b ${housename} - Two
    bridge: ${default_bridge}
    location: 'singlequoted strings are ${notinterpolated}'

  mqtt:topic:three: !include mqtt_template.inc.yaml  # simple include - inherit main variables

  # full include with vars - compact syntax
  mqtt:topic:four: !include { file: mqtt_template.inc.yaml, vars: { name: four, prefix: d, thingid: thing-four } }

  # full include, expanded syntax
  mqtt:topic:five: !include
    file: mqtt_template.inc.yaml
    *VARNAME: # Aliases even work as keys although vscode highlighting gets confused
      name: *LOUNGEROOM #anchors and aliases work everywhere in the same file
      prefix: e
      thingid: thing-five

  mqtt:topic:six: !include
    file: mqtt_template.inc.yaml
    vars:
      name: six
      prefix: f
      # interpolations can be nested up to 10 levels deep (just an arbitrary limit)
      location: ${${lounge}${room}} # => ${loungeroom} => Lounge Room
      thingid: thing-six

  # This matches the package, which allows you to override the package
  mqtt:topic:livingroom-light1:
    config:
      payloadAvailable: active
      payloadNotAvailable: inactive

mqtt_template.inc.yaml

in this file it starts from the top level (zero indentation), but it gets inserted into whatever level !include was invoked, so it can be as simple as !includeing a plain string, or a full on yaml structure.

label: ${prefix} Included template for ${name}
bridge: ${default_bridge}
location: ${location}
config:
  availabilityTopic: tuya/${thingid}/status
  payloadAvailable: online
  payloadNotAvailable: offline
channels:
  power:
    type: switch
    config:
      stateTopic: tuya/${thingid}/state
      commandTopic: tuya/${thingid}/command
  dimmer:
    type: dimmer
    config:
      stateTopic: tuya/${thingid}/white_brightness_state
      commandTopic: tuya/${thingid}/white_brightness_command

esphome-light.inc.yaml

things:
  mqtt:topic:${thingid}:
    bridge: ${default_bridge}
    label: ${label}
    config:
      availabilityTopic: ${thingid}/status
      payloadAvailable: online
      payloadNotAvailable: offline
    channels:
      power:
        type: switch
        config:
          stateTopic: ${thingid}/light/state
          commandTopic: ${thingid}/light/command
          transformationPattern:
            - JSONPATH:$.state
          formatBeforePublish: '{"state":"%s"}'
      dimmer:
        type: dimmer
        config:
          stateTopic: ${thingid}/light/state
          commandTopic: ${thingid}/light/command
          transformationPattern:
            - JSONPATH:$.brightness
          formatBeforePublish: '{"state":"ON","brightness":"%s"}'
          min: 0
          max: 255
          step: 1

# Create three items for this package
items:
  ${name}: # The main equipment
    type: Group
    label: ${label}
    groups:
      - ${location}
    tags:
      - Lightbulb

  ${name}_Power:
    type: Switch
    label: ${label} Power
    icon: light
    autoupdate: false
    groups:
      - ${name}
    tags: [Switch, Power]
    metadata:
      alexa:
        value: PowerState
      ga:
        value: lightPower
    channels:
      # This part isn't yet done I think?

  ${name}_Dimmer:
    type: Dimmer
    label: ${label} Dimmer
    icon: light
    autoupdate: false
    groups:
      - ${name}
    tags: [Control, Brightness]
    metadata:
      alexa:
        value: Brightness
      ga:
        value: brightness
    channels:
      # This part isn't yet done I think?

Variables

Supported syntax (subject to change):

• ${var} returns the var, or a blank string if var is not defined
• ${var-default_value} - returns default_value only if var is undefined
• ${var:-default_value} - returns default_value if var is undefined, or blank
• ${var-${nested}} - nested vars are supported up to 10 levels deep

The default_value can optionally be enclosed in single or double quotes. The quotes are necessary when the default value contains a closing brace }, e.g. ${var:-'{foo: bar}'}. The resulting value does not include the enclosing quotes.

Single quoted strings won't be interpolated, e.g.

key: '${this_will_not_be_interpolated}'

Note that default values within single quotes are still subject to interpolation, e.g.

variables:
  foo: bar

things:
  mqtt:topic:foo:
    config:
      payloadAvailable: "${available:-'${foo}'}" 
      # => because ${available} is undefined, it defaults to
      # ${foo} which resolves to bar

Invalid syntax won't be interpolated, simply because the regex pattern won't match

key: ${vars can't have spaces}

$var syntax is not supported. The braces are mandatory, because I'm worried it might interfere with things like jsonpath patterns.

Escaping is current not supported: Plain text \${not_interpolated_because_its_escaped}.

Variable Scope / Precedence

• Variables declared in the including file (global) has precedence over variables declared inside the included file (local).
• Variables declared in the !include { vars: { xxxx: } } overrides both global and local variables.

This is designed so that you can "include" a file and override its defaults / values and adjust it as needed.

Special Variables

• __FILE__ full absolute path of the current file, e.g. /path/to/file.inc.yaml
• __FILE_NAME__ only the filename portion without the extension or leading path, e.g. file.inc
• __FILE_EXT__ only the extension portion, e.g. yaml
• __DIRECTORY__ only the directory portion of the current file, e.g. /path/to
• __DIR__ alias for __DIRECTORY__

They can be accessed using the same variable syntax, i.e. ${__FILE_NAME__}

Including Other Files

Two syntaxes are supported for including other files as the "value" of a key

• Simple syntax: keyname: !include <filename>
• Full syntax:

keyname: !include
  file: <filename>
  vars:
    varname: value
    anothervar: anothervalue

# Or alternatively

keyname: !include { file: <filename>, vars: { varname: value, anothervar: anothervalue } }

Changes made to the included files will trigger an automatic reload of the main yaml file.

Variable interpolation is done on the full syntax, e.g.

keyname: !include
  file: "${__FILE_NAME__}.inc.yaml" # needs to be double quoted

Include variable precedence

The variables defined in the vars key for the !include statement takes the highest priority over the toplevel variables.

main.yaml

variables:
  var: toplevel

keyname: !include
  file: subfile.inc.yaml
  vars:
    var: set_by_include

subfile.inc.yaml

variables:
  var: locally_set

subkey: ${var} # => set_by_include

Packages

A Package file looks just like the main file except version: and readOnly: keys aren't needed and shouldn't be in it.
It can contain any number of valid elements (things, items, tags, and anything else that may be be implemented in the future).

Packages are simply merged into the main file into their corresponding top-level elements.

Package combined with variable substitutions allow the use of one package file into many actual instances because the variable substitutions make each inclusion to have unique IDs.

A package file can contain the full definition of a "device", which includes Things and Items related for that device. This can be "instantiated" by providing the package with specific device ID (thing id, item names) using include vars.

[jimtng]

@lolodomo FYI.

This shouldn't conflict with anything that you're working on, I think.

[jimtng]

rebased to main

[jimtng]

Todo: merge lists in packages

[kaikreuzer]

@lolodomo When you find time, I would appreciate a short review from your end.

[spacemanspiff2007]

Great Work!
I'll take a deeper look at this this week but some questions / remarks so I have a better understanding:

1. Do per file variables need to be defined as global variables or is it sufficient to define them in the include directive?
2. Shouldn't __PATH__ be named __FOLDER__, __DIRECTORY__ or __PARENT__
3. Single quoted strings won't be interpolated,
   What's your reasoning or idea behind this?
4. What is returned for ${var:-'{}'} if var is undefined?

[jimtng]

1. Do per file variables need to be defined as global variables or is it sufficient to define them in the include directive?

No, they only need to be defined in the include directive.

2. Shouldn't __PATH__ be named __FOLDER__, __DIRECTORY__ or __PARENT__

Out of the three options, I prefer __DIRECTORY__. Is PATH too ambiguous?

3. Single quoted strings won't be interpolated,
   What's your reasoning or idea behind this?

So that you can have ${xxx} without being interpolated?

4. What is returned for ${var:-'{}'} if var is undefined?

Good question. As it is now, it will return '{}' - however, it will trip up when you have ${var:-blah}${var:-blah} because the pattern for the "default" value is greedy.

To fix this, I need to make it ungreedy, but it would no longer work for ${var:-'{}'} case. I think this is an acceptable compromise without making the regex pattern more complex or implementing a separate parser. So the next change will have this requirement: The default value cannot contain } character.

[jimtng]

@spacemanspiff2007 I added a commit that supports quoted defaults (in which you can use braces).

• ${var:-'{}'} => {} when var is undefined

[spacemanspiff2007]

@jimtng I really like what you did in this PR! This is going to be a great feature which will help a lot and make configuration a breeze.

---

Is PATH too ambiguous?

In python there is a pathlib which deals with paths to dirs, files, drives, shares so PATH became synonymous to me with location of an arbitrary file system object. I think __DIRECTORY__ would be a nicer and also more correct.

---

Instead of packages how about we name the top level key templates or includes?
Package sounds like a couple of files yet it's only one file that's used.

Would it also be possible to not have to use the !include directive?
It's clear from the top level key that it's a reference to another file and what's actually happening is a merge and not an include.
E.g. like this:

packages:
  livingroom-light1:
    file: esphome-light.inc.yaml
    vars:
      thingid: livingroom-light1

---

How about we also add a templates? (or whatever name we settle for the top level key) and includes? subfolder to the YamlModelRepositoryImpl exclusion regex?

---

So that you can have ${xxx} without being interpolated?

I understand. I guess it's quite unexpected that " and ' quoted strings behave differently.
Maybe we can add the possibility to escape variable expansion with \${...} later or add a global option to not do the expansion for a file at all later. Then the behavior could be aligned which would be more user friendly.

[jimtng]

@spacemanspiff2007

__PATH__ is now __DIRECTORY__ although I was tempted to shorten it to __DIR__. It was already done in a commit prior to your last message.

Instead of packages how about we name the top level key templates or includes?
Package sounds like a couple of files yet it's only one file that's used.

templates is definitely not correct for it.
includes is awkward, IMO.

I think packages is a good name. The idea originally came from https://esphome.io/components/packages.html

Package sounds like a couple of files yet it's only one file that's used.

The packages section would contain a list of includes, so it does contain several files/packages

packages:
  package1:
  package2:
  package3:

the package keys not being used bothers me a bit. Perhaps it should just be a list instead of a map, but I went with a map to make it all consistent with the rest of the convention for things, items, tags.

Perhaps I can pass an implicit variable ${PACKAGE_KEY} into the included file to make it useful.

Would it also be possible to not have to use the !include directive?

I'll consider it.

How about we also add a templates? (or whatever name we settle for the top level key) and includes? subfolder to the YamlModelRepositoryImpl exclusion regex?

I think with the exclusion of *.inc.yaml it should be enough. Adding more subdirectories to exclusion may potentially cause confusion.

You can of course structure your folder as ./includes/foo.inc.yaml if you wish.

So that you can have ${xxx} without being interpolated?

I understand. I guess it's quite unexpected that " and ' quoted strings behave differently.

This is consistent with sh / bash / ruby, e.g.

$ A=test
$ echo "$A"
test
$ echo '$A'
$A

Maybe we can add the possibility to escape variable expansion with \${...} later

I thought of this, but it's a can of worm that I'd rather not have to deal with yet.

[spacemanspiff2007]

__PATH__ is now __DIRECTORY__ although I was tempted to shorten it to __DIR__. It was already done in a commit prior to your last message.

I saw it after I sent the message. Both names are fine with me.

the package keys not being used bothers me a bit. Perhaps it should just be a list instead of a map, but I went with a map to make it all consistent with the rest of the convention for things, items, tags.

Perhaps I can pass an implicit variable ${PACKAGE_KEY} into the included file to make it useful.

For items and things the key is the unique identifier. Since these packages are fully merged configs they have no unique identifier (which would be the path to the file). If you find a good use case for the PACKAGE_KEY then a mapping is okay, otherwise I would prefer a list.

Will you provide support for the short syntax e.g. !include esphome-light.inc.yaml?

I'll consider it.

I thought some more and it might make sense to keep the !include directive.
File exclusion is done by matching the .inc.yaml so the !include should be kept since it aligns the the inc part in the filename.

I think packages is a good name. The idea originally came from https://esphome.io/components/packages.html

Yes - I noticed how everything is inspired by esphome.
However I'm not fully sold on packages, maybe just merge?
Since that's what it does - it merges the configs together.

---

Will a change in a package/included file trigger a reload of the referencing file?

[jimtng]

the package keys not being used bothers me a bit. Perhaps it should just be a list instead of a map, but I went with a map to make it all consistent with the rest of the convention for things, items, tags.
Perhaps I can pass an implicit variable ${PACKAGE_KEY} into the included file to make it useful.

For items and things the key is the unique identifier. Since these packages are fully merged configs they have no unique identifier (which would be the path to the file). If you find a good use case for the PACKAGE_KEY then a mapping is okay, otherwise I would prefer a list.

It would be entirely up to the user to use it how they see fit. For me, I would use the PACKAGE_KEY as the thingid in the example above so there is no need to specify thingid vars.

Will you provide support for the short syntax e.g. !include esphome-light.inc.yaml?

!include supports short syntax, so yes, that's automatically possible in the packages section too. However, a plain include like that isn't very useful because you could just as easily turn that into a full fledged yaml file on its own (with version: xx) and have it be loaded just like any other normal yaml file.

I thought some more and it might make sense to keep the !include directive.
File exclusion is done by matching the .inc.yaml so the !include should be kept since it aligns the the inc part in the filename.

That is true, although that wasn't the main reason. Using !include is a "free" operation courtesy of the main/generic !include implementation for other purposes. All the packages section is concerned about is merely how to merge the (already included) packages.

I've just tried implementing what you suggested and it resulted in more complexity and code duplication. This is of course simply an implementation detail invisible to the user.

Will a change in a package/included file trigger a reload of the referencing file?

Not currently, however I did consider this possibility. There are pros and cons to doing this, and I opted not doing this, at least for now. The cons: sometimes I might want to make changes to several templates without having it reloaded, and then I can just trigger one reload of the main yaml file. But perhaps it's fine for it to reload several times too.

[jimtng]

However I'm not fully sold on packages, maybe just merge?

I think packages is a more apt term. It means that you can package a device elements (things, items, custom tags) into one file.

merge is more "generic" in meaning, and it doesn't convey the meaning that it's a way to create a cohesive unit like the term package does.

[Nadahar]

Instead of packages how about we name the top level key templates

templates is a OH concept (not sure that it will ever end up in these configuration files, but still..) so I would leave that alone.

I guess it's quite unexpected that " and ' quoted strings behave differently.

I think it's rather common that these have different "rules" when it comes to substitutions, escaping etc. I'm not weighing in on whether it's a good idea here, I'm just saying that generally I think that is acceptable. But, doesn't YAML already have some "rules" regarding quoting to take into account? As far as I know, YAML only allows gives the escape character \ "special meaning" in double quotes, which means that you have to use double quotes if you want to include a character that needs escaping.

although I was tempted to shorten it to __DIR__.

That's preferable IMO. When making such "special codes", you might as well make them easy to use.

but I went with a map to make it all consistent with the rest of the convention for things, items, tags.

Don't. It shouldn't become "a mantra" that is more important than reason. People will have to learn to use arrays too, and it's the only thing that makes sense here.

However I'm not fully sold on packages, maybe just merge?

In reality the terms matter less than we think, as soon as something is chosen and its meaning is well-defined and documented, people will quickly "get used to it". If the similarity with the ESPHome format is kept, parity with it might have a value in itself (for users using both systems).

[spacemanspiff2007]

!include supports short syntax, so yes, that's automatically possible in the packages section too. However, a plain include like that isn't very useful because you could just as easily turn that into a full fledged yaml file on its own (with version: xx) and have it be loaded just like any other normal yaml file.

Your right about the !include in packages.
But how about reusing parts of the configuration like this:

items:
  my_item_name: !include energy_item.inc.yaml
  my_other_name: !include energy_item.inc.yaml

energy_item.inc.yaml:

label: Energy
format: "%.1f"
unit: "kWh"
...

So this will be supported?

I've just tried implementing what you suggested and it resulted in more complexity and code duplication. This is of course simply an implementation detail invisible to the user.

Let's keep the !include then

Not currently, however I did consider this possibility. There are pros and cons to doing this, and I opted not doing this, at least for now. The cons: sometimes I might want to make changes to several templates without having it reloaded, and then I can just trigger one reload of the main yaml file. But perhaps it's fine for it to reload several times too.

But imho this would work the same way if you need to do changes now in an items and things file.
You edit the files outside the folder and then move them into the openhab config folder.
It's unexpected if some file edits will be picked up by openHAB and other won't.

If I reuse an .inc.yaml file in multiple files I have to remember which files to reload when I do a change.
Do you think it would be much effort to implement a reload when the include file changes?

[jimtng]

Your right about the !include in packages. But how about reusing parts of the configuration like this:

items:
  my_item_name: !include energy_item.inc.yaml
  my_other_name: !include energy_item.inc.yaml

energy_item.inc.yaml:

label: Energy
format: "%.1f"
unit: "kWh"
...

So this will be supported?

I haven't tried this exact scenario but yes it should work just like you expected. That's essentially how !include works in all cases.

Not currently, however I did consider this possibility. There are pros and cons to doing this, and I opted not doing this, at least for now. The cons: sometimes I might want to make changes to several templates without having it reloaded, and then I can just trigger one reload of the main yaml file. But perhaps it's fine for it to reload several times too.

But imho this would work the same way if you need to do changes now in an items and things file. You edit the files outside the folder and then move them into the openhab config folder.

I think it depends. Besides, most people edit files in-place.

It's unexpected if some file edits will be picked up by openHAB and other won't.

The two files are different. One is an include file, the other is the main file.

If I reuse an .inc.yaml file in multiple files I have to remember which files to reload when I do a change.

Yes, exactly the idea, but I do see the convenience of having it auto reload and on the other hand, it can be annoying for those who like total control that eventually someone would ask for the option to turn it off. I am so used to doing it manually and the way I currently do it is using one big main file, so I only need to reload one file.

Do you think it would be much effort to implement a reload when the include file changes?

It would be an effort yes, and I am currently on vacation with very limited "computer" time until end of June. This will have to be a separate PR later.

but I went with a map to make it all consistent with the rest of the convention for things, items, tags.

Don't. It shouldn't become "a mantra" that is more important than reason. People will have to learn to use arrays too, and it's the only thing that makes sense here.

It's not merely following a mantra. I actually much prefer to have it as a map. But supporting a list is easy(easier), I'll see what it looks like and try it and if it looks OK, I'll consider supporting both syntaxes perhaps.

[Nadahar]

Yes, exactly the idea, but I do see the convenience of having it auto reload and on the other hand, it can be annoying for those who like total control that eventually someone would ask for the option to turn it off. I am so used to doing it manually and the way I currently do it is using one big main file, so I only need to reload one file.

I think the only sensible thing is that the templates "hot reload" too, otherwise you'd have to figure out and chase down which files are affected. I mean, the point of using a template is to avoid duplicating information. If you still have to hunt down the individual files and resave them, you could almost might as well just have copy/pasted.

But, I guess that this can be a challenge implementation-wise. You would have to keep track of basically all "YAML model" files, and then trigger reparsing when required. It sounds like a path full of pitfalls.

[jimtng]

As I said, it depends on your workflow. I've been using a similar system and workflow in openhab for at least three years and preferred manual reloading.

I have an implementation in mind on how to do the auto reload but will also need to add a setting not to do it for those who prefer not having it. I also have in mind how to do this

A.global option through service config
B. Granular option through include Param (least preferred)

I'll implement this in a late pr if/when time permits or at the end of June.

[jimtng]

Everybody was tired today and returned early, so I had some time to implement the include dependency tracking.

Now whenever any include files are changed, the main yaml will be reloaded.

[lolodomo]

@jimtng : snakeyaml is not a new dependency, it was an indirect dependency through Jackson YAML, correct ?
I vaguely remember a discussion at the very beginning of YAML implementation to not use snakeyaml but I do not remember what was the reason. Maybe something in link with what YAML version is supported ?

[jimtng]

@jimtng : snakeyaml is not a new dependency, it was an indirect dependency through Jackson YAML, correct ? I vaguely remember a discussion at the very beginning of YAML implementation to not use snakeyaml but I do not remember what was the reason. Maybe something in link with what YAML version is supported ?

Correct, it's not a new dependency and that's why I haven't added it to pom.xml.
So it's the same SnakeYaml that Jackson is using, and therefore supports the same Yaml version (Yaml 1.1)

I don't know what the considerations were in the beginning.

[lolodomo]

I don't know what the considerations were in the beginning.

Maybe it was not SnakeYaml but another library we decided to not use. I do not remember exactly.