Make flex-config more explicit about production vs consumption

[nhoening]

It should be more clear which sensors in the flex-model are consumption and which production. For some this is done (e.g. soc-usage vs. soc-gain), but we are ambiguous for others.

• We try to explain our approach here (users should save data in the way they want, but when it comes to schedules, we need one unified way)
• We worked on a quick fix in Fix/scheduling/sign #1348 but that still works via a marker attribute (consumption_is_positive)
• That attribute should be retired and replaced with explicit flex-model fields

TODO:

• Migrate inflexible-devices to inflexible-loads and inflexible-generators on flex-context. Change schema, UI and interpretation in scheduler (positive load data is treated as consumption, negative load data is treated as generation, and vice versa for generators). The data migration would assume that existing inflexible devices are meant to be loads generators, unless they had the consumption_is_positive attribute on the sensor, in which case they were meant to be loads.
• Migrate sensor field on flex-model to consumption: {"sensor": x} or production: {"sensor": x}. Only one of them is accepted. Change schema, UI and interpretation in scheduler. The data migration would assume that existing sensor entries were meant to be storing consumption production, or consumption in case of the consumption_is_positive attribute.
• Expand the documentation section linked above with a better example (the one with solar and PV can be clearer)

Q: In the migration, if a sensor has consumption_is_positive, how should we treat it? If it is set to True, would we change the default assumption and store them as generators? We can document that of course, and we can also ask in the next TSC if people use this attribute.
A: We could have the migration deal with relevant flex-model fields stored on the db, but exactly opposite to what you suggest: we would assume generators by default and if the consumption_is_positive attribute is set to True, we would assume a load instead. Furthermore, we should keep the original attribute on the sensor, in case they were used in combination with relevant flex-model fields in the trigger message (which we can not migrate).

[Ragnar-the-mighty]

@nhoening
I have combined my pv production and my house consumption to a single sensor. E.g I have both negative and positive values, negative values during night and positive values during day.
I also have facilities with multiple batteries and I am playing with EV that are bi directional. They can also have positive and negative values.

I find inflexible-loads and inflexible-generators do be confusing what is a storage device like battery?

I think the main issues that the meaning of positive and negative energy values.

(users should save data in the way they want, but when it comes to schedules, we need one unified way)

I disagree that the user should be able to save the data in the way they want. They should be able to view the data in UI the way they want. The persisted values in the DB should be one way. You translate on import/export if needed.

If you are consistent then why is breaking up inflexible-devices in two list better?
You need to add all prior predicted sensor that will affect the 'site-power-capacity' that is available for the next device. One list is as good as two.

E.g. the problem is not inflexible-devices the problem is that you allowed the persistent values to have inconsistent sign. You will have issues and unnecessary complexity until you enforce that consistency. E.g. the DB model has energy sign one way only, translation happen on the way in/out.

/edit fixed type and improved clarity

[nhoening]

Bringing in @Flix6x

[nhoening]

I find inflexible-loads and inflexible-generators do be confusing what is a storage device like battery?

A good question, but also batteries are never among the inflexible devices :)

Your suggestion to enforce sign consistency in the database (plus letting users decide how to view them) is a big and breaking change. And it requires more work up-front from users, as they often have data only in one format, and would need conversions before even having a dashboard in FM. We already (as you know) make it a bit difficult by enforcing event start times to be on the clock (something we should hopefully make better by this idea or even moving to TimeScaleDB).

So I am not saying that should never happen, but I see it as a hige change we would need more time for. This proposed change is already not very small, but we believe it will remove a lot of uncertainty with overseeable effort.

Maybe your idea would go into a larger discussion for the not-immediate roadmap on how FlexMeasures should fit in the data pipelining of the layers it interacts with.

[Ragnar-the-mighty]

if I have two batteries would not the first one after the schedule is done be in inflexible devices?
Same with multiple EV that soon are bi-directional?

So if you implement the proposed changes would they be viewed as generator or loads?

[Flix6x]

• If you then pass the battery power sensor as an inflexible load, its positive values would be taken into account as inflexible load, and its negative values would be taken into account as inflexible generation.
• Conversely, if you instead pass the battery power sensor as an inflexible generator, its positive values would be taken into account as inflexible generation, and its negative values would be taken into account as inflexible load.
• So, you'd want to pick the key that fits the sign you chose to store your battery's power data in.