feat: Floor datetimes on ingestion

[BelhsanHmida]

Description

This PR addresses Round datetimes on ingestion #1767 by implementing the FlexMeasures-side flooring behavior for off-clock datetimes.

The main goal of this branch is to make ingestion more robust when datetimes are slightly off the expected sensor clock, such as 10:00:40 instead of 10:00:00.

• Floor the start datetime for direct POST sensor data on non-instantaneous sensors
• Floor datetimes in uploaded sensor data before saving and before any further handling
• Preserve submitted flex-model datetimes in schedule job kwargs, so the job description reflects the user’s original request
• Keep the normal POST endpoint behavior otherwise aligned with the previous compatibility rules
• Added changelog item in documentation/changelog.rst

Look & Feel

N/A

Examples of the behavior in this branch:

• direct POST: 2025-11-21T10:00:40+01:00 -> 2025-11-21T10:00:00+01:00
• upload datetimes: 2025-11-21T10:15:40+01:00 -> 2025-11-21T10:15:00+01:00
• schedule job kwargs: soc-minima.datetime=2015-01-02T23:00:40+01:00 remains 2015-01-02T23:00:40+01:00

How to test

Run the relevant test modules:

• uv run pytest flexmeasures/api/common/schemas/tests/test_sensor_data_schema.py -vv
• uv run pytest flexmeasures/api/v3_0/tests/test_sensor_data.py -vv
• uv run pytest flexmeasures/api/v3_0/tests/test_sensor_data_fresh_db.py -vv
• uv run pytest flexmeasures/api/v3_0/tests/test_sensor_schedules.py -vv
• uv run pytest flexmeasures/api/v3_0/tests/test_sensors_api_freshdb.py -vv

Further Improvements

• Revisit broader mismatched-period resampling separately if that is still desired for upload flows or future follow-up work
• Follow-up scheduling semantics for off-tick soc-minima, soc-maxima, and soc-targets are handled in Feat/offtick soc normalization #2194

Related Items

• Addresses Round datetimes on ingestion #1767
• Builds on the earlier spike in Floor event start #1828
• Keeps the current PR scoped to FlexMeasures-side datetime flooring, without requiring the Timely Beliefs spike branch

Closes #1767

---

Sign-off

• I agree to contribute to the project under Apache 2 License.
• To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

[Copilot]

Pull request overview

This PR implements datetime flooring at multiple ingestion points (direct POST, file upload, and scheduling flex-model inputs) to make ingestion/scheduling robust against slightly off-clock timestamps, as discussed in issue #1767.

Changes:

• Add support for flooring timed-event datetime/start/end fields based on a configurable event resolution in schema validation.
• Floor event starts in uploaded sensor data before further handling/resampling, gated by round_datetimes_on_ingestion.
• Floor flex-model timed-event datetimes (soc-minima/maxima/targets) before enqueuing scheduling jobs, plus add/adjust tests for all three flows.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
flexmeasures/data/schemas/sensors.py	Adds timed-event flooring support and floors uploaded-data event starts before conversions/resampling.
flexmeasures/data/schemas/scheduling/storage.py	Passes a rounding resolution into flex-model schemas so timed-event datetimes can be floored consistently.
flexmeasures/api/v3_0/sensors.py	Floors flex-model timed-event datetimes prior to job creation in trigger_schedule.
flexmeasures/api/common/schemas/sensor_data.py	Floors direct-POST start for non-instantaneous sensors and adds validation for the zero-duration single-value edge case.
flexmeasures/api/v3_0/tests/test_sensors_api_freshdb.py	Adds coverage for flooring behavior during upload ingestion.
flexmeasures/api/v3_0/tests/test_sensor_schedules.py	Adds coverage for flooring flex-model datetimes prior to scheduling.
flexmeasures/api/v3_0/tests/test_sensor_data.py	Adds coverage for flooring start in direct POSTs for non-instantaneous sensors.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[BelhsanHmida]

@copilot can you also add add a regression test to check that an invalid flex-model datetime still returns a 422 validation error instead of causing a 500.

[Copilot]

@copilot can you also add add a regression test to check that an invalid flex-model datetime still returns a 422 validation error instead of causing a 500.

Implemented in 3a56e0c: I added a regression test (test_trigger_schedule_with_invalid_flex_model_datetime) that sends an invalid flex-model datetime and verifies the API returns 422, and I fixed the trigger path to convert the flooring validation error into unprocessable_entity(...) instead of returning a 500.

[Flix6x]

• An addition to the documentation seems in order, perhaps at the end of this section.
• We are introducing an undocumented sensor attribute.
• Discussion point: is flooring always the best choice? I see no examples like an soc-target at 2026-05-12T08:29:58+02, which would now be floored to 08:15.
• Discussion point: for the flex-config, the conversion could also happen after the job is picked up rather than before the job is created. Otherwise, we are not storing the user's original intention in the job description. I wonder which choice would lead to fewer questions.

[BelhsanHmida]

• An addition to the documentation seems in order, perhaps at the end of this section.
• We are introducing an undocumented sensor attribute.
• Discussion point: is flooring always the best choice? I see no examples like an soc-target at 2026-05-12T08:29:58+02, which would now be floored to 08:15.
• Discussion point: for the flex-config, the conversion could also happen after the job is picked up rather than before the job is created. Otherwise, we are not storing the user's original intention in the job description. I wonder which choice would lead to fewer questions.

I addressed first two points concerning the documentation and attribute parts in the latest commits: floor_datetimes_to_resolution is now documented in the API notation docs, sensor schema/OpenAPI examples, and changelog.

I see the point. For flex-model constraints, flooring before job creation can change both the user’s submitted request and what we store in the job kwargs.

Your soc-target example shows the semantic risk: flooring 08:29:58 to 08:15 makes the target earlier and potentially stricter than intended.

My guess is that preserving the submitted flex-model in the job kwargs would lead to fewer questions. The job would show exactly what the user sent, which makes debugging easier, and the scheduler could still normalize internally when interpreting the constraints.

So my preferred direction would be to move flex-model datetime normalization out of the API trigger path and into scheduler deserialization/execution. Another option is to keep the current behavior but document clearly that timed flex-model fields are floored before the job is stored. I lean toward the first option.

[Flix6x]

I agree with keeping the submitted flex-model in the job kwargs. For the normalization of soc-minima, soc-maxima and soc-targets I feel we can stay a little closer to the user's intent. I propose to study #10 (comment) and come up with a more elegant proposal than blindly flooring.

I also suggest to always turn on relax-soc-constraints if those fields are not originally on the tick, just in case more complex combinations would lead to infeasibilities.

[BelhsanHmida]

I agree with keeping the submitted flex-model in the job kwargs. For the normalization of soc-minima, soc-maxima and soc-targets I feel we can stay a little closer to the user's intent. I propose to study #10 (comment) and come up with a more elegant proposal than blindly flooring.

I also suggest to always turn on relax-soc-constraints if those fields are not originally on the tick, just in case more complex combinations would lead to infeasibilities.

I addressed the first part here: the submitted flex-model now stays unchanged in the job kwargs.

For the soc-minima, soc-maxima and soc-targets part, I opened #2194 as a follow-up proposal based on the #10 example. The idea there is to stop blindly flooring those SoC timestamps. Instead, the schema keeps the original event times, and the storage scheduler projects off-tick point constraints onto the scheduling grid using the charge/discharge capacity.

So for an off-tick soc-target, we no longer turn it into the same target at the floored tick. We add an adjusted bound at the previous tick based on what can still be charged/discharged before the original timestamp, and keep the exact target on the next tick.

I also added the relax-soc-constraints part in #2194: when any original soc-targets, soc-minima or soc-maxima timing field is off-tick, we enable relax-soc-constraints before flex-context deserialization, so the existing default breach-price logic is reused.