Different TSAM versions lead to different aggregated time series

[phil-fzj]

Version Checks (indicate both or one)

• I have confirmed this bug exists on the lastest commit on the master branch

• I have confirmed this bug exists on the lastest commit on the dev branch.

• I have confirmed this bug exists for a specific version the software.

Issue Description

When aggregating the time series data from the fine model defined in this test:
https://github.com/FZJ-IEK3-VSA/FINE/blob/b9bce326cd77ff64e97747da7eac480bf11791d2/test/conftest.py#L368

we find different results depending on the tsam version used (e.g. 2.3.9 and 2.3.2)

E.g. The output for the aggregated time series of the component ("Existing run-of-river plants") differs:
esM.getComponent("Existing run-of-river plants").aggregatedOperationRateFix[0].mean()

Env1 (pandas 2.2.3, python 3.13, tsam 2.3.9):
cluster_0    0.544570
cluster_1    0.563770
cluster_2    0.530722
cluster_3    0.538196
cluster_4    0.519660
cluster_5    0.000000
cluster_6    0.000000
cluster_7    0.545162

Env3 (pandas 2.2.3, python: 3.13.0, tsam 2.3.7)
cluster_0    0.544570
cluster_1    0.563770
cluster_2    0.530722
cluster_3    0.538196
cluster_4    0.519660
cluster_5    0.000000
cluster_6    0.000000
cluster_7    0.545162
dtype: float64

differ from this:

Env2 (pandas 1.5.3, python 3.10, tsam  2.3.2):
cluster_0    0.544570
cluster_1    0.566054
cluster_2    0.530722
cluster_3    0.538196
cluster_4    0.519660
cluster_5    0.000000
cluster_6    0.000000
cluster_7    0.545162

E.g. compare the values for cluster_1.
It does not differ much but enough to result in a different optimization result for fine.

For refenence also refer to FZJ-IEK3-VSA/FINE#475.

The following tsam settings are passed from fine to TSAM:

esM.aggregateTemporally(
        numberOfTypicalPeriods=3,
        segmentation=False,
        sortValues=True,
        representationMethod=None,
        rescaleClusterPeriods=True,
    )

Reproducible Example

Use the code from this test:
https://github.com/FZJ-IEK3-VSA/FINE/blob/b9bce326cd77ff64e97747da7eac480bf11791d2/test/conftest.py#L368

Expected Behavior

Time series should be the same after aggregation when using different tsam versions.

pip list -v

[phil-fzj]

@jo-omoyele

[jo-omoyele]

A small version-dependent difference exists when using rescaleClusterPeriods = True with numberOfTypicalPeriods = 3 for the data.
rescaleClusterPeriods rescales the values of the clustered Periods such that the mean of each time series in the typical Periods fits the mean value of the original time series, without changing the values of the extreme Periods.

With pandas 1.5, I get:

cluster_1 = 0.566054

With pandas 2.3, I get:

cluster_1 = 0.563770

That’s about a 0.4 % deviation, while all other clusters match exactly.

It only happens when both numberOfTypicalPeriods = 3 and rescaleClusterPeriods = True are used for this example. Results are identical if rescaleClusterPeriods = False.

It seems related to pandas 2.x Copy-on-Write changes affecting _rescaleClusterPeriods.

Also, a warning appears when this happens:

_UserWarning: Max iteration number reached for "Existing run-of-river plants_operationRateFix_cluster_1" while rescaling the cluster periods.
The integral of the aggregated time series deviates by: 0.000849%_

This will be further looked into.
Until this is resolved, users should pay attention to this warning and may try a different number of typical periods if the warning appears.