Skip to content

Comments

Fix delta_kt_prime calculation for series with internal NaN values#2698

Open
ishaan-arora-1 wants to merge 2 commits intopvlib:mainfrom
ishaan-arora-1:fix/delta-kt-prime-multiday-1847
Open

Fix delta_kt_prime calculation for series with internal NaN values#2698
ishaan-arora-1 wants to merge 2 commits intopvlib:mainfrom
ishaan-arora-1:fix/delta-kt-prime-multiday-1847

Conversation

@ishaan-arora-1
Copy link

  • Closes Calculation of delta kt' #1847
  • I am familiar with the contributing guidelines
  • Tests added
  • Updates entries in docs/sphinx/source/reference for API changes.
  • Adds description and name entries in the appropriate "what's new" file in docs/sphinx/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:num or this Pull Request with `:pull:`num. Includes contributor name and/or GitHub username (link with :ghuser:user``).
  • New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.
  • Pull request is nearly complete and ready for detailed review.
  • Maintainer: Appropriate GitHub Labels (including remote-data) and Milestone are assigned to the Pull Request and linked Issue.

Problem

_delta_kt_prime_dirint incorrectly calculates delta_kt_prime for time series containing internal NaN values (e.g., multi-day data with nighttime gaps).

The previous implementation only handled NaN at the very first and last positions of the series (lines 2034-2035). For multi-day input, nighttime NaN values create internal boundaries that were not accounted for. At these boundaries, the fill_value=0 in the .add() call caused one of the two delta terms to contribute 0, but the 0.5 factor was still applied — effectively halving the delta value. This violates Perez eqn 3, which specifies that when only one neighbor is available, the full (unhalved) delta should be used.

Solution

Replace the manual NaN-filling logic with pd.DataFrame.mean(axis=1), as suggested by @cwhanse:

delta_kt_prime = pd.DataFrame({
    'next': (kt_prime - kt_next).abs(),
    'prev': (kt_prime - kt_previous).abs(),
}).mean(axis=1)

DataFrame.mean(axis=1) naturally skips NaN values, so:

  • Both neighbors valid → average of both deltas (Perez eqn 2)
  • One neighbor valid → that single delta value (Perez eqn 3)
  • No neighbors valid → NaN

This is simpler, correct, and handles all edge cases including the first/last positions of the series, internal NaN gaps, and single-element series.

Testing

Added test_delta_kt_prime_dirint_multiday which uses the exact scenario from the bug report (a series with leading/trailing NaN values simulating nighttime gaps) and verifies:

  • Edge positions use the full single-neighbor delta (Perez eqn 3)
  • Interior positions use the mean of both deltas (Perez eqn 2)
  • NaN positions remain NaN

All 121 existing irradiance tests continue to pass.

The previous implementation of _delta_kt_prime_dirint only handled NaN
at the very first and last positions of the series. For multi-day data
with nighttime NaN gaps, the edge positions adjacent to internal NaN
boundaries had their delta values incorrectly halved (the 0.5 factor
was applied even when only one neighbor was valid).

Replace the manual NaN-filling approach with pd.DataFrame.mean(axis=1),
which naturally skips NaN values. This correctly implements both Perez
eqn 2 (average of both deltas when two neighbors exist) and eqn 3
(single delta when only one neighbor exists).

Fixes pvlib#1847
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Calculation of delta kt'

1 participant