Clarify paging behaviour around data changes

[captaincoordinates]

What is the expected behaviour for paging within search results around the time of a data change? Is this considered a specification concern, or is it at the discretion of the implementer? Can this be documented?

For example:

• Caller executes a search where result size > limit
• STAC API returns next link
• Data change occurs which alters the content or sort order of the search
• Caller requests the next link

The following all seem like viable options:

• continue as if nothing has changed, potentially resulting in
  • an empty result set or error if the caller pages "off the end" of a result set which has reduced in size
  • a gap in the data where the first item of the next page no longer follows the last item of the previous page because sort order has changed either due to new items or changes to existing items
  • repeated data from an earlier page because sort order has changed either due to new items or changes to existing items
• page according to the state of the result set at the time of the initial search, potentially requiring long-lived or expensive data snapshots
• return a 4xx response informing the caller that their request is no longer valid. Several 4xx status codes could be appropriate here

Any thoughts much appreciated.

[m-mohr]

It is at the discretion of the implementer, I think. The backends are so different that we can't prescribe a behaviour that works for everyone. PRs are welcome.

[captaincoordinates]

Perhaps this is a use-case for an extension, such as a data-versioning extension? The extension could add versioning metadata and provide guarantees or options around behaviour across version changes.

I disagree that concerns about backends should influence this decision for two reasons:

1. All backends exist to provide access to data and all - presumably - support the ability for data to change, either via the transactions extension or data publisher changes. Data change behaviour should be considered backend-agnostic.
2. A spec should be driven by the needs of the spec, and not by knowledge of available implementations. Differences between backends should determine the extent to which each backend supports the spec, but not what is written in the spec.

If there is general agreement on an approach I will be happy to contribute to that effort.

[m-mohr]

With backends I meant database engines, I have no knowledge of specific implementations. I'm following the same reasoning here why we don't describe how pagination works in detail, just follow next links, which can be page numbers, offsets, etc.

There won't be an agreement until someone writes something more specific up for discussion, I guess. For example an extension. But again, I believe that due to differences in db engines, we won't come to a general consensus. Happy to be proven wrong.

[captaincoordinates]

I will see if I can make time for an extension proposal. Or a proposal to change the existing paging spec.