Skip to content

bug: Potential to miss data when paginating with $filter and gt #6

@ReubenFrankel

Description

@ReubenFrankel

This can happen when a full page of data ends with some replication key value, where the next request - by virtue of using the greater than operator (gt) - will exclude data that has the same exact replication key value:

  1. initial request made
  2. last replication key UpdatedDate value is 2026-02-25T11:55:33.1676158Z
  3. next request made with $filter expression UpdatedDate gt 2026-02-25T11:55:33.1676158Z
  4. other data with an UpdatedDate of 2026-02-25T11:55:33.1676158Z will be excluded

elif isinstance(next_page_token, str):
params["$filter"] = f"{self.replication_key} gt {next_page_token}"

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions