CBG-5429: Fix SequenceID String() and Before() for LowSeq/TriggeredBy corner cases

[RIT3shSapata]

CBG-5429

• Fix SequenceID.String() and SequenceID.Before() for corner cases involving both LowSeq and TriggeredBy, and add a changes feed integration test to verify no regression in backfill behaviour.
• String() fix: the old implementation only emitted the LowSeq prefix when LowSeq < Seq. When LowSeq > Seq (which occurs during a backfill-in-progress with active skipped sequences), the LowSeq field was silently dropped, causing the client to receive an incorrect last_seq. On the next request the changes feed would resume from the wrong position, potentially missing backfill documents.
• Before() fix — the old implementation routed all comparisons through SafeSequence() (which returns LowSeq when non-zero), ignoring the Seq field entirely when LowSeq is set. This produced wrong ordering for cases such as:
  • 1234:5677:2000.Before(1234::5677) → returned false, expected true
  • 1234:5677:2000.Before(1300::1100) → returned false, expected true
• Added a test that creates a scenario where skipped sequences (LowSeq mode) and a mid-stream channel access grant (TriggeredBy backfill) are both active simultaneously, verifying that a since value of the form {LowSeq: L, TriggeredBy: T, Seq: S} is correctly serialised, sent back, and processed by the changes feed without missing or duplicating documents.

Pre-review checklist

• Removed debug logging (fmt.Print, log.Print, ...)
• Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
• Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Dependencies (if applicable)

• Link upstream PRs
• Update Go module dependencies when merged

Integration Tests

• https://jenkins.sgwdev.com/job/SyncGatewayIntegration/909/
• https://jenkins.sgwdev.com/job/SyncGatewayIntegration/982/

[Copilot]

Pull request overview

This PR fixes ordering/serialization edge cases for db.SequenceID used by the _changes feed, specifically around compound sequences that include both LowSeq (skipped-sequence mode) and TriggeredBy (channel-grant backfill). It also adds an integration test intended to prevent regressions in multi-channel backfill behavior.

Changes:

• Fix SequenceID.String() to preserve LowSeq for LowSeq:TriggeredBy:Seq cases where LowSeq > Seq.
• Rewrite SequenceID.Before() to correctly order sequences when LowSeq and/or TriggeredBy are present.
• Add a _changes integration test scenario combining skipped sequences (LowSeq mode) with a mid-stream channel grant (TriggeredBy backfill).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
db/sequence_id.go	Adjusts string formatting and comparison semantics for compound SequenceID corner cases affecting _changes ordering and continuation.
db/sequence_id_test.go	Extends unit coverage for String() and ordering with LowSeq+TriggeredBy combinations.
rest/changestest/changes_api_test.go	Adds an integration test meant to cover LowSeq+TriggeredBy behavior during changes backfill.

[gregns1]

false, you will get sequence 9 again here in this scenario when stable sequence moves on

[gregns1]

I think this may be worth doing

[gregns1]

I think we have issues with these changes with interrupted/oneshot feeds. I don't think this is effected for continuous feeds but will be effected for continuous feeds on the first iteration of the feed. So lower blast radius.

Scenario 1 — continuing backfill skips a doc

Seq	Channel	Note
1	(user1)	access: ABC
2,3,4	DEF	backfill targets
5	ABC
6	—	gap (skipped)
7	ABC

1. GET _changes as user1 → last_seq = 5::7.
2. Grant user1 access to ABC, DEF → lands at seq 10.
3. POST _changes with since=5::7&limit=2 (cuts the backfill short) → last_seq = 5:10:3.
   Old code: LowSeq(5) is not < Seq(3), so it emitted 10:3.
4. Seq 6 arrives; the feed's lowSequence moves to 7.
5. POST _changes with since=5:10:3 (no/high limit) → returns only the user doc, last_seq = 7::10.

Result: Seq 4 is not delivered in the feed. Line 850 in changes.go will see that since low seq is != 0 but since low seq != low seq on feed (it’s moved to 7 now) and this does not set since low seq to 0. This stays as 5 here. Then in changesFeed SafeSequence() calculates safe seq to be low seq, in this case 5 and this leads to sequence 4 above getting missed on the feed.

Scenario 2 — grant's fresh backfill is suppressed

Seq	Channel	Note
1	(user1)	access: ABC
2	DEF	backfill target
3	GHI	backfill target
4	(grant)	user1 → ABC, DEF
5	(grant)	user1 → ABC, DEF, GHI
6,7	ABC
8	—	gap (skipped)
9	ABC

1. POST _changes with limit=1 (cuts the backfill short) → last_seq = 7:4:2.
2. Seq 8 arrives; the feed's lowSequence moves off 7.
3. POST _changes with since=7:4:2.

Result: GHI's backfill doc (seq 3) is not delivered in feed. On this first iteration isNewChannel is false (changedChannels is only populated on later iterations of a waiting feed), so the decision to backfill falls to backfillRequired at changes.go:929
So seqAdded at for GHI will be seq 5:
So seqAddedAt > 1 -> true
But 7:4:2.Before(5) -> false
backfillRequired is false

I think overall it may be worth discussing as a team this PR in sync up.

[gregns1]

false, you will get sequence 9 again here in this scenario when stable sequence moves on

[gregns1]

I think this may be worth doing

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

db/sequence_id.go:68

• The PR description says the String() fix should preserve LowSeq in the wire format when LowSeq > Seq (to avoid returning an incorrect last_seq during backfill), but intSeqToString still explicitly omits LowSeq when Seq < LowSeq (and the new changestest assertions expect a 2-component string like "TriggeredBy:Seq"). Please reconcile the intended behaviour: either update the PR description/tests to match the documented rule "ignore LowSeq when LowSeq > Seq", or change the formatting rule so LowSeq is preserved for the backfill corner case described in the PR.

// intSeqToString implements the formatting rules documented on String() above.
func (s SequenceID) intSeqToString() string {
	// LowSeq is omitted from the output here for two independent reasons: it's zero (there's
	// nothing to report), or it's stale - greater than Seq. LowSeq is stale when this entry is a
	// previously-skipped sequence being delivered after the feed's lowest contiguous sequence has
	// already moved past it, so including it here would misrepresent this entry's position.
	// TriggeredBy, if set, reflects an in-progress channel backfill and is unrelated to which of
	// those two reasons applies.
	if s.LowSeq == 0 || s.Seq < s.LowSeq {
		if s.TriggeredBy > 0 {
			return fmt.Sprintf("%d:%d", s.TriggeredBy, s.Seq)
		}
		return strconv.FormatUint(s.Seq, 10)
	}

db/sequence_id.go:166

• SequenceID marshaling determines the public REST API representation of both changes feed seq entries and last_seq/since values (number for simple sequences, colon-delimited string for compound). The OpenAPI spec currently documents Changes-feed.results[].seq as a number and last_seq as a string, which doesn’t match the actual behavior (both may be string/number depending on whether the sequence is compound). Since this PR changes compound sequence formatting/ordering, please also update docs/api to reflect the correct types/formats (e.g., oneOf: [string, number] and note colon-delimited forms are opaque).

// MarshalJSON implements json.Marshaler, encoding a SequenceID via String() when TriggeredBy or
// LowSeq is set (any compound form), or as a bare integer for a simple sequence.
func (s SequenceID) MarshalJSON() ([]byte, error) {

	if s.TriggeredBy > 0 || s.LowSeq > 0 {
		return fmt.Appendf(nil, "\"%s\"", s.String()), nil
	} else {

[adamcfraser]

A few changes needed.

[adamcfraser]

"stale" is the wrong way to describe this scenario. If LowSeq is greater than s.Seq, it represents a case where the sequence of the revision being sent (s.Seq) is older than the cache's stable sequence (LowSeq), and so it's not necessary to include the stable sequence information.
The low (stable) sequence is only required for revisions with sequences higher than the stable sequence, to keep track of the fact that we're sending a sequence later than the stable sequence, and so there are potentially sequences n in the range s.LowSeq < n < s.Seq that the client hasn't yet received.

[adamcfraser]

This is incorrect - this should be 100:105:90.

105:90 indicates that the changes feed is currently at sequence 105, and there's a channel grant at sequence 105 that has triggered a backfill. Sequence 90 is being sent as part of that backfill, but the overall feed is still at sequence 105. Omitting lowSeq here loses track of the fact that the client hasn't been sent all sequences between 100 and 105.

LowSeq should be included whenever lowSeq is lower than the overall position of the feed. For simple sequences (no triggered by), that's just when lowSeq < Seq. But if there's a backfill, the position of the feed is TriggeredBy, so you need to compare lowSeq < TriggeredBy.

[adamcfraser]

I added another table to the spreadsheet for all the String() permutations, have a look and add them as use cases for this test.

[adamcfraser]

The evaluation to determine whether lowSeq should be sent (s.Seq < s.LowSeq) is only valid when triggeredBy=0. If triggeredBy is non-zero the evaluation should be (s.Seq < s.TriggeredBy). See the notes in the unit test below.