Skip to content

PS-11393 - Add more group replication test cases - #9

Open
tplavcic wants to merge 10 commits into
mainfrom
ps-11393-group-replication
Open

PS-11393 - Add more group replication test cases#9
tplavcic wants to merge 10 commits into
mainfrom
ps-11393-group-replication

Conversation

@tplavcic

@tplavcic tplavcic commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

This is adding following test cases in PS GR framework:
test_secondary_isolation_ist.py - secondary network-partitioned, rejoins via IST
test_secondary_isolation_sst.py - same, binlogs purged so it rejoins via clone/SST
test_primary_isolation_failover.py - primary partitioned: automatic failover
test_majority_loss.py - both secondaries cut off: quorum loss, no writes
test_equal_partition.py - 4-node 2-2 split: split-brain prevention

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a set of Group Replication (GR) network-partition and quorum-loss test cases to the Percona Server GR test framework, plus supporting framework and documentation updates to reliably simulate partitions, observe membership from surviving nodes, and recover groups after quorum loss.

Changes:

  • Introduces new GR partition tests covering secondary IST/SST recovery, primary isolation failover, majority loss, and 4-node equal split behavior.
  • Extends the GR helper framework with network-partition primitives (disconnect/connect, link sever/restore), quorum recovery helpers, and improved observer selection for membership polling.
  • Updates docs/README and refines proxy startup/refresh behavior to account for partitions and IP changes.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
test_scripts/ps/group-replication/test_secondary_isolation_sst.py New test: isolate a secondary, purge binlogs on donors, validate clone/SST recovery and data consistency.
test_scripts/ps/group-replication/test_secondary_isolation_ist.py New test: isolate a secondary, heal before binlog purge, validate IST recovery and data consistency.
test_scripts/ps/group-replication/test_primary_shutdown_failover.py Renames/clarifies the shutdown-based failover test and points to the new isolation variant.
test_scripts/ps/group-replication/test_primary_isolation_failover.py New test: isolate the primary via network partition, assert automatic failover, write refusal on isolated primary, and rejoin behavior.
test_scripts/ps/group-replication/test_majority_loss.py New test: quorum loss (3-node), assert writes refused, then force/recover membership and validate no leaked writes.
test_scripts/ps/group-replication/test_equal_partition.py New test: 4-node 2–2 split using in-container route blackholing, assert no split brain and operator recovery via force_members().
test_scripts/ps/group-replication/README.md Documents new tests and adds detailed guidance for partition/quorum scenarios and helper APIs.
test_scripts/ps/group-replication/group_replication_helper.py Adds partition/heal/force-members helpers, observer selection for polling, proxy refresh changes, and other test-enabling utilities.
test_scripts/ps/group-replication/docker_helper.py Adds capabilities/network helpers (cap_add, network connect/disconnect, container state/IP) used by new partition tests.
test_scripts/ps/group-replication/conftest.py Extends fixture parameterization to support variable cluster sizes (e.g., 4-node equal partition test).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test_scripts/ps/group-replication/docker_helper.py
Comment thread test_scripts/ps/group-replication/group_replication_helper.py
Comment thread test_scripts/ps/group-replication/conftest.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

test_scripts/ps/group-replication/group_replication_helper.py:445

  • wait_members_unreachable() returns the last observed membership map on timeout (it does not guarantee members are UNREACHABLE), but the docstring reads like it always waits until the condition is met. This can mislead callers into assuming the condition held without checking the return value.
    def wait_members_unreachable(
        self, names: list[str], node: str, timeout: int = 60
    ) -> dict[str, tuple[str, str]]:
        """Wait until every named member is seen as UNREACHABLE from `node`; return that view."""

test_scripts/ps/group-replication/README.md:576

  • README says wait_members_unreachable() will "block until" members are UNREACHABLE, but the helper returns the last view when the timeout expires. Update the wording so future test authors know they still need to assert on the returned view.
- `gr_cluster.wait_members_unreachable(nodes, node=...)` — block until the given members are
  seen as `UNREACHABLE` from an observer node, and return that view.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants