MySQL Router Docker Test Job - #531
Merged
Merged
Conversation
kaushikpuneet07
force-pushed
the
PS-8631
branch
6 times, most recently
from
September 8, 2023 13:30
6adf3e0 to
10cd0a4
Compare
kaushikpuneet07
force-pushed
the
PS-8631
branch
from
September 10, 2023 06:37
10cd0a4 to
55f229b
Compare
Remove '-e' before '--net'
kaushikpuneet07
force-pushed
the
PS-8631
branch
from
September 28, 2023 11:18
ed9c828 to
d09bdc3
Compare
kaushikpuneet07
force-pushed
the
PS-8631
branch
from
September 29, 2023 07:37
cf9ec71 to
a4a8d36
Compare
kaushikpuneet07
force-pushed
the
PS-8631
branch
from
October 4, 2023 06:40
1ebd946 to
a512db0
Compare
kaushikpuneet07
force-pushed
the
PS-8631
branch
from
October 4, 2023 06:41
a512db0 to
952d9da
Compare
- Remove -it from all docker exec calls, which fails under Jenkins'
non-interactive sh step ("input device is not a TTY")
- Replace unrendered {{ ansible_hostname }} template with mysql$N
- Fix missing space in sysbench args (--mysql-port=6446--mysql-user)
- Write cluster1.json/cluster.json with > instead of >> so
Fault_tolerance doesn't accumulate stale data across reruns
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test_mysqlrouter_directory_permissions, test_mysql_user, test_mysqlrouter_ports, and test_mysqlrouter_config declared a test_router_bootstrap fixture parameter that has never existed in this codebase, while their bodies referenced the module-level host fixture directly. This left them one commit away from failing with "fixture 'test_router_bootstrap' not found". Switch all four to take host, matching test_mysqlrouter_version and test_mysqlsh_version. Also add the missing assert keywords in test_mysqlrouter_ports, which previously evaluated the socket checks and discarded the result, always passing regardless of whether the ports were open. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Cluster setup in test_router_static.py runs at module import time,
outside any pytest fixture, so it had no teardown at all for the
mysql1-4 containers, the mysql-router container, or the innodbnet
network. A crash during setup (as seen when mysql1 died before
create_new_user() could run) left everything stranded on the agent,
which could then break the next run ("network already exists",
"name already in use").
Add cleanup_cluster(), call it before setup to clear anything left
over from a prior crashed run, call it again on setup failure before
re-raising, and register it with atexit so it also runs at the end
of a normal (pass or fail) test session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
--recoveryMethod=increamental is not a valid mysqlsh option value (it should be incremental), so every cluster add-instance call in add_slave() likely fails immediately. Since these run with check=False, the failure is silently swallowed and the script still sleeps out all four 120s waits regardless, burning CI time on add-instance attempts that never actually join a node to the cluster. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test_router_static.py's create_mysql_config() still wrote the deprecated slave_preserve_commit_order/slave_parallel_type variable names, while router-docker_test.sh already uses the modern replica_preserve_commit_order/replica_parallel_type equivalents. On any MySQL version where the deprecated aliases have been removed, mysqld would refuse to start with these in my.cnf, which lines up with one cluster node consistently dying shortly after startup. Align both scripts to the same, current variable names. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
subprocess.run/check_call/check_output calls for docker exec, mysqlsh cluster operations, and microdnf install had no timeout, so a stuck operation (e.g. mysqlsh retrying against an unhealthy cluster member) would hang the whole test run forever with no further output. Bound each call: quick docker/mysql client commands at 30-60s, docker run/rm and microdnf install at 60-120s, cluster create/add-instance/gtid-reset at 180-300s. A stuck call now fails with a clear TimeoutExpired instead of hanging indefinitely. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add tests/test_router_cluster_ha.py alongside the existing test_router_attr.py/test_router_static.py: it asserts the sbtest data is reachable and correct via the router, replicated on the surviving nodes, and that cluster status is OK_PARTIAL after Fault_tolerance stops mysql1. It does no setup of its own - it expects router-docker_test.sh to have already built the cluster, loaded data, and triggered the fault-tolerance check, and is meant to be run explicitly (./run.sh tests/test_router_cluster_ha.py) so router_docker's default ./run.sh collection is unaffected. Remove router-docker_test.sh's verify_status(), since the new pytest suite now owns that assertion instead of a bash-level exit code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fault_tolerance() (and the matching pytest assertion) ran mysqlsh ... cluster status through mysql-router:6446, but AdminAPI commands need a direct session to a cluster member to read cluster metadata - a connection proxied through the router doesn't provide that. The captured "JSON" ended up being just mysqlsh's warning banner, which jq/json.loads then failed to parse. Connect directly to a surviving member (mysql2, since mysql1 is the one just stopped) instead, matching how create_cluster()/add_slave() already connect directly to a member rather than through the router. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fault_tolerance()'s cluster status check has now failed identically twice - once via mysql-router, once connecting directly to mysql2 - with jq unable to parse cluster.json either time. Rather than guess a third fix blind, dump the raw captured output to the console so the next run shows exactly what mysqlsh produced. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The raw cluster1.json dump confirmed mysqlsh's locale/password
warnings go to stderr, not stdout - stdout captured via `>` is
already complete, valid JSON with the status correctly showing
OK_PARTIAL after mysql1 is stopped. sed '1d' was unconditionally
deleting the JSON's opening '{' assuming a banner line that was
never in this stream, which is what made jq fail. Drop the sed step
and just copy the file. Fix the identical assumption in the
matching pytest assertion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The captured setup output confirmed microdnf install net-tools was printing "Is this ok [y/N]:" and then blocking forever waiting for input that never comes under a non-interactive docker exec, until our recently-added timeout killed it after 120s. Pass -y so it doesn't prompt at all. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…er membership test_router_attr.py's inspect_data fixture ran the router container with no env vars at all, so the entrypoint had nothing to bootstrap against and exited immediately - that's what made test_status report 'exited' in build Percona-QA#10, the first run of this newly-wired-in phase. Point it at the mysql1-4 cluster that test_router_static.py already builds at import time, on the same innodbnet network. Separately, add_slave() runs each add-instance call with check=False and only prints on failure, so a node that never joined the cluster went unnoticed until the router/HA tests failed downstream with confusing symptoms (closed ports, a half-written mysqlrouter.conf stuck at 0o777). Add verify_cluster_membership() to fail loudly right after add_slave(), at the point the cluster actually broke. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ll cluster membership" This reverts commit 7fd4092.
severalnines/sysbench has no arm64 build on Docker Hub, so data_add()'s docker run of it would fail outright once this pipeline runs on the docker-32gb-aarch64 fleet node to test the arm64 percona-server/router images. percona-server and percona-mysql-router are proper multi-arch manifests, so nothing else in the flow needed to change. Replace the sysbench oltp_insert.lua prepare step with an equivalent CREATE TABLE + INSERT ... SELECT (using ROW_NUMBER() OVER () against information_schema.columns to cheaply generate 10000 rows) run through the mysql client already in the mysql-client container. That container image is multi-arch, so this works on amd64 and arm64 alike, and drops the dependency on an unmaintained personal Docker Hub image entirely. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
mysql1 has been intermittently dying within the 60s startup window before create_new_user ever runs, surfacing only as an opaque "container ... is not running" from docker exec - the actual mysqld error was never visible. innodb_dedicated_server=ON auto-sizes the buffer pool off the *host's* total memory since these containers have no -m limit, so 4 of them each grabbing ~75% of host RAM on the same node is a plausible way for one to fail to allocate and exit right after start. Pin innodb_buffer_pool_size to a small fixed value so each node's footprint is bounded regardless of host RAM. Also check each container is actually running right after start and dump its log immediately if not, so if this happens again we get the real mysqld error instead of tracking it down after the fact. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… memory The startup-log dump added last time paid off immediately: build Percona-QA#13's mysql1 crash was InnoDB failing to write its redo log with OS error 28, "No space left on device" - not an OOM from innodb_dedicated_server as guessed. cleanup() only removes this job's own named containers/network, so unused image layers and stopped containers from prior runs pile up on this node's docker data-root across builds until it's actually full. Add reclaim_disk_space(), run right after cleanup() and before anything else: log df -h for visibility, then docker system prune -af --volumes to clear out that leftover cruft. Safe here since this label runs one executor per node (execMap['docker'] = '1' in the ps80.cd cloud config), so nothing else is using this node's docker daemon concurrently. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mirrors the assertion pattern from test_router_static.py's test_mysqlrouter_version/test_mysqlsh_version (docker_tag/ps_version as a substring of --version output), applied to the containers this suite actually has running: mysql-router for ROUTER_VERSION, and mysql2 for PS_VERSION since mysql1 is already stopped by Fault_tolerance() by the time pytest runs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
mysql1 was failing --initialize on PS 8.4.10-10 with: [ERROR] unknown variable 'binlog_transaction_dependency_tracking=WRITESET'. That variable was deprecated in 8.0 (still worked there, just a warning) and removed outright in 8.4, so writing it unconditionally into every my$N.cnf broke every 8.4+ run. create_mysql_config() now takes the percona-server image ref and only emits that line for 8.0.x tags. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…g by version It's removed in PS 8.4+ and was only ever a deprecated performance-tuning knob for writeset dependency tracking on 8.0, not something these functional tests need. Simpler to just not set it for any version than to carry a version-conditional case statement in create_mysql_config() - drops the need for it to take the image ref at all. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
mysql1-4 now start fine on PS 8.4 (disk reclaim + dropping binlog_transaction_dependency_tracking both held), but create_new_user() still failed on 8.4.10-10 with a syntax error on 'reset master' - that statement was renamed to RESET BINARY LOGS AND GTIDS in 8.4 and the old name doesn't exist there at all. Try the new name first, fall back to the old one on failure, so this works across 8.0 and 8.4+ without needing to pass the PS version into create_new_user() at all. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Renames the function/call-site naming in both router-docker_test.sh and test_router_static.py. Left untouched: the literal 'reset master;' SQL fallback and its explanatory comment in router-docker_test.sh - that's the actual deprecated command name MySQL 8.0 still requires, not our own naming, so it has to stay verbatim for the fallback to work. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
data_add() failed on PS 8.4.10-10 with: ERROR 1524 (HY000): Plugin 'mysql_native_password' is not loaded That plugin isn't loaded by default on 8.4+/9.7+. Just let CREATE USER use the server's default auth plugin instead, same as the 'inno' user is already created elsewhere in this script without specifying one - that already works across 8.0 and 8.4 alike. Also updated the matching line in test_router_static.py's disabled/commented test_data_add for consistency, even though it isn't executed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Dropped from both router-docker_test.sh's create_mysql_config() and the equivalent (inactive) one in test_router_static.py. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.