PMM-15398 Count shared memory as used, not cache - #5855
Conversation
Used was MemTotal - (MemFree + Buffers + Cached). /proc/meminfo:Cached includes Shmem, and PostgreSQL shared_buffers is anonymous shared memory, so every byte of it was subtracted out of Used and re-drawn as Cache. A node with shared_buffers=8G reported ~1.5G used while free(1) reported ~10G. MySQL was unaffected because it allocates AnonPages. Used now subtracts (Cached + SReclaimable - Shmem) instead of Cached, and the Cache series plots that same quantity, so the stacked panels still sum to MemTotal exactly. Folding in SReclaimable also addresses PMM-11119. rds_exporter exports MemTotal, MemFree, Buffers and Cached but not SReclaimable or Shmem, so the new terms carry a zero-fallback: Cached + (SReclaimable - Shmem or Cached * 0) With those metrics absent the result is bit-identical to the previous expression, leaving RDS nodes unchanged. The same expression is cloned in the telemetry config; it is updated here so the shipped pmm_server_node_memory_usage series does not diverge from the dashboard it was copied from. This is not parity with free(1), which uses MemTotal - MemAvailable. Measured on a PostgreSQL 18 node with shared_buffers=1G, the error against free drops from -14.2% to -3.1%. The Used line steps upward on existing installations, so alert rules keyed to the old expression will shift. Signed-off-by: theTibi <tkorocz@gmail.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5855 +/- ##
==========================================
+ Coverage 43.59% 43.80% +0.21%
==========================================
Files 415 432 +17
Lines 43134 35298 -7836
Branches 0 591 +591
==========================================
- Hits 18804 15463 -3341
+ Misses 22454 18311 -4143
+ Partials 1876 1524 -352 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
🔗 Linked repositories identifiedCodeRabbit considers these linked repositories for cross-repo context during reviews:
Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review. WalkthroughThe change updates memory calculations in four OS dashboards and the Merge Risk: ⚪ Minimal · up to The change updates memory accounting expressions and their matching telemetry definition across the affected dashboards. No actionable merge-blocking risk remains beyond normal checks and review. 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Full details: Description checkExplanation The description is complete and directly related to the changes. It includes the ticket, feature build, root cause, implementation details, affected panels, compatibility behavior, verification results, risks, and related work. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2 participants
Ticket number: PMM-15398
Feature build: Percona-Lab/pmm-submodules#4550
Summary
Usedmemory never counts PostgreSQLshared_buffers. The reporter variedshared_buffersfrom 128 MB to 8 GB on PostgreSQL 18.6 and PMM never showed more than ~2 GB used, whilefreeshowed ~10 GB.Used = MemTotal - (MemFree + Buffers + Cached)./proc/meminfo:CachedincludesShmem, andshared_buffersis anonymous shared memory — so it was subtracted out ofUsedand re-plotted asCache. MySQL looked correct only because it allocatesAnonPages.Usednow subtracts(Cached + SReclaimable - Shmem)instead ofCached, and theCacheseries plots that same quantity. Folding inSReclaimablealso addresses PMM-11119.Panels changed
OS/Node_Summary.jsonOS/Memory_Details.jsonOS/Nodes_Compare.jsonOS/Nodes_Overview.jsonmanaged/services/telemetry/config.default.ymlPMMNodeMemoryUsageThe telemetry entry carries a character-for-character clone of the
Node_Summaryexpression; without it the shippedpmm_server_node_memory_usageseries would permanently disagree with the dashboard it was copied from.RDS is deliberately unaffected
percona/rds_exporterexportsnode_memory_{MemTotal,MemFree,Buffers,Cached}_bytesbut notSReclaimable,ShmemorMemAvailable. A plain rewrite would make these panels return an empty series for every RDS node. The new terms therefore carry a zero-fallback:orbinds loosest, so this is(SReclaimable - Shmem) or (Cached * 0). Verified by fault injection against a live VictoriaMetrics: with those metrics absent the output is bit-identical to the previous expression. Note this relies on MetricsQL ignoring__name__in set-operator matching (Prometheus does not) — the same property PMM's existingMemAvailable or (…)idiom already depends on.Verification
Live PMM Server 3.9.0, real PostgreSQL 18.6 with
shared_buffers=1GBprewarmed viapg_prewarm, 905 MB pgbench dataset, nodepmm15398-pg-node. Before/after captured over the identical 23-minute window by swapping only the dashboard JSON./proc/meminfoat that moment:Shmem = 1,044,932 kB(theshared_buffers),SReclaimable = 374,984 kB.free -mreported 6393 MB used.MemTotalexactly — residual 0.000 MiB.freedrops from -14.2% to -3.1%.Please read before approving
free(1). ModernfreecomputesMemTotal - MemAvailable, a third definition again. This change closes most of the gap but a ~3% residual remains, caused byMemAvailable's watermark and half-slab discount.Home_Dashboard→ "Top 20 Used Memory" already uses theMemAvailabledefinition, so it will sit ~200 MB above "Top 5 Used Memory" on the same node. Before this change that gap was ~870 MB.Usedline steps upward on every existing installation. Alert rules and customer baselines keyed to the old expression will shift. This needs a release note.freeexactly is not available, becauseMemTotal - MemAvailablebreaks theUsed + Free + Buffers + Cache = MemTotalidentity the stacked panels depend on.Test plan
python3 dashboards/misc/cleanup-dash.py --check-only <file>— clean on all 4 dashboardsplugin.jsonreferenced-path check and JSON validitygo test ./managed/services/telemetry/ -run TestServiceConfigUnmarshal— PASS (TestRunTelemetryServicefails identically onmain; it needs a testdb Postgres)shared_buffers=1GB, identical time windowmainRelated
SReclaimablehalf of PMM-11119 (Take into account slab cache in Memory Utilization)Memory_Details.jsonhas a pre-existingav_over_timetypo (missingg) in the hidden "Used Swap" target of the same panel. Dormant because the target ishide: true; present onmain.