statistics: ignore system tables in stats cache by 0xPoe · Pull Request #64097 · pingcap/tidb

0xPoe · 2025-10-22T09:52:50Z

What problem does this PR solve?

Issue Number: close #64080, close #57176

Problem Summary:

What changed and how does it work?

Avoid putting system tables into the stats cache, as it would pollute the cache.
Do not always place small tables in the “unneeded” range; otherwise, it may confuse users into thinking the table hasn’t been analyzed when it actually has.
Added a pseudo label to separate it from the unneeded label.

After these changes: total tables = pseudo tables + unneeded analyze tables + tables in healthy buckets.

Check List

Tests

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

修复系统表会污染表健康度监控的问题
Fix the issue where system tables could pollute the stats health metrics

codecov · 2025-10-22T10:52:19Z

Codecov Report

❌ Patch coverage is 88.88889% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.7222%. Comparing base (82e2c6f) to head (cf127b0).
⚠️ Report is 45 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #64097        +/-   ##
================================================
+ Coverage   72.6962%   74.7222%   +2.0259%     
================================================
  Files          1865       1889        +24     
  Lines        506195     523002     +16807     
================================================
+ Hits         367985     390799     +22814     
+ Misses       115838     108211      -7627     
- Partials      22372      23992      +1620

Flag	Coverage Δ
integration	`48.1535% <47.8632%> (?)`
unit	`72.5433% <88.8888%> (+0.2772%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.8700% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`62.3775% <ø> (+16.1619%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull Request Overview

This PR prevents system tables from being cached in the statistics cache to avoid memory overhead and unnecessary cache pollution. The change addresses issues where commands like show stats_meta and show stats_healthy would cause system tables to be added to the stats cache.

Key Changes:

Added logic to detect and skip caching pseudo table statistics for system tables
Refactored stats healthy metrics calculation with improved code structure
Added test coverage to verify system tables are not cached

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pkg/statistics/handle/handle.go	Core logic to filter out system tables from stats cache using schema detection
pkg/statistics/handle/handletest/handle_test.go	Test case verifying system tables don't pollute the cache after show commands
pkg/statistics/handle/cache/statscache.go	Refactored healthy metrics calculation with cleaner bucket indexing logic
pkg/statistics/handle/cache/statscache_test.go	New test for UpdateStatsHealthyMetrics with mock infrastructure
pkg/statistics/handle/handletest/BUILD.bazel	Incremented shard count for additional test
pkg/statistics/handle/cache/BUILD.bazel	Added dependencies for new test utilities
pkg/statistics/handle/BUILD.bazel	Added filter package dependency

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot

Pull Request Overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

0xPoe

🔢 Self-check (PR reviewed by myself and ready for feedback)

Code compiles successfully
Unit tests added
All tests pass
Bazel files updated
Comments added where necessary
PR title and description updated
Documentation PR created (or confirmed not needed)
PR size is reasonable

/cc @time-and-fate @winoros

0xPoe · 2025-11-12T11:08:40Z

/test all

test: fix borken tests test: fix broken tests fix: update bazel fix fix: update fix test: fix broken tests fix: bazel update fix: rename fix: better code refatcor: better code fix: better code test: assert the labels fix: remove useless file fix: make lint happy fix: correct the failpoint name fix: update comments feat: add isMemSchemaID function feat: add system schema cache refator: use IsMemSchemaID fix: make lint happy fix: move to the right place fix: remove fix doc: add comments test: assert the cache fix: make lint happy

0xPoe · 2025-11-12T13:45:30Z

Just rebased. Nothing changed.

0xPoe · 2025-11-12T16:51:50Z

/test all

0xPoe · 2025-11-12T18:27:00Z

/retest

0xPoe · 2025-11-12T21:19:09Z

/retest

0xPoe · 2025-11-13T09:46:33Z

/test all

0xPoe · 2025-11-13T10:35:41Z

/retest

0xPoe · 2025-11-13T10:59:53Z

/retest

0xPoe · 2025-11-13T11:45:30Z

/retest

0xPoe · 2025-11-13T14:46:17Z

/test all

0xPoe · 2025-11-13T17:30:32Z

/retest

0xPoe · 2025-11-13T21:04:28Z

/retest

0xPoe · 2025-11-13T22:21:43Z

/retest

fix: no need to assert type fix: check if the pool is nil fix

0xPoe · 2025-11-21T12:06:41Z

/retest

0xPoe · 2025-11-21T12:49:49Z

/retest

ti-chi-bot · 2025-11-21T13:31:58Z

In response to a cherrypick label: new pull request created to branch release-7.5: #64627.
But this PR has conflicts, please resolve them!

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-11-21T13:32:42Z

In response to a cherrypick label: new pull request created to branch release-8.5: #64628.
But this PR has conflicts, please resolve them!

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

close #57176, close #64080

0xPoe force-pushed the poe-patch-ignore-sys-tables branch from 657ec9b to 491112f Compare October 22, 2025 09:53

0xPoe changed the title ~~WIP: statistics: remove global max table id variable~~ WIP: statistics: ignore system tables in stats cache Oct 22, 2025

0xPoe mentioned this pull request Oct 22, 2025

statistics: skip system table when to metric stats health in the next-gen #64087

Closed

13 tasks

0xPoe force-pushed the poe-patch-ignore-sys-tables branch from 93b7a77 to 2c31275 Compare October 22, 2025 09:59

ti-chi-bot Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 22, 2025

ti-chi-bot Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 22, 2025

0xPoe requested a review from Copilot October 22, 2025 11:42

Copilot AI reviewed Oct 22, 2025

View reviewed changes

Comment thread pkg/statistics/handle/handle.go Outdated

Comment thread pkg/statistics/handle/cache/statscache.go

ti-chi-bot Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 22, 2025

0xPoe force-pushed the poe-patch-ignore-sys-tables branch from 7a5d152 to 61ab8e0 Compare October 22, 2025 13:09

0xPoe changed the title ~~WIP: statistics: ignore system tables in stats cache~~ statistics: ignore system tables in stats cache Oct 22, 2025

ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 22, 2025

0xPoe requested a review from Copilot October 22, 2025 13:26

Copilot AI reviewed Oct 22, 2025

View reviewed changes

Comment thread pkg/statistics/handle/cache/statscache.go Outdated

0xPoe commented Oct 22, 2025

View reviewed changes

ti-chi-bot Bot requested review from time-and-fate and winoros October 22, 2025 13:29

ti-chi-bot Bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Oct 22, 2025

0xPoe added 4 commits November 12, 2025 14:42

refactor: avoid mess the real function by failpoint

e45cf4c

fix: use better API

a432e29

fix: import

05db961

0xPoe force-pushed the poe-patch-ignore-sys-tables branch from c6a13a8 to 05db961 Compare November 12, 2025 13:44

fix: do not call with sctx

cf127b0

fix: no need to assert type fix: check if the pool is nil fix

0xPoe force-pushed the poe-patch-ignore-sys-tables branch from fd54cdf to cf127b0 Compare November 21, 2025 11:35

ti-chi-bot Bot merged commit 9f3ae48 into pingcap:master Nov 21, 2025
32 checks passed

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Nov 21, 2025

This is an automated cherry-pick of pingcap#64097

f72e18a

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Nov 21, 2025

statistics: ignore system tables in stats cache (#64097) #64627

Open

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Nov 21, 2025

This is an automated cherry-pick of pingcap#64097

18a5922

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Nov 21, 2025

statistics: ignore system tables in stats cache (#64097) #64628

Merged

13 tasks

ti-chi-bot Bot pushed a commit that referenced this pull request Dec 9, 2025

statistics: ignore system tables in stats cache (#64097) (#64628)

23dadf9

close #57176, close #64080

Conversation

0xPoe commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

codecov Bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

0xPoe left a comment

Choose a reason for hiding this comment

Uh oh!

0xPoe commented Nov 12, 2025

Uh oh!

0xPoe commented Nov 12, 2025

Uh oh!

0xPoe commented Nov 12, 2025

Uh oh!

0xPoe commented Nov 12, 2025

Uh oh!

0xPoe commented Nov 12, 2025

Uh oh!

0xPoe commented Nov 13, 2025

Uh oh!

0xPoe commented Nov 13, 2025

Uh oh!

0xPoe commented Nov 13, 2025

Uh oh!

0xPoe commented Nov 13, 2025

Uh oh!

0xPoe commented Nov 13, 2025

Uh oh!

0xPoe commented Nov 13, 2025

Uh oh!

0xPoe commented Nov 13, 2025

Uh oh!

0xPoe commented Nov 13, 2025

Uh oh!

0xPoe commented Nov 21, 2025

Uh oh!

0xPoe commented Nov 21, 2025

Uh oh!

Uh oh!

ti-chi-bot commented Nov 21, 2025

Uh oh!

ti-chi-bot commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

0xPoe commented Oct 22, 2025 •

edited

Loading

codecov Bot commented Oct 22, 2025 •

edited

Loading