Skip to content

PT-2237 - fixed PostgreSQL DB logs collection (Reopen)#1099

Merged
svetasmirnova merged 25 commits into3.xfrom
PT-2237_fix_pg_log_collector
Apr 27, 2026
Merged

PT-2237 - fixed PostgreSQL DB logs collection (Reopen)#1099
svetasmirnova merged 25 commits into3.xfrom
PT-2237_fix_pg_log_collector

Conversation

@BON4
Copy link
Copy Markdown
Contributor

@BON4 BON4 commented Mar 31, 2026

What was added:

  • To keep changes minimal, the existing test TestIndividualFiles was modified to cover the new case.
    The test was refactored because the previous implementation couldn't handle files with variable names.
    In /pgdata/<cluster_name>/pg_log, logs are named according to the day/week, so filenames are not known at runtime.

  • A new function getAllFilesFromDirectory was added to dumper.go.
    Its purpose is the same as the existing getIndividualFiles, but it supports extracting multiple files from a directory. It is useful because pg log filenames can vary, and there can be multiple log files.

  • The contributed code is licensed under GPL v2.0
  • Contributor Licence Agreement (CLA) is signed
  • util/update-modules has been ran
    (/lib has not been changed)
  • Documentation updated
  • Test suite update

BON4 added 18 commits January 6, 2026 16:15
This fix ensures `pt-k8s-debug-collector` collects postgreSql databse
logs which is stored in `/pgdata/<cluster_name>/pg_log`.
A test `TestIndividualFiles` was refactored, and modified for a new
case with new feature.
This refactor includes replacing all of the `kubectl` cli calls
with golang sdk for k8s. Additionaly dumper now has new structure,
new logger, tar file path controll, and multithreaded approach
for downloading and exporting files form multiple pods.
…cona-toolkit into PT-2237_fix_pg_log_collector
Copy link
Copy Markdown
Collaborator

@svetasmirnova svetasmirnova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pgo test fails:

=== RUN   TestCollectorRunner/Operator_pgo/TestIndividualFiles/Resource_pgo
    main_test.go:401: 
        	Error Trace:	/home/sveta/src/percona/percona-toolkit/src/go/pt-k8s-debug-collector/main_test.go:401
        	            				/home/sveta/go/pkg/mod/github.com/stretchr/testify@v1.11.1/suite/suite.go:115
        	Error:      	Received unexpected error:
        	            	exit status 2
        	Test:       	TestCollectorRunner/Operator_pgo/TestIndividualFiles/Resource_pgo
    main_test.go:403: 
        	Error Trace:	/home/sveta/src/percona/percona-toolkit/src/go/pt-k8s-debug-collector/main_test.go:403
        	            				/home/sveta/go/pkg/mod/github.com/stretchr/testify@v1.11.1/suite/suite.go:115
        	Error:      	Preprocessor Check
        	Test:       	TestCollectorRunner/Operator_pgo/TestIndividualFiles/Resource_pgo
        	Messages:   	test pgo_pg_logs_exist
        	            	resource:pgo
        	            	namespace: pgo
        	            	output is not as expected
        	            	Output: 
        	            	Wanted: [.log]
=== RUN   TestCollectorRunner/Operator_pgo/TestIndividualFiles/Resource_auto
    main_test.go:401: 
        	Error Trace:	/home/sveta/src/percona/percona-toolkit/src/go/pt-k8s-debug-collector/main_test.go:401
        	            				/home/sveta/go/pkg/mod/github.com/stretchr/testify@v1.11.1/suite/suite.go:115
        	Error:      	Received unexpected error:
        	            	exit status 2
        	Test:       	TestCollectorRunner/Operator_pgo/TestIndividualFiles/Resource_auto
    main_test.go:403: 
        	Error Trace:	/home/sveta/src/percona/percona-toolkit/src/go/pt-k8s-debug-collector/main_test.go:403
        	            				/home/sveta/go/pkg/mod/github.com/stretchr/testify@v1.11.1/suite/suite.go:115
        	Error:      	Preprocessor Check
        	Test:       	TestCollectorRunner/Operator_pgo/TestIndividualFiles/Resource_auto
        	Messages:   	test pgo_pg_logs_exist
        	            	resource:auto
        	            	namespace: pgo
        	            	output is not as expected
        	            	Output: 
        	            	Wanted: [.log]

Since we will deprecate pgo soon as per EOL note here: https://docs.percona.com/percona-operator-for-postgresql/1.6.0/, you can simply remove this test.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes PostgreSQL log collection in the Kubernetes debug collector by enabling extraction of all files from a log directory (where filenames vary at runtime), and updates the integration test to validate that at least one PostgreSQL .log file is included in the produced dump archive.

Changes:

  • Added support for dumping an entire directory of “individual files” (used for PostgreSQL pg_log) instead of only known, fixed filenames.
  • Added pg log directory mappings for PG v1 (“pgo”) and PG v2 (“pgv2”) resources.
  • Refactored TestIndividualFiles to run namespace-specific assertions and added checks for PostgreSQL log presence.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/go/pt-k8s-debug-collector/main_test.go Refactors individual-file tests and adds checks that PG .log files exist in the archive.
src/go/pt-k8s-debug-collector/dumper/resources.go Registers PostgreSQL log directories (pg_log) for PG v1 and PG v2 dump collection.
src/go/pt-k8s-debug-collector/dumper/kube_utils.go Introduces streaming tar execution helpers used for directory/file extraction.
src/go/pt-k8s-debug-collector/dumper/individual_files.go Reworks individual file dumping to support env substitution and directory extraction via tar streaming.
src/go/pt-k8s-debug-collector/dumper/dumper.go Normalizes CR type handling when matching pods and dumping summaries/files.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/go/pt-k8s-debug-collector/dumper/kube_utils.go Outdated
Comment thread src/go/pt-k8s-debug-collector/dumper/kube_utils.go Outdated
Comment thread src/go/pt-k8s-debug-collector/dumper/individual_files.go Outdated
Comment thread src/go/pt-k8s-debug-collector/dumper/individual_files.go Outdated
Comment thread src/go/pt-k8s-debug-collector/dumper/individual_files.go
@it-percona it-percona temporarily deployed to PT-2237_fix_pg_log_collector - percona-toolkit PR #1099 April 27, 2026 20:42 — with Render Destroyed
@svetasmirnova svetasmirnova merged commit b1299d2 into 3.x Apr 27, 2026
4 checks passed
@svetasmirnova svetasmirnova deleted the PT-2237_fix_pg_log_collector branch April 27, 2026 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants