feat(server): implement log rotation based on size and retention

[Svecco]

Redirected from issue #46 , the progress:

• Implemented _install_log_rotation_handler function, which was previously marked as todo;
• Implemented cleanup_log_files to delete logs based on the configured time threshold & size;
• Merged multiple directory traversals into a single traversal;
• Reduced multiple metadata() calls by collecting data in single traversal;
• Split log cleanup logic into smaller, more manageable functions;
• Improved error handling with detailed error messages and proper error propagation;
• Fixed an issue where the directory did not exist when loading logs;
• Added disk space checking before initializing file logging;

Before merge:

• Verified to comply with Apache specifications;
• Add a relevant unit test for logging;
• Confirm that the code has been optimized well?
• Squash commits to a proper number after polishing;
• Solve conflicts and confirm it is ready for being merged;

Plan:
Around the next several weekends.

[hubcio]

I think changes in core/message_bus are not related to log retention, looks like merge problem.

[Svecco]

I think changes in core/message_bus are not related to log retention, looks like merge problem.

Yes, that was the case. I have just rebased it to the correct state.

[Svecco]

@hubcio @spetz I've just squashed commits into one.

[hubcio]

I tested your code manually. In server.toml i changed settings to:

max_file_size = "1 MB"
max_total_size = "10 MB"
rotation_check_interval = "1 s"

then I rebulilt server and started it via:
RUST_LOG=trace target/debug/iggy-server --with-default-root-credentials --fresh

and in other shell i run benchmark to generate logs:
target/debug/iggy-bench -T 1000MB pp -p 1 tcp

result:
I can never see more than 2 files, it looks like retention is ignored and 10 MB of max total size is not reached.

can you write integration test that would test for this scenario? see how TestServer is used, in integration module (e.g. core/integration/tests/server/specific.rs)

[Svecco]

Hi, I've completed some initial modifications and added integration tests.

I first manually observed the runtime behavior locally by running in the terminal:
RUST_LOG=trace cargo run --bin iggy-server -- --fresh --with-default-root-credentials &&
cargo run --bin iggy-bench -- -T 1000MB pp -p 1 quic
I verified that the current log rotation behavior is working as expected by monitoring with
watch -n 0.1 "eza -la ~/Apache/iggy/local_data/logs"

eza -la with 10:100

.rw-r--r--@ 2.4M svecco 11 Jan 13:10 iggy-server.log
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.1
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.2
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.3
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.4
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.5
.rw-r--r--@  10M svecco 11 Jan 13:04 iggy-server.log.6
.rw-r--r--@  10M svecco 11 Jan 13:03 iggy-server.log.7
.rw-r--r--@  10M svecco 11 Jan 13:03 iggy-server.log.8
.rw-r--r--@  10M svecco 11 Jan 13:03 iggy-server.log.9

Additionally, the result of the simple integration test cargo test -p integration server::specific::log_rotation_should_launch -- --nocapture execution seems fine, by generating sufficient logs by sending a large volume of messages to trigger the log rotation mechanism.

Console

╭─ $ [svecco] ~/A/iggy git!(46-size-logs)
╰─ > cargo test -p integration server::specific::log_rotation_should_launch -- --nocapture
   Compiling libz-sys v1.1.23
   Compiling liblzma-sys v0.4.4
   Compiling hwlocality-sys v0.6.4
   Compiling libgit2-sys v0.18.3+1.9.2
   Compiling liblzma v0.4.5
   Compiling hwlocality v1.0.0-alpha.11
   Compiling compression-codecs v0.4.35
   Compiling async-compression v0.4.36
   Compiling async_zip v0.0.18
   Compiling git2 v0.20.3
   Compiling vergen-git2 v1.0.7
   Compiling server v0.6.1-edge.3 (/home/svecco/host/apache/iggy/core/server)
   Compiling integration v0.0.1 (/home/svecco/host/apache/iggy/core/integration)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7.71s
     Running unittests src/lib.rs (target/debug/deps/integration-375a7f260e09ad62)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/mod.rs (target/debug/deps/mod-0f48c9a972f7dba1)

running 1 test
Log directory verified: local_data_136642763006894046097605999160554396417/logs
Succeeded Verified Log Rotation
test server::specific::log_rotation_should_launch ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 1190 filtered out; finished in 2.12s