Feature/rosbag2 circular logging size duration 2217

[luke-alloy]

Description

Adds circular logging by split count, record size and duration to rosbag2 recording.

• CLI: adds --max-splits --max-bag-files
• Writer: deletes oldest files when exceeding split count

Fixes #2217

Is this user-facing behavior change?

Yes.

• Older recordings are deleted once max split is reached:
  • --max-splits N --max-bag-files: limit number of retained bagfile splits.
• Examples:
  • ros2 bag record -a --max-bag-size 100MB --max-bag-files 5

Did you use Generative AI?

Yes.

• Tool/model: GPT-5
• Used for: naming suggestions, CLI/help text wording, and drafting PR description.
• Code impact: assisted in scaffolding changes (option wiring, param names); final code implementation was completed, built, run, and manually reviewed and tested locally.

Additional Information

• The max split limit is checked together after each bag split
• The writer logs a warning when deleting files due to limits. Adjust --log-level if needed.
• metadata.yaml currently retains aggregate message_count and topics_with_message_count across the entire capture, even when older bagfiles are pruned by circular logging. File lists and durations reflect pruning, but those two fields still represent the full session totals.
• Circular logging changes are only in sequential_writer.cpp (deleting the oldest file if needed); the rest involve adding StorageOptions parameters, updating the shell UI, and adjusting related unit tests.

[luke-alloy]

I've committed changes that should address the unit test error. I'm also happy to include @fujitatomoya ’s suggestion to remove --max-record-size and --max-record-duration, pending confirmation to proceed.

[MichaelOrlov]

@luke-alloy Thank you for your contribution. This is a long-time-wanted feature that will be very useful.

I've made a thorough review, and overall implementation looks good to me. However, I have a few suggestions:

1. The naming is hard. I would propose to rename --max-splits parameter to --max_bag_files to avoid confusion and multiple user questions, like:

• Will splits stop after reaching max_splits?
• Why is there one file when max_splits = 1? I would expect to have at least two files after the split.

2. I would like to see at least one integration test with real files writing and deletion.
   To facilitate this, we do have a special TemporaryDirectoryFixture. Please consider modifying your tests or adding another one. I recently added a similar test in one of my PR. Please, refer to the 81ef9fd#diff-c14a7d44179cf64868ce86b5e60c7f741f957ef368c04a0477ccfe7ccd89f1f2R138-R164 as an example from the #2224.

[luke-alloy]

@luke-alloy Thank you for your contribution. This is a long-time-wanted feature that will be very useful.

I've made a thorough review, and overall implementation looks good to me. However, I have a few suggestions:

1. The naming is hard. I would propose to rename --max-splits parameter to --max_bag_files to avoid confusion and multiple user questions, like:

• Will splits stop after reaching max_splits?
• Why is there one file when max_splits = 1? I would expect to have at least two files after the split.

2. I would like to see at least one integration test with real files writing and deletion.
   To facilitate this, we do have a special TemporaryDirectoryFixture. Please consider modifying your tests or adding another one. I recently added a similar test in one of my PR. Please, refer to the 81ef9fd#diff-c14a7d44179cf64868ce86b5e60c7f741f957ef368c04a0477ccfe7ccd89f1f2R138-R164 as an example from the Fix for C++ Recorder failure on stop() -> record() due to reusing the bag name #2224.

@MichaelOrlov Thanks for the detailed review! I will update the PR soon.

I completely agree regarding the naming confusion—good catch. To maintain consistency with the other CLI flags, would --max-bag-files (using hyphens instead of _) be the preferred convention?

[MichaelOrlov]

@luke-alloy Yes. Need to use --max-bag-files (using hyphens instead of _).
Sorry for the confusion. It was my mistake, I copied and pasted the name from another post where I was referring to the storage_options

[luke-alloy]

Committed changes to address (most) review comments.
Will address the last comment (ie, add integration test with real file writing and deletion) next.

[luke-alloy]

@MichaelOrlov Thanks for the review. I've pushed fixes addressing the comments. Ready for a re-review!

[MichaelOrlov]

@luke-alloy Thank you for the follow-up.
As regards

metadata.yaml currently retains aggregate message_count and topics_with_message_count across the entire capture, even when older bagfiles are pruned by circular logging. File lists and durations reflect pruning, but those two fields still represent the full session totals.

I think it is a sort of inconsistency and would be considered as a bug.
I would suggest adjusting message_count and topics_with_message_count accordingly when old files are deleted.

Update:
I also think that there are no values in knowing the total messages in a session if messages are not available.

[MichaelOrlov]

@luke-alloy Thank you one more time.

I've made one more round of review. Overall looks good to me, with one concern regarding adjustments of the message_count and topics_with_message_count in metadata when we are deleting old files, and a few nitpicks findings in the newly added unit test.

[MichaelOrlov]

We do have a make_test_msg(); helper function

Suggested change

	auto msg = std::make_shared<rosbag2_storage::SerializedBagMessage>();
	msg->topic_name = topic_name;
	msg->recv_timestamp = static_cast<rcutils_time_point_value_t>(i * 100);
	msg->send_timestamp = msg->recv_timestamp;
	uint32_t data = static_cast<uint32_t>(i);
	msg->serialized_data = rosbag2_storage::make_serialized_message(&data, sizeof(data));
	auto msg = make_test_msg();
	msg->topic_name = topic_name;
	msg->recv_timestamp = static_cast<rcutils_time_point_value_t>(i * 100);
	msg->send_timestamp = msg->recv_timestamp;```

[MichaelOrlov]

@luke-alloy Can you please also regenerate Python stub files (.pyi) according to the https://github.com/ros2/rosbag2/blob/rolling/rosbag2_py/README.md ?
The Test rosbag2 / build_and_test (pull_request) fials due to this.

[luke-alloy]

@MichaelOrlov Thanks for the review. I've pushed fixes addressing the comments. Ready for a re-review!

Thanks for the feedback! I agree this should be fixed.

The challenge: FileInformation only stores total message_count per file, not per-topic counts. So fixing topics_with_message_count requires tracking per-file per-topic counts somewhere.

Two options:

Option	Approach	Trade-off
A	Add topic_message_counts to FileInformation	Persisted, but requires metadata version bump (9→10), YAML serialization changes, and touches readers/info tools
B	Track per-file per-topic counts in-memory in SequentialWriter	Simpler & localized, but lost on restart

Plan: I'll go with Option B since circular logging is a runtime feature anyway—if the process restarts, the session ends. Smaller blast radius too. Can extend to A later if needed.

Let me know if you'd prefer Option A.

[MichaelOrlov]

@luke-alloy Thank you for the follow-up. I agree with proposed option B.