The memory usage of FastDDS was gradually increasing under security mode

[Barry-Xu-2018]

Is there an already existing issue for this?

• I have searched the existing issues

Expected behavior

After running for a long time, FastDDS memory usage hasn't changed significantly.

Current behavior

After running for 12 hours, the memory usage of FastDDS increased by 11.8M.

Steps to reproduce

I modified example SecureHelloWorldExample.
Barry-Xu-2018@43e3d2d

The publisher will create 20 topics and 20 DataWriters. One of the DataWriters will send a message to the subscriber and then finish. The publisher program will be executed in a loop.

Under FastDDS v2.14.4, apply above change and build SecureHelloWorldExample

First terminal to run subscriber

# cd /Path/To/fastrtps/examples/cpp/dds/SecureHelloWorldExample/build
# ./DDSSecureHelloWorldExample subscriber

Second terminal to run publisher
I use a script to run publisher continuously

# cat loop.sh 
#!/bin/bash

while true
do
./DDSSecureHelloWorldExample publisher
done

# cd /Path/To/fastrtps/examples/cpp/dds/SecureHelloWorldExample/build
# ./loop.sh

Third terminal to record memory usage of ./DDSSecureHelloWorldExample subscriber
Use the following script to record memory usage every 5 seconds for a total of 12 hours.

# cat get_memory_usage.sh
#!/bin/bash

PID=`pgrep -f "./DDSSecureHelloWorldExample subscriber"`

for i in {1..8640}
do
  ps -p ${PID} -o rss --no-headers
  sleep 5 
done

Here is an output of memory usage on my test environment
test_12_hours_0725.log

Fast DDS version/commit

v2.14.4

Platform/Architecture

Other. Please specify in Additional context section.

Transport layer

Default configuration, UDPv4 & SHM

Additional context

Platform/Architecture: Ubuntu 22.04/amd64

I also tested version v2.13.1 using the above steps. This issue is more severe in the older version. Testing for just 10 minutes showed a memory usage increase of 17MB.

I didn't find any memory leak issues in FastDDS when using Valgrind.

XML configuration file

Relevant log output

Network traffic capture

No response

[Barry-Xu-2018]

This issue seems to be related to the discovery of endpoints in FastDDS. If external DDS endpoints keep changing, it can lead to an increase in memory usage.

[Barry-Xu-2018]

Last night, I tested for 12 hours, and the memory usage increased from 22,372 KB to 34,744 KB (about 12 MB).
The log of get_memory_usage.sh output is test_12_hours.log

[MiguelCompany]

@Barry-Xu-2018 Version 2.13.x reached end-of-life. Could you check whether this reproduces in any of the supported branches (2.14.x, 3.2.x) ?

[Barry-Xu-2018]

@MiguelCompany

Version 2.13.x reached end-of-life. Could you check whether this reproduces in any of the supported branches (2.14.x, 3.2.x) ?

Thank you for your reminder. I will check other branches.

I encountered this issue while using DDS Router v2.1.0 and ROS2 Humble.
The version of FastDDS used in Humble is 2.6.x, whereas DDS Router v2.1.0 is compiled with FastDDS version 2.13.1. There's already a difference in FastDDS versions.
If I choose a higher version of DDS Router, the corresponding FastDDS version will become 2.14.x or 3.x.x. If the FastDDS versions differ significantly, will there be compatibility issues?
Could you please provide your suggestions?

[Barry-Xu-2018]

@MiguelCompany

I used FastDDS v2.14.4 and followed the reproduction steps mentioned above for testing. After 12 hours of testing, I found that the memory usage increased from 22,400KB to 34,568KB (about 11.8MB). This test result is also close to the results from version 2.13.6 (Refer to #5900 (comment)).

I will update the description of issue to use FastDDS v2.14.4.

[fujitatomoya]

this issue looks like pretty much it does not deallocate the memory that is allocated via endpoint discovery.

I didn't find any memory leak issues in FastDDS when using Valgrind.

but it does not seem like a memory leak with valgrind.

the test executes the way more stress than usual usage, but it would be similar situation if the DDS router runs weeks without restarting. @Barry-Xu-2018 how about version v3.x.x latest? do we have the similar problem with it? if it does not have it, i guess we can close this issue with suggestion for updating the Fast-DDS version.

If I choose a higher version of DDS Router, the corresponding FastDDS version will become 2.14.x or 3.x.x. If the FastDDS versions differ significantly, will there be compatibility issues?

i think difference of minor version should be fine for API/ABI and inter-connect compatibility, but major version. major version is likely to be upgraded because of API/ABI incompatibility including interfaces.

[Barry-Xu-2018]

@fujitatomoya

@Barry-Xu-2018 how about version v3.x.x latest? do we have the similar problem with it? if it does not have it, i guess we can close this issue with suggestion for updating the Fast-DDS version.

I haven't tested the v3.x.x version.
DDS-Router can be upgraded to use FastDDS v3.x.x, but the forwarded messages come from Humble (FastDDS version is v2.6.x). There is a significant version difference here., and I'm concerned about potential issues with using it this way.

[fujitatomoya]

what i mean is that in general Fast-DDS v3.x.x still has the this issue or not. that is probably not for our specific use case with ROS 2 Humble but as open source perspective, it would be really nice to check?

DDS-Router can be upgraded to use FastDDS v3.x.x, but the forwarded messages come from Humble (FastDDS version is v2.6.x). There is a significant version difference here., and I'm concerned about potential issues with using it this way.

as we already know, we will not use this combination.

[Barry-Xu-2018]

what i mean is that in general Fast-DDS v3.x.x still has the this issue or not. that is probably not for our specific use case with ROS 2 Humble but as open source perspective, it would be really nice to check?

Yes. I will check it.

as we already know, we will not use this combination.

Yes.

[Barry-Xu-2018]

@fujitatomoya @MiguelCompany

I tested the latest version of Fast-DDS (v3.2.2).
This issue can be reproduced on the latest FastDDS.

Below are the test results from last night, which ran for 12 hours. It was found that the memory usage of the subscriber program increased from 23,808 KB to 47,876 KB (an increase of 23 MB).

v3.2.2_test_12_hours_0704.log

The modifications to the example program corresponding to v3.2.2 are as follows:
Barry-Xu-2018@e842e03

[fujitatomoya]

@Barry-Xu-2018 thanks for testing!

@MiguelCompany
with the result of latest Fast-DDS and valgrind, probably is this intentional behavior to keep the endpoint history data or something else as long as the participant is alive? it is unlikely this test situation (recreating endpoints each seconds) happens in the real use case though.

[MiguelCompany]

@Barry-Xu-2018 @fujitatomoya Can we close this now that #5935 has been merged?

[Barry-Xu-2018]

@MiguelCompany

Thank you. I close this issue.