Currently, the memory management in Sidecar relies on a provided chunk size for memory allocation, and it uses a very simple algorithm that searches for free memory from the head. When the communication sizes are not multiples of the chunk size, there will be significant internal fragmentation if the chunks are too large. However, using a smaller chunk size can hurt allocate performance. This problem is highlighted by Gemma3, which has projectors with different hidden sizes. There hasn't been a microbenchmark on the performance of the memory manager, but we will need a more generic memory manager with better performance (latency) and less fragmentation.
https://github.com/cornserve-ai/cornserve/blob/master/python/cornserve/services/sidecar/shm_manager.py#L105
Currently, the memory management in Sidecar relies on a provided chunk size for memory allocation, and it uses a very simple algorithm that searches for free memory from the head. When the communication sizes are not multiples of the chunk size, there will be significant internal fragmentation if the chunks are too large. However, using a smaller chunk size can hurt
allocateperformance. This problem is highlighted by Gemma3, which has projectors with different hidden sizes. There hasn't been a microbenchmark on the performance of the memory manager, but we will need a more generic memory manager with better performance (latency) and less fragmentation.https://github.com/cornserve-ai/cornserve/blob/master/python/cornserve/services/sidecar/shm_manager.py#L105