LLCoprocedurePool uses a great deal of memory

[monty-linden]

[ Moved from https://github.com/secondlife/viewer-private/issues/320 ]

Part of the viewer avatar experience improvement effort going on in 4Q24. Diving into causes of slow/delayed avatar appearance as described here: https://docs.google.com/document/d/1FMLp1B3JOnQYm5Jvz52YweggMF6hJQrwmAYQuHwWTis/edit?tab=t.0

Here I was familiarizing myself with the coro adoption that occured 10 years ago. I stumbled across a multi-thread request buffer in the form of LLCoprocedurePool. We have four of these which present during startup as follows:

2024-11-15T21:33:24Z INFO #CoProcMgr# llmessage/llcoproceduremanager.cpp(355) LLCoprocedurePool::LLCoprocedurePool : Created coprocedure pool named "Upload" with 1 items, queue max 1048576

The four classes of things handled by these pools are: Upload, AIS, AssetStorage, ExpCache. The request queue is based on boost::fiber::buffered_channel. The typical implementation of this is as a ring buffer operating on a single allocation of maximum size, here 1048576 elements in all four cases. This amounts to something like 64-256MB of memory that is typically unused. When it is used, it slowly crawls around the ring buffer dirtying pages and requiring residency.

There are other reasons (to show up in other GHIs) not to use buffered_channel and LLCoprocedureManager as a work queue and as an interface between multi-threads and multi-coros. But while we are using them, tuning this down to a correct sizing for each class would recover a good chunk of memory.

[akleshchev]

here 1048576 elements in all four cases.

Upload and experience cache now get 2048.

There are other reasons (to show up in other GHIs) not to use buffered_channel and LLCoprocedureManager

My personal experience with viewer's implementation of coroutines is mostly negative and very very crashy. They are easy to make but hard to maintain. At least for http handling I would prefer if we returned to handlers.