I decided to implement a “Production-Ready” Balancer.
The assignment suggests various simpler approaches (e.g., a queue serving one client at a time, round-robin, etc.).
However, the production-ready version requires dynamic parallel processing and fair distribution of capacity among all registered clients—hence it’s the most challenging. My approach dynamically recalculates capacity whenever the system’s state changes (e.g., when a request completes, or a new client registers).
I started by analyzing it on paper, as I’m used to. It helps me think it through better, and it’s easy to make adjustments.
Here is an image from my paper analysis:
Then I moved on to the actual implementation. I’m used to using Copilot or an LLM chat for my work, but for this task I completely skipped chat and only used Copilot as usual.
The implementation itself took about four hours—honestly, it was more challenging than I initially expected, but I eventually arrived at a working solution.
For logging purposes, I added an ID field to the Client. After successfully implementing everything, I consulted the code with GPT for possible improvements; in the end, I ignored most of the suggestions except for using sync.Cond instead of time.Sleep. It was my first time using this feature from the sync package, and it made a lot of sense. Finally, I used Copilot to construct the tests for the balancer.
Imagine a standard client-server relationship, only in our case, the server is very fragile (and very expensive). We can't let an arbitrary number of clients smashing the server with arbitrary number of requests, but as it's so expensive to run, we'd like to utilize it as much as possible. And that's why we need a balancer.
A balancer is a gateway between the clients and the server. A client registers themselves in balancer and the balancer is responsible for distributing the service capacity between registered clients.
The balancer must ensure that at any given time, the number of parallel requests sent to the server does not exceed the provided limit.
Based on your skill, you can choose to implement a very simple balancer or production-ready balancer.
The implementation of a simple balancer can, for example, be one of:
- A balancer that serves one client at a time, enqueuing others.
- A balancer that serves registered clients in a round-robin fashion.
- A balancer that is processing batches of registered clients and distributing the capacity among them, while enqueuing incoming clients and once one batch is done, process the next batch.
The ultimate, production balancer would run the requests from multiple clients in parallel according to their weights and process the work fairly from all the clients. A client with twice the weight of another client would be allowed to run twice more parallel requests at any given time.
In any case, the balancer must ensure that the number of requests in process at any given time equals the provided limit of the server, aka we never can over-utilise the server, but we also don't want to under-utilise it.