Redis output balancing

[wiibaa]

From LOGSTASH-2260

The Redis output currently uses a failover model to switch hosts. I'd like to see a balance option added that defaults to the current failover setup but also allows round robin or least connection balancing.
I tried to implement this myself earlier but there are a million ways to handle a connection pool in JRuby and I don't know enough about threading to decide on a single maintainable method.
Please consider this both a feature request and guidance request as I'm happy to work on this myself if some direction on load balancing could be provided.

Is it related to congestion management in #4 or is it something else?

[hatt]

This was originally an issue I submitted to Jira, and is unrelated to #4. When an array of hosts is passed to Redis output, it uses them as a failover list and only moves to the next entry when a previous one fails, never going back. The issue was to try and round robin or otherwise intelligently balance between members of the list like the input plugin does.

[ppf2]

+1

[blavoie]

Also interested that specified Redis hosts list is used for round robin load balancing rather than failover.

We tried something like HAProxy, but once the connection is opened the channel is kept and no balancing is done.

As a workaround, we added to events a random number field. Then in the output, using modulo on this field we can balance to different Redis servers. Each Redis is shown as a virtual IP, that in case of one server failure the others can pick up the queuing. We don't have tested performance impact of this yet.

[jordansissel]

+1 on having the redis output be able to do this. Can you describe your use
case a bit more? What is reading from redis on the other side? Should it be
round-robin or load/congestion oriented?

On Wednesday, January 27, 2016, Bruno Lavoie notifications@github.com
wrote:

Also interested that specified Redis hosts list is used for round robin
load balancing rather than failover.

We tried something like HAProxy, but once the connection is opened the
channel is kept and no balancing is done.

As a workaround, we added to events a random number field. Then in the
output, using modulo on this field we can balance to different Redis
servers. Each Redis is shown as a virtual IP, that in case of one server
failure the others can pick up the queuing. We don't have tested
performance impact of this yet.

—
Reply to this email directly or view it on GitHub
#7 (comment)
.

[blavoie]

Hi Jordan,

To be very clear about the way our workaround works, here is the output part to our Redis VIPs. This is the output on our front logstash nodes receiving all events from clients (beats, tcp, syslog, ...).

output { # generated random field between 0 and 100 if [random_field_name] <= 50 { redis { host => ["redis-vip-1.domain.com"] ... } } else { redis { host => ["redis-vip-2.domain.com"] ... } } }

Then on the other side, the processing cluster that dequeue the lists must specify 1 input for each virtual IP addresses:
input { redis { host => [ "redis-vip-1.domain.com" ] ... } redis { host => [ "redis-vip-2.domain.com" ] ... } }

This simply give us some resiliency and bigger buffer at peak times or more time making processing cluster updates. It's not as rock solid as a true round robin balancing done directly in the plugin, retrying on another node in case of failure. When your random number drives you to a specific Redis, you're stuck with. The VIP/HAProxy stuff may help to alleviate this drawback.

Hope that it asked precisely to your questions.

[jordansissel]

@blavoie thanks for the details! I think I understand a bit more now. :)

[mrjameshamilton]

+1

[screwyy]

+100

[vijaypatnana]

+100