eNATS and 2+ replicas

[nilcolor]

I'm still trying to find a way to deploy 2+ anycables with eNATS using this chart. I.e. I deployed anycable (replica: 1) and it works just fine. Client can connect, receive.send messages. All is good. Note: I'm using HTTP broadcast adapter. Idea is to avoid any extra services like redistribution, etc. The amount of message is pretty low. So, setup work just fine with single replica. But if I will set somethign like replica: 2 - it stop working. Anycable logs still show that client(s) can connect and they are connected to different nodes. But sending a message from Rails is no longer works. I expect it is required to setup a eNATS cluster. But docs tells "start first server like this, start others like that". This is not possible with this chart, right?
Supercluster... but that require 2+ cluster to be created first.
What did I miss? Is the only way to have 2+ replicas is to have Redis?

[palkan]

Hey!

I can't help with k8s configs, but in general it should work without needing to worry about the order or whatever. We do that for Fly.io deployments atomically (see the video and source code).

It's gonna be something like:

ANYCABLE_EMBED_NATS=true
ANYCABLE_ENATS_ADDR=nats://0.0.0.0:4242
ANYCABLE_ENATS_CLUSTER=nats://0.0.0.0:4342
ANYCABLE_ENATS_CLUSTER_ROUTES=nats://<anycable-go-service>:5222

[wsmolkowski]

@nilcolor have you any progress with this issue?

[nilcolor]

Hey @wsmolkowski
yeah, I took this chart and made a few changes like added NATS ports like

      containers:
        - name: anycable-go
          ports:
            {{- if .anycableNatsClientPort }}
            - name: nats-client
              containerPort: {{ .anycableNatsClientPort }}
              protocol: TCP
            {{- end }}
            {{- if .anycableNatsClusterPort }}
            - name: nats-cluster
              containerPort: {{ .anycableNatsClusterPort }}
              protocol: TCP
            {{- end }}

So this way NATS ports are exposed and RoR app can connect. Rails app uses nats backend. So, basically this is what I have now in Anycable values YAML:

  anycableEmbedNats: "true" # ANYCABLE_EMBED_NATS
  anycableNatsClientPort: 4222
  anycableNatsClusterPort: 5222
  anycableEnatsAddr: "nats://0.0.0.0:4222" # ANYCABLE_ENATS_ADDR
  anycableEnatsCluster: "nats://0.0.0.0:5222" # ANYCABLE_ENATS_CLUSTER
  anycableEnatsClusterName: "anycable-cluster" # ANYCABLE_ENATS_CLUSTER_NAME
  anycableEnatsClusterRoutes: "nats://app-anycable-anycable-go:5222" # ANYCABLE_ENATS_CLUSTER_ROUTES

and this is RoR (anycable.yml)

  ANYCABLE_BROADCAST_ADAPTER: "nats"
  ANYCABLE_NATS_SERVERS: "nats://app-anycable-anycable-go:4222"
  ANYCABLE_HTTP_BROADCAST_URL: "http://app-anycable-anycable-go:8080/_broadcast"
  ANYCABLE_WEBSOCKET_URL: "wss://app-anycable.stg-01.dev/cable"

and a few keys for grpc and jwt things. This might not be ideal and may require more work but at least it works now with replicas set to 2+. I think an issue was with NATS ports not visible between Anycable pods and from a client...