.Net example Code fails due to stream client bug

[adammckay]

Describe the bug

I have a docker container running on a linux host.
My .Net code is .Net 8.0 running from Windows 11.
Linux, Windows, and containers are all running IPv4 only. NO IPv6 (but even if I turn it on, nothing changes).

When running the sample .net 8.0 code, I am unable to get past the point of creating a producer.
I am speaking of this code here:
https://github.com/rabbitmq/rabbitmq-tutorials/blob/main/dotnet-stream/Send/Send.cs

When I run that code, I am getting the following error:
"No connection could be made because the target machine actively refused it."

Exception has occurred: CLR/System.Net.Sockets.SocketException
An exception of type 'System.Net.Sockets.SocketException' occurred in System.Private.CoreLib.dll but was not handled in user code: 'No connection could be made because the target machine actively refused it.'
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location ---
   at RabbitMQ.Stream.Client.Connection.<Create>d__26.MoveNext()
   at RabbitMQ.Stream.Client.Client.<Create>d__39.MoveNext()
   at RabbitMQ.Stream.Client.Routing.<CreateClient>d__4.MoveNext()
   at RabbitMQ.Stream.Client.RoutingHelper`1.<LookupConnection>d__0.MoveNext()
   at RabbitMQ.Stream.Client.RoutingHelper`1.<>c__DisplayClass4_0.<<LookupLeaderConnection>b__0>d.MoveNext()
   at RabbitMQ.Stream.Client.ConnectionsPool.<GetOrCreateClient>d__16.MoveNext()
   at RabbitMQ.Stream.Client.RoutingHelper`1.<LookupLeaderConnection>d__4.MoveNext()
   at RabbitMQ.Stream.Client.RawProducer.<Create>d__7.MoveNext()
   at RabbitMQ.Stream.Client.StreamSystem.<CreateRawProducer>d__23.MoveNext()
   at RabbitMQ.Stream.Client.Reliable.ProducerFactory.<StandardProducer>d__5.MoveNext()
   at RabbitMQ.Stream.Client.Reliable.ProducerFactory.<CreateProducer>d__3.MoveNext()
   at RabbitMQ.Stream.Client.Reliable.Producer.<CreateNewEntity>d__7.MoveNext()
   at RabbitMQ.Stream.Client.Reliable.ReliableBase.<MaybeInit>d__14.MoveNext()
   at RabbitMQ.Stream.Client.Reliable.ReliableBase.<Init>d__15.MoveNext()
   at RabbitMQ.Stream.Client.Reliable.ReliableBase.<Init>d__13.MoveNext()
   at RabbitMQ.Stream.Client.Reliable.Producer.<Create>d__6.MoveNext()
   at Program.<<Main>$>d__0.MoveNext() in x:\Location\Send.cs:line 27

First, here is my docker compose:

---
services:

  rabbitmq:
#I have tried the ubuntu image and i have tried earlier versions.  no dice.
    image: rabbitmq:4.1-management-alpine # Specifies RabbitMQ with management plugin
    container_name: rabbitmq # Assigns a name to the container
    environment:
      RABBITMQ_DEFAULT_USER: user # Sets the default username
      RABBITMQ_DEFAULT_PASS: pass # Sets the default password
      #It doesn't matter if this is enabled or not
      RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbitmq_stream advertised_host localhost advertised_port 5552"
    ports:
      - "5552:5552" # Streams
      - "5672:5672" # AMQP port for client connections
      - "15672:15672" # Management UI port
    volumes:
      - /my/storage/RabbitMQ:/var/lib/rabbitmq # Persistent storage for RabbitMQ data
    restart: unless-stopped

#doesn't matter whether I do this or not.
    network_mode: host
#ipv6 is off on my host.  if it is not, I use these lines to turn it off in my container
#    sysctls:
#      net.ipv6.conf.all.disable_ipv6: 1

After my container spins up, I am enabling the streams plugin which gives me a message that it's successful:
sudo docker exec rabbitmq rabbitmq-plugins enable rabbitmq_stream

So far so good...

Just to be sure, I am gonna test those ports from Windows 11 Powershell to the Linux box:

PS C:\Users\me> Test-NetConnection -computername mycomputername -port 5552
ComputerName     : mycomputername
RemoteAddress    : 192.168.1.100
RemotePort       : 5552
InterfaceAlias   : Wi-Fi
SourceAddress    : 192.168.1.100
TcpTestSucceeded : True

PS C:\Users\me> Test-NetConnection -computername mycomputername -port 5672
ComputerName     : mycomputername
RemoteAddress    : 192.168.1.100
RemotePort       : 5672
InterfaceAlias   : Wi-Fi
SourceAddress    : 192.168.1.100
TcpTestSucceeded : True

PS C:\Users\me> Test-NetConnection -computername mycomputername -port 15672
ComputerName     : mycomputername
RemoteAddress    : 192.168.1.100
RemotePort       : 15672
InterfaceAlias   : Wi-Fi
SourceAddress    : 192.168.1.100
TcpTestSucceeded : True

Now I am verifying that everything works. Here is the management page for queues:

Yep. We're good.

Now I'm running the example code:

using System.Text;
using System.Net;
using RabbitMQ.Stream.Client;
using RabbitMQ.Stream.Client.Reliable;

//I had to add a username / password / endpoint.  This is annoying because all documentation tells you to
//create the server with a user/pass but the example gives you no direction on how to set these up.
var streamSystem = await StreamSystem.Create(new StreamSystemConfig
{
    UserName = "user",
    Password = "pass",
    Endpoints = new List<EndPoint>
    {
        new DnsEndPoint("192.168.1.100", 5552)
    }
});

await streamSystem.CreateStream(new StreamSpec("hello-stream")
{
    MaxLengthBytes = 5_000_000_000
});
// At this point, you can see from my screenshot that I have created a stream successfully.
// There is no doubt in my mind that I am properly connected to the server via port 5552.


// I have made it this far but the next line gives the following error:
var producer = await Producer.Create(new ProducerConfig(streamSystem, "hello-stream"));

await producer.Send(new Message(Encoding.UTF8.GetBytes($"Hello, World")));

Console.WriteLine(" [x] Sent 'Hello, World'");
Console.WriteLine(" [x] Press any key to exit");
Console.ReadKey();
await producer.Close();
await streamSystem.Close();

So even though we get the error, a stream is created on the server. I just couldn't create the Producer and send the message.

My problem is very similar to this one but I have made sure that all machines and containers only have IPV4 enabled
#374

Reproduction steps

1. Spin up docker container
2. Enable streams plugin
3. Run Example code
4. Get error
   ...

Expected behavior

No error in example code, Stream and Message should be created.

Additional context

I realize that the example is centered around connecting to a RabbitMQ instance on the same machine. But I need this to run on a DEV server and it's a reasonable expectation that RabbitMQ does not run on your laptop in PROD.

[lukebakken]

@Gsantomaggio I don't have the permission here to convert this to a discussion.

Related discussion in rabbitmq/rabbitmq-tutorials: rabbitmq/rabbitmq-tutorials#655

@adammckay - how are you running RabbitMQ's docker container? WSL? I ask because, on my Windows 11 laptop, when I run RabbitMQ via docker in WSL2, it only listens on localhost:

> netstat -an|rg 5552
  TCP    127.0.0.1:5552         0.0.0.0:0              LISTENING
  TCP    [::1]:5552             [::]:0                 LISTENING

I realize your Test-NetConnection output shows differently, which is why I'm asking how you are running RabbitMQ.

If I start RabbitMQ on Windows, via the PS functions I shared here, I can connect using your code.

C:\Users\lrbakken\development\lukebakken\rabbitmq-stream-dotnet-client-434
> Test-NetConnection -ComputerName $env:COMPUTERNAME -Port 5552

ComputerName     : SEA-3LG5HVJUWJK
RemoteAddress    : fe80::8ac1:d6d4:4884:c81c%26
RemotePort       : 5552
InterfaceAlias   : Wi-Fi
SourceAddress    : fe80::8ac1:d6d4:4884:c81c%26
TcpTestSucceeded : True

C:\Users\lrbakken\development\lukebakken\rabbitmq-stream-dotnet-client-434
> dotnet run
 [x] Sent 'Hello, World'
 [x] Press any key to exit

Finally, I would try the following. Run this command:

Resolve-DnsName -Name $env:COMPUTERNAME

It should output 192.168.1.100. Then, take the value of $env:COMPUTERNAME and use it in RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS:

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbitmq_stream advertised_host COMPUTERNAME advertised_port 5552"

Finally, you could also change your code to use the hostname instead of an IP. Let us know if that resolves your issue.

[Gsantomaggio]

@lukebakken thank you for looking the issue.

[adammckay]

First, thank you for your response. To answer your question, I am running the container on a separate physical Arch Linux box.

Whether I run the container using host network or it's own network makes no difference. From within the container, I am able to ping out to the internet, see things on the host network, etc.

I did go down the same avenue of thought that you mentioned. I tried the container with some different advertized_host values:

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbitmq_stream advertised_host <myArchServerHostname> advertised_port 5552"
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbitmq_stream advertised_host <external DNS pointing to host> advertised_port 5552"
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbitmq_stream advertised_host <myArchServerIP 192.168.1.100> advertised_port 5552"

These both actually caused the container to run in a bad state. I will dig up the errors from that later today.

What makes me wonder about all of this is why does the example code successfully create a stream but then it can't create the producer. Clearly, the IPs and Ports are fine up to the point of creating the stream or you wouldn't see it in the screenshot. Why then are the host ports problematic at the point of creating the Producer?

[Gsantomaggio]

What makes me wonder about all of this is why does the example code successfully create a stream

Because the client uses a dedicated connection to create a stream (called locator connection you should see it in the UI)

The producer must be connected to the leader node and must look up the leader using the hostname.
So the client need to resolve the hostname directly or using advertised_host

There is also the loadbalancer configuration

[adammckay]

OK. Here's what happens when I change the advertised_host in my docker compose:

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbitmq_stream advertised_host 192.168.1.100 advertised_port 5552"

I completely rebuilt the container and made sure no remnants from the prior volume.

The container starts fine - enabling the stream plugin:
sudo docker exec rabbitmq rabbitmq-plugins enable rabbitmq_stream

Gets me this error:

Enabling plugins on node rabbit@mylinuxserversname:
rabbitmq_stream
The following plugins have been configured:
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_prometheus
  rabbitmq_stream
  rabbitmq_web_dispatch
Applying plugin configuration to rabbit@mylinuxserversname...
Stack trace:

** (CaseClauseError) no case clause matching: {:error, {:bad_environment_value, ~c"192.168.1.100"}}
    (rabbitmqctl 4.1.4) lib/rabbitmq/cli/plugins/plugins_helpers.ex:112: RabbitMQ.CLI.Plugins.Helpers.update_enabled_plugins/4
    (rabbitmqctl 4.1.4) lib/rabbitmq/cli/plugins/commands/enable_command.ex:127: anonymous fn/6 in RabbitMQ.CLI.Plugins.Commands.EnableCommand.do_run/2
    (elixir 1.17.3) lib/stream.ex:1593: anonymous fn/2 in Stream.iterate/2
    (elixir 1.17.3) lib/stream.ex:1820: Stream.do_unfold/4
    (elixir 1.17.3) lib/stream.ex:1891: Enumerable.Stream.do_each/4
    (elixir 1.17.3) lib/stream.ex:1052: Stream.do_transform_inner_enum/7
    (elixir 1.17.3) lib/stream.ex:1891: Enumerable.Stream.do_each/4
    (elixir 1.17.3) lib/enum.ex:2585: Enum.reduce_while/3

{:case_clause, {:error, {:bad_environment_value, ~c"192.168.1.100"}}}

And of course my code faults because no enpoints can be found.

I also tried just using the hostname.

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbitmq_stream advertised_host mylinuxserversname advertised_port 5552"

Same error. But yet in the web interface, it names the node rabbit@mylinuxserversname either way.

Enabling plugins on node rabbit@mylinuxserversname:
rabbitmq_stream
The following plugins have been configured:
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_prometheus
  rabbitmq_stream
  rabbitmq_web_dispatch
Applying plugin configuration to rabbit@mylinuxserversname...
Stack trace:

** (CaseClauseError) no case clause matching: {:error, {:bad_environment_value, ~c"mylinuxserversname"}}
    (rabbitmqctl 4.1.4) lib/rabbitmq/cli/plugins/plugins_helpers.ex:112: RabbitMQ.CLI.Plugins.Helpers.update_enabled_plugins/4
    (rabbitmqctl 4.1.4) lib/rabbitmq/cli/plugins/commands/enable_command.ex:127: anonymous fn/6 in RabbitMQ.CLI.Plugins.Commands.EnableCommand.do_run/2
    (elixir 1.17.3) lib/stream.ex:1593: anonymous fn/2 in Stream.iterate/2
    (elixir 1.17.3) lib/stream.ex:1820: Stream.do_unfold/4
    (elixir 1.17.3) lib/stream.ex:1891: Enumerable.Stream.do_each/4
    (elixir 1.17.3) lib/stream.ex:1052: Stream.do_transform_inner_enum/7
    (elixir 1.17.3) lib/stream.ex:1891: Enumerable.Stream.do_each/4
    (elixir 1.17.3) lib/enum.ex:2585: Enum.reduce_while/3

{:case_clause, {:error, {:bad_environment_value, ~c"mylinuxserversname"}}}

So now I set everything back. Earlier, you had mentioned a 'locator connection'? I am hoping you mean the connection tab in the web ui.

This is what I see:

(192.168.1.101 is the IP of the Windows machine where I am running the .NET example. 192.168.1.100 is the IP of my Linux Server that hosts docker)

[adammckay]

You got me in the right direction. For anyone else running into the same thing, the 'advertised_host' must be set with quotes around the ip address in docker compose.

So this is what you do when you are doing everything on your local machine:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbitmq_stream advertised_host localhost advertised_port 5552"

If you want to specify an IP, DO NOT do this:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbitmq_stream advertised_host 192.168.1.100 advertised_port 5552"

If you want to specify an IP, you must do this:
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbitmq_stream advertised_host '192.168.1.100' advertised_port 5552"

In my defense, I didn't find any resources to really tell me that.

Anyways, thanks a lot #Gsantomaggio!

[Gsantomaggio]

@adammckay, great :) ! Thanks for sharing how you solved it.

[lukebakken]

Thanks @adammckay. I didn't get a chance to look at this discussion yesterday, but as soon as I caught up this morning and saw this

(CaseClauseError) no case clause matching: {:error, {:bad_environment_value, ~c"mylinuxserversname"}}

I knew it was a quoting issue. Erlang is "fun" when it comes to arguments passed to the VM via - that are to be interpreted as IP addresses or host names.

I'll see if I can reproduce this and will update the docs.