Buffer errors

[troepolik]

Describe the bug

We faced with very rare errors in consuming :

System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'length')
   at System.ThrowHelper.ThrowStartOrEndArgumentValidationException(Int64 start)
   at System.Buffers.ReadOnlySequence`1.Slice(Int64 start, Int64 length)
   at RabbitMQ.Stream.Client.RawConsumer.<>c__DisplayClass19_0.<ParseChunk>g__MessageFromSequence|0(ReadOnlySequence`1& unCompressedData, Int32& compressOffset) in /_/RabbitMQ.Stream.Client/RawConsumer.cs:line 269

Error | | Error while parsing message Consumer id 0 for stream: xxx, identifier: , reference: xxx, OffsetSpec RabbitMQ.Stream.Client.OffsetTypeOffset Client ProvidedName xxx, No SuperStream, IsSingleActiveConsumer: False, Token IsCancellationRequested: False . The message will be skipped. Please report this issue to the RabbitMQ team on GitHub https://github.com/rabbitmq/rabbitmq-stream-dotnet-client/issues

Issue looks like corrupted internal buffer or mb buffer that used by few threads in the same time.
I dont know if it related or not - we have few parallel consumers, each consumes own queue.

Some times we also have some different type of error, but it also look like corrupted or used by few threads buffer:

System.Text.Json.JsonException: 'r' is an invalid end of a number. Expected a delimiter. Path: $ | LineNumber: 0 | BytePositionInLine: 7850.

We control message body, so I can guarantee that we publish proper json.
We deserialize messages like that:

Utf8JsonReader reader = new(message.Data.Contents);
var msg = JsonSerializer.Deserialize<TMessage>(ref reader);

client version : 1.8.14 - latest
server: RabbitMQ 3.12.0, Erlang 26.2.5

Reproduction steps

1. Error happens very rare - once per 2 days , so I can not reproduce it locally

Expected behavior

I expect consuming without errors

Additional context

No response

[Gsantomaggio]

@troepolik thank you for reporting the issue. Do you have the same error if you re-consume the same messages?

I dont know if it related or not - we have few parallel consumers, each consumes own queue.

What do you mean? Can you explain better? ty

used by few threads buffer

What do you mean here?
By the library, or by your code?

[Gsantomaggio]

Can you enable the Crc32? here an example:

rabbitmq-stream-dotnet-client/Tests/Crc32.cs

Line 9 in b5d784f

public class Crc32 : ICrc32

[troepolik]

Do you have the same error if you re-consume the same messages?

@Gsantomaggio As I understand, error happens inside library and message was skiped, so we dont have possibility to make retry.
"The message will be skipped."
But seccond error about json deserialization - I'l try to make retry there.

I dont know if it related or not - we have few parallel consumers, each consumes own queue.
What do you mean? Can you explain better?

We have few queues and they are consumed by few consumers in parallel in one application.

used by few threads buffer
What do you mean here? By the library, or by your code?

By library. error stack trace is inside library

System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'length')
   at System.ThrowHelper.ThrowStartOrEndArgumentValidationException(Int64 start)
   at System.Buffers.ReadOnlySequence`1.Slice(Int64 start, Int64 length)
   at RabbitMQ.Stream.Client.RawConsumer.<>c__DisplayClass19_0.<ParseChunk>g__MessageFromSequence|0(ReadOnlySequence`1& unCompressedData, Int32& compressOffset) in /_/RabbitMQ.Stream.Client/RawConsumer.cs:line 269

[troepolik]

Can you enable the Crc32? here an example:

@Gsantomaggio thank you! I'l try

[Gsantomaggio]

@troepolik, I am unable to reproduce the issue.
Please

• Let me know if the CRC fails
• Provide a code on how you use the library
• Check the server logs if there are errors

[Gsantomaggio]

Looking at the issue:

System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'length')

The buffer seems to be corrupted here.

Let's see if the CRC passes before going ahead with the tests

[troepolik]

@Gsantomaggio yes, after couple days we've spotted two crc errors, both about the same time:

2025-06-19 02:33:15.530
Error | | CRC32 does not match, server crc: 4116709835, local crc: 3226410350, Consumer id 0 for stream: xxx, identifier: xxx, reference: xxx, OffsetSpec RabbitMQ.Stream.Client.OffsetTypeOffset Client ProvidedName xxx, No SuperStream, IsSingleActiveConsumer: False, Token IsCancellationRequested: False , Chunk Consumed 75

on rmq server side I see only that warning:
2025-06-19 02:33:15.541742+00:00 [warning] <0.136764009.1> Socket #Port<0.441349> closed [<0.136764009.1>]

this warning was in both of crc error cases.

[Gsantomaggio]

Repeated and real CRC issues are likely to be a symptom of a more significant underlying problem.
It is essential to know if the chunk is always corrupted in this case: Chunk Consumed 75.

Can you please create a simple consumer?, like:

 var consumerConfiguration = new ConsumerConfig(streamSystem, "<<your_stream>>")
        {
            OffsetSpec = new OffsetTypeFirst(),
            MessageHandler = async (stream, consumerSource, context, message) =>
            {
                Console.WriteLine($"offset: {context.Offset}, ChunkId:{context.ChunkId}");  

Could you see if it is systematic?

Run it more than once.

[troepolik]

@Gsantomaggio it's not systematic. we have 50rps in this queue and only two errors for 2 days.
one with Chunk Consumed 1
another with Chunk Consumed 75

[acogoluegnes]

Can you confirm that if you read and re-read the exact same stream, with the same program, the CRC error happens just from time to time and you are able to read the messages correctly most of the time?

[troepolik]

btw what driver do when crc error happens? will this message be just skipped?
or it depends on few factors? because as we see Socket was closed after error, so as I understand it was reconnection.

[Gsantomaggio]

The TCP connection is closed if you enable the CRC and it doesn't match.
This behaviour can be changed, of course, but we should understand the root problem: Why the CRC fails.

If you want to skip corrupted chunks, you can disable the CRC check.

Is your client remote or in a local network?

[troepolik]

@Gsantomaggio @acogoluegnes yes, we've enabled crc.
"The TCP connection is closed if you enable the CRC and it doesn't match."
And I just want to know what happens next? message will be reprocessed?
If yes, then answer to question:
"Can you confirm that if you read and re-read the exact same stream, with the same program, the CRC error happens just from time to time and you are able to read the messages correctly most of the time?"

will be "yes, error happens just from time to time, and it was successfully readed next time"

[acogoluegnes]

Can you share a simple application that you use to read and re-read and that reproduces the problem?

[troepolik]

@acogoluegnes

Can you confirm that if you read and re-read the exact same stream, with the same program, the CRC error happens just from time to time and you are able to read the messages correctly most of the time?

I've just connected to stream with the same program with timestamp before error and processed all messages without errors.
So, yes, I can confirm.

I will try to prepare simplified program example.

[Gsantomaggio]

@troepolik, per conversation with the team, we suggest updating to RabbitMQ 4.1. You are using an outdated RabbitMQ version that is no longer supported. There was a bug fixed here: rabbitmq/osiris@0888a21 that could cause your error.

Please let us know if the issue occurs again after upgrading to 4.1. Thank you.

[troepolik]

@Gsantomaggio @acogoluegnes thank you a lot! for your responses and your time!
I'll be back after version updating, but it can take a week at least to prepare and test.

anyway, @acogoluegnes here you can find the example if it still matter.
https://github.com/troepolik/StreamConsumeExample/tree/main/StreamConsumeExample

[Gsantomaggio]

https://github.com/troepolik/StreamConsumeExample/tree/main/StreamConsumeExample

I can't reproduce the issue btw

[Gsantomaggio]

@troepolik this PR #423 adds Crc32 functionalities.
Any feedback is welcome

[Gsantomaggio]

@troepolik do you have news about this issue? We released the new client version with more control for the CRC32

[Gsantomaggio]

@troepolik I will close the issue.

Let us know how it goes.

[troepolik]

@Gsantomaggio good day. Yesterday we've updated server to last rmq version.
RabbitMQ 4.1.4 Erlang 27.3.4.3
But error still with us...

CRC32 does not match, server crc: 1395186296, local crc: 579286124, Consumer id 0 for stream: xxx, identifier: xxx, reference: xxx, OffsetSpec RabbitMQ.Stream.Client.OffsetTypeOffset Client ProvidedName xxx, No SuperStream, IsSingleActiveConsumer: False, Token IsCancellationRequested: False , Chunk Consumed 1

at RabbitMQ.Stream.Client.RawConsumer.<>c__DisplayClass21_0.<<Init>b__0>d.MoveNext() in /_/RabbitMQ.Stream.Client/RawConsumer.cs:line 609
--- End of stack trace from previous location ---
   at RabbitMQ.Stream.Client.Client.HandleIncoming(Memory`1 frameMemory) in /_/RabbitMQ.Stream.Client/Client.cs:line 548
   at RabbitMQ.Stream.Client.Connection.ProcessIncomingFrames() in /_/RabbitMQ.Stream.Client/Connection.cs:line 188

driver version 1.8.14

[Gsantomaggio]

A few questions:

• Does it always happen to the same service/machine?
• How many clients do you have running? (Does the error happen in all the machines/instances/pods?)
• Are the clients remote?Outside the local area network?
• Does it always happen in the same stream(s)?
• Given the client with error, does it happen with all the stream(s) in that client?

[Gsantomaggio]

@troepolik may I ask to try the new client version? https://github.com/rabbitmq/rabbitmq-stream-dotnet-client/releases/tag/v1.9.1 the CRC is enabled by default, and try again? Thank you

[troepolik]

Sure. I've upgraded. But it will take some time to check because error is very rare.

[troepolik]

@Gsantomaggio tonight there was only one error, it is not CRC check, but I guess it throws before crc check:

Error while parsing message Consumer id 0 for stream: xxx, identifier: xxx, reference: xxx, OffsetSpec RabbitMQ.Stream.Client.OffsetTypeOffset Client ProvidedName xxx, No SuperStream, IsSingleActiveConsumer: False, Token IsCancellationRequested: False . The message will be skipped. Please report this issue to the RabbitMQ team on GitHub https://github.com/rabbitmq/rabbitmq-stream-dotnet-client/issues

error
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'length')
at System.ThrowHelper.ThrowStartOrEndArgumentValidationException(Int64 start)
at System.Buffers.ReadOnlySequence1.Slice(Int64 start, Int64 length) at RabbitMQ.Stream.Client.RawConsumer.<>c__DisplayClass18_0.<ParseChunk>g__MessageFromSequence|0(ReadOnlySequence1& unCompressedData, Int32& compressOffset) in /_/RabbitMQ.Stream.Client/RawConsumer.cs:line 26

next error has same timestamp so I think they are related.
Error while Dispatching message, ChunkId : 735383 Consumer id .....
error
System.NullReferenceException: Object reference not set to an instance of an object.
at RabbitMQ.Stream.Client.RawConsumer.<>c__DisplayClass18_0.<g__DispatchMessage|1>d.MoveNext()

[troepolik]

@Gsantomaggio CRC check error is also reproducing with new driver version