Kafka Producer Network Thread Blocked on ArrayDeque Lock and Virtual Threads stall on ProducerBatch class Initialization in Java 21 Loom

[AdithyaViswanathan]

Hi, I’m running into what appears to be a thread contention or lock inversion issue when using virtual threads with the Kafka producer client. I’m seeing this thread blocked:

"kafka-producer-network-thread | producer-1" #548 [86033] daemon prio=5 os_prio=0 cpu=98.19ms elapsed=4017.58s tid=0x00007fb97820a620 nid=86033 waiting for monitor entry  [0x00007fb7b05e2000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.kafka.clients.producer.internals.RecordAccumulator.partitionReady(RecordAccumulator.java:717)
	- waiting to lock <0x0000000400f3dcc8> (a java.util.ArrayDeque)
	at org.apache.kafka.clients.producer.internals.RecordAccumulator.ready(RecordAccumulator.java:792)
	at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:368)
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:347)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:250)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

At the same time, I have many virtual threads like this:

"ForkJoinPool-1-worker-22" #550 [86035] daemon prio=5 os_prio=0 cpu=2.19ms elapsed=4017.57s tid=0x00007fb99840c760  [0x00007fb7b03e0000]
   Carrying virtual thread #546
	at jdk.internal.vm.Continuation.run([email protected]/Continuation.java:251)
	- waiting on the Class initialization monitor for org.apache.kafka.clients.producer.internals.ProducerBatch
	at java.lang.VirtualThread.runContinuation([email protected]/VirtualThread.java:245)
	at java.lang.VirtualThread$$Lambda/0x00007fb9ac759860.run([email protected]/Unknown Source)
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec([email protected]/ForkJoinTask.java:1423)
	at java.util.concurrent.ForkJoinTask.doExec([email protected]/ForkJoinTask.java:387)
	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec([email protected]/ForkJoinPool.java:1312)
	at java.util.concurrent.ForkJoinPool.scan([email protected]/ForkJoinPool.java:1843)
	at java.util.concurrent.ForkJoinPool.runWorker([email protected]/ForkJoinPool.java:1808)
	at java.util.concurrent.ForkJoinWorkerThread.run([email protected]/ForkJoinWorkerThread.java:188)

I try to send Kafka messages via virtual threads, and during the first Kafka message send, class loading happens lazily.

It seems like the virtual thread trying to send a message touches ProducerBatch and gets blocked on JVM class initialization, while possibly holding the ArrayDeque lock. Meanwhile, the Kafka sender thread is blocked waiting for that same Deque, resulting in a stall. And all my virtual threads that try to send a kafka message seems to wait on the class initialization monitor on ProducerBatch

I use Spring Kafka 3.3.6, with JDK 21, spring version 6.2.7.
This issue is happening randomly and is not easily reproducible. I use KafkaTemplate's send to send the messages.

Is this a known issue when using Loom-based virtual threads with the Kafka producer? Should ProducerBatch be eagerly initialized on startup in a warm-up phase to avoid this situation? Any guidance helps here

[artembilan]

Unfortunately, this is a known limitation of Kafka Client implementation.
Please, consider to raise this concern (or join existing discussion ) with Apache Kafka itself. There is nothing we can do from Spring perspective.
See more info here : #3074.

closed as duplicated.
Please , consider to use search first before creating a new issue.