High TCPConnector.connect latency in aiohttp when running in Kubernetes

[ramundomario]

Description

We're experiencing unexpectedly high latency during TCP connection establishment (TCPConnector.connect) when running our Python application using aiohttp client inside Kubernetes.

Sometimes the TCPConnector.connect step takes approximately 15 seconds per request.
When it happens, the number looks pretty static, with only minor variations (a few milliseconds), which suggests a systematic issue rather than random network delays.

---

Context

• Runtime: Python + asyncio + aiohttp
• Architecture: An asynchronous worker built with aio-pika that consumes jobs from RabbitMQ and makes, in addition, HTTP requests to internal services using aiohttp.
• DNS Settings in Kubernetes:
  ndots is set to 1 so every URL must by FQDN.
• aiohttp usage pattern:
  The HTTP client is created using aiohttp.ClientSession together with aiohttp.TCPConnector for each request (we're not re-using the ClientSession nor the TCPConnector)

---

Observations

• The latency consistently happens during the TCPConnector.connect phase
• The issue is independent of the remote service, as it happens across different internal hosts
• Changing force_close and other TCPConnector settings did not resolve the issue (as reported in aiohttp.ClientSession together with ClusterIp not load balancing #6629)
• Based on DataDog traces, out of 293 indexed spans of TCPConnector.connect, only 13 showed the long (~15s) latency.
  Traces are subject to sampling and do not represent total calls

---

Versions

• aiohttp version: 3.11.18
• Python version: 3.11.13

---

Looking for help understanding this behavior and any guidance or best practices to mitigate the latency.
Any pointers on diagnosing, configuration, or known issues would be greatly appreciated.

Thank you for your support and for maintaining aiohttp!

[Dreamsorcerer]

Does switching between ThreadedResolver and AsyncResolver show a change in behaviour?

[ramundomario]

Hey @Dreamsorcerer, first of all thanks a lot for your feedback

Does switching between ThreadedResolver and AsyncResolver show a change in behaviour?

Nope, I didn't try that, will do it tomorrow
So you suspect some weird scenario with DNS resolution in thread?

[Dreamsorcerer]

No idea. There's been a few changes in aiodns causing problems for some people, so could be something happening there.