Significant lag in filebeat shutdown when configured with kafka output

[khushijain21]

I was able to consistently reproduce this with filebeat on main branch. Filebeat took about ~2mins to shut down.

Use below config to reproduce this

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /var/log/*.log

setup.template.settings:
  index.number_of_shards: 1

output.kafka:
  hosts: ["0.0.0.0:9092"]
  enabled: true
  topic: beats

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Wait for some time before stopping filebeat and notice the lag. I am attaching the logs here and observe the timestamps as well. Shutdown was triggered at 11::48:: and filebeat finally shuts down at 11:50::

https://github.com/khushijain21/beats/blob/newbranch/filebeat/filebeat-20260303-5.ndjson

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[AndersonQ]

Hi, out of curiosity, did you try with other outputs to see how long they take to shutdown?