Is there a best practice when the LLM response is too slow?

[WilliamChen-luckbob]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

When the LLM response is extremely slow, I strongly suspect that Dify spawns and hangs a large number of background threads to handle these tasks.
As users keep retrying and refreshing, the connection pool gets exhausted, triggering numerous exceptions that eventually cause a full page crash. The most obvious symptom is seeing many retries in the F12 dev tools, all ending with (failed)net::ERR_CONNECTION_RESET.

Meanwhile, refreshing the page usually leads to the degraded page showing you're offline like below.

Are there any best practices to mitigate this issue? For example, setting a response timeout for the LLM? Or other ways to optimize these extremely time-consuming processes?
Currently, Dify only freezes the session and response nothing. Even though the response might be compensated into the session after a long wait, it does bring down the entire page. No client can wait this long time.

2. Additional context or comments

No response

[WilliamChen-luckbob]

@dosu

[dosubot[bot]]

Hi! It looks like you mentioned me but didn't include a question. How can I help you? I can assist with questions about Dify, help you find information in the codebase or documentation, or troubleshoot issues you're experiencing.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[dosubot[bot]]

Yes, there are several best practices to mitigate slow LLM response issues in Dify. The symptoms you're describing (connection pool exhaustion, ERR_CONNECTION_RESET, page crashes) are typically caused by mismatched timeout configurations across different layers.

Key Timeout Settings to Configure:

Add these to your .env or docker-compose.yaml:

# Increase Gunicorn timeout (default 200s, recommend 360s+)
GUNICORN_TIMEOUT=360

# Application-level timeouts
APP_MAX_EXECUTION_TIME=1200
WORKFLOW_MAX_EXECUTION_TIME=1200

# Plugin daemon timeout (if using v1.10.0+)
DIFY_BACKWARDS_INVOCATION_READ_TIMEOUT=600000  # 10 minutes in ms

# Nginx timeouts (critical for streaming)
NGINX_PROXY_READ_TIMEOUT=3600s
NGINX_PROXY_SEND_TIMEOUT=3600s

Critical Nginx Configuration:

If you're using a reverse proxy, ensure these settings to prevent HTTP/2 protocol errors:

location /api {
    proxy_http_version 1.1;    # Required - HTTP/2 breaks SSE
    proxy_set_header Connection '';
    proxy_buffering off;       # Critical for streaming
    proxy_cache off;
}

Connection Pool Tuning:

If you're hitting database connection limits, reduce SQLALCHEMY_POOL_SIZE rather than only increasing database connections:

SQLALCHEMY_POOL_SIZE=20
SQLALCHEMY_MAX_OVERFLOW=10

Important Limitation:

There's a hardcoded 240-second timeout for LLM streaming in agent strategies that cannot be changed via environment variables. If your LLM operations regularly exceed this, you may need to modify the source code in core/model_runtime/model_providers/__base/llm/llm_model.py or use the plugin-daemon architecture with DIFY_BACKWARDS_INVOCATION_READ_TIMEOUT.

Additional Tips:

• Use streaming mode for LLM operations to get immediate feedback and reduce timeout risks
• Ensure workflow_storage queue is included in Celery worker commands if using v1.10.0+
• Consider enabling first token timeout retry mechanism for automatic retries on slow LLM responses

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[dosubot[bot]]

This is a known issue with Dify v1.13.0! The version introduced a major architectural change that moves workflow execution from API servers to Celery workers, but the default Celery worker concurrency is set to 1. This means even 1-2 concurrent requests will queue up and appear stuck.

Critical fix — add these to your .env file:

# Increase Celery worker concurrency (default is 1!)
CELERY_AUTO_SCALE=true
CELERY_MAX_WORKERS=15
CELERY_MIN_WORKERS=5

# Or use fixed concurrency
# CELERY_WORKER_CONCURRENCY=10

Also add memory limits for plugin-daemon (which can consume all available memory and cause OOM). Edit your docker-compose.yaml:

plugin_daemon:
  mem_limit: 2g

Additional settings to check in .env:

# Increase connection pool
SQLALCHEMY_POOL_SIZE=50

# Plugin daemon timeout
PLUGIN_MAX_EXECUTION_TIMEOUT=2400
PYTHON_ENV_INIT_TIMEOUT=320

Then do a full restart:

docker compose down && docker compose up -d

To debug when stuck, check these logs:

docker compose logs worker --tail=100
docker compose logs plugin_daemon --tail=100
docker stats  # See which container is consuming resources

The symptom you're describing (works for a bit, then freezes completely requiring restart) is classic Celery worker concurrency exhaustion combined with possible plugin-daemon memory issues.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[WilliamChen-luckbob]

@dosu
As a matter of fact, my server is very powerful—far more powerful than needed to serve a 10-person team. However, what I’ve observed is that everything works normally only for a short while after a fully restart, and it freezes completely just a few minutes later. Meanwhile, the earlier issue (failed)net::ERR_CONNECTION_RESET still persists.

I'm sure to set the .env file and run the generate_docker_compose then restart the full docker compose. But from observation, the situation doesn't seem to have improved much.

Is there any way to make Dify support a higher concurrency level? At the very least, it should not pile up requests when LLM calls get stuck. My single server can handle hundreds of thousands of QPS, and it shouldn’t be overwhelmed by a service for only 10 people. And this server is running alone in the server without any other servers!

[dosubot[bot]]

I understand your frustration — your server hardware isn't the bottleneck here. The issue is that Dify's default configuration is extremely conservative and workflows are serialized by default, causing requests to pile up even with powerful hardware.

Here's a comprehensive concurrency configuration for your .env:

# === API Server ===
# Set to (CPU cores × 2) + 1 for concurrent request handling
SERVER_WORKER_AMOUNT=17  # Adjust based on your CPU cores
SERVER_WORKER_CONNECTIONS=100
GUNICORN_TIMEOUT=360

# === Celery Workers (Critical!) ===
CELERY_AUTO_SCALE=true
CELERY_MAX_WORKERS=20
CELERY_MIN_WORKERS=5
# Or fixed concurrency:
# CELERY_WORKER_CONCURRENCY=15
# CELERY_WORKER_AMOUNT=2

# === Database Pool ===
SQLALCHEMY_POOL_SIZE=100
SQLALCHEMY_MAX_OVERFLOW=20

# === Application Limits ===
APP_MAX_EXECUTION_TIME=1200
WORKFLOW_MAX_EXECUTION_TIME=1200

Important limitation to be aware of: Dify's workflow execution is constrained by Python's GIL and has a hardcoded thread pool of max_workers=10 for parallel node execution. This means even with perfect configuration, a single worker process can only handle ~5 concurrent LLM-heavy requests.

For higher throughput, you may need to run multiple worker containers. You can scale horizontally since all Dify components are stateless:

docker compose up -d --scale worker=3

To diagnose what's blocking, run this when it freezes:

docker compose logs worker --tail=200 | grep -E "(timeout|error|stuck|waiting)"
docker compose exec worker celery -A app.celery inspect active

The ERR_CONNECTION_RESET persisting suggests connections are timing out somewhere. Can you share what GUNICORN_TIMEOUT and CELERY_WORKER_CONCURRENCY values you currently have set?

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[WilliamChen-luckbob]

@dosu
do I have to do any other settings if I use docker compose up -d --scale worker=3?

[dosubot[bot]]

Good news — Dify workers are stateless, so scaling is straightforward. No unique identifiers, sticky sessions, or leader election needed.

The main things to check:

1. Increase database connection pool — with 3 workers, you'll need more connections:

   # In .env, adjust based on worker count
   SQLALCHEMY_POOL_SIZE=100  # Default 30 may not be enough

2. For v1.12.0+, configure Redis PubSub for better horizontal scaling:

   PUBSUB_REDIS_CHANNEL_TYPE=sharded

3. Keep these consistent across all workers (which docker-compose handles automatically):

   • Same PostgreSQL and Redis endpoints
   • Same PLUGIN_DIFY_INNER_API_KEY

That's it! Workers will automatically share the same Redis and PostgreSQL, and Celery will distribute tasks across all instances.

Run:

docker compose down
docker compose up -d --scale worker=3

You can verify all workers are running with:

docker compose ps | grep worker

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}