Skip to content

[Performance] Hot key read contention due to exclusive lock in expireKeyIfNeeded #459

Description

@wsa-wsa

Description

When multiple clients concurrently read the same key (e.g., thousands of GET requests on a hot key), performance degrades significantly.

Expected Behavior

Read commands should be able to execute concurrently on the same key without blocking each other. Only write commands need exclusive access.

Context

Image

Root cause: The expireKeyIfNeeded function acquires a lock via the RdLock(), which always uses an LOCK_X(exclusive lock). This lock is acquired for all commands, including pure read-only commands like GET, HGET, LRANGE, ZRANGE, and TTL. Consequently, all operations on the same key are fully serialized, even though read operations do not modify any data.

The exclusive lock in expireKeyIfNeeded exists because it may synchronously delete expired keys. But for read commands, returning nil for expired keys is sufficient — deletion can be deferred.

IndexManager::scanExpiredKeysJob already provides background expired key cleanup as a safety net.
This bottleneck is most pronounced under high-concurrency hot key workloads.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions