scx_p2dq: Add DHQ support #3035
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hodgesds
requested review from
JakeHillion,
arighi,
etsal,
htejun,
multics69 and
tommy-u
hodgesds
changed the title
hodgesds
force-pushed
the
p2dq-dhq
branch
6 times, most recently
from
f611110 to
ea228ea
Compare
Contributor
Author
|
Performance tests using
|
hodgesds
force-pushed
the
p2dq-dhq
branch
3 times, most recently
from
a56db2a to
05388d2
Compare
Implement Double Helix Queue (DHQ), a dual-strand priority queue designed for LLC-aware task migration in multi-cache systems. DHQ maintains two parallel strands (analogous to DNA's double helix) with coordinated access to preserve cache affinity while enabling work-stealing. Key features: - Fixed-size implementation with pre-allocated capacity for use in non-sleepable BPF contexts (enqueue/dispatch callbacks) - Strand pairing constraint prevents unbounded imbalance between strands - Three dequeue modes: Priority (lowest vtime), Alternating (fair), and Balanced (load-aware) - Both FIFO and VTime ordering modes within each strand - Arena-based allocation for scalable concurrent access DHQ advantages over single queues: - Cache locality: Each strand maps to an LLC, preserving cache warmth - Controlled migration: max_imbalance parameter limits cross-LLC movement - Lower lock contention: Operations distributed across two strands - Work conservation: Priority mode allows stealing while respecting affinity - Prevents pathological cases where one LLC monopolizes migration queue Implementation details: - Backed by red-black trees (via scx_minheap) for O(log n) operations - Strand constraint enforced at both enqueue and dequeue time - Returns -EAGAIN when strand imbalance would be violated - Returns -ENOSPC when capacity is reached - Thread-safe via arena spinlocks Files added: - lib/dhq.bpf.c: Core DHQ implementation - lib/DHQ_README.md: Comprehensive documentation with complexity analysis - lib/selftests/st_dhq.bpf.c: Unit tests for DHQ operations Designed for LLC migration, where: - LLC pairs in same NUMA node share one DHQ - Strand A = LLC 0 tasks (cache-warm to LLC 0) - Strand B = LLC 1 tasks (cache-warm to LLC 1) - Priority mode migrates highest-urgency tasks across LLCs - max_imbalance controls migration rate Signed-off-by: Daniel Hodges <[email protected]>
Integrate Double Helix Queue (DHQ) as an alternative to ATQ for LLC-aware task migration, and fix critical race condition causing migration-disabled task errors. DHQ Integration: - Add --dhq-enabled flag to enable DHQ mode for LLC migration - Add --dhq-max-imbalance parameter (default: 3) to control strand balance - Create one DHQ per pair of LLCs in same NUMA node - Map each LLC to a specific strand (A or B) for cache affinity - Each CPU inherits strand from its LLC for proper load distribution - DHQ provides cache-aware migration with controlled cross-LLC movement Strand-Specific DHQ Operations: Use scx_dhq_peek_strand() and scx_dhq_pop_strand() instead of generic operations to ensure CPUs only consume from their designated strand. This preserves cache locality and prevents load imbalance. Data Structure Changes: - Add mig_dhq and dhq_strand to cpu_ctx and llc_ctx - Add llc_pair_dhqs[] for shared DHQs between LLC pairs - Add llcs_per_node[] to track LLCs per NUMA node - Add P2DQ_ENQUEUE_PROMISE_DHQ_VTIME enqueue promise type - Add enqueue_promise_dhq struct for DHQ-specific metadata Configuration: - p2dq_config.dhq_enabled: Enable DHQ mode - p2dq_config.dhq_max_imbalance: Control strand pairing (0 = unlimited) - Priority mode: lowest vtime wins across strands Build System: - Add lib/dhq.bpf.c to scx_p2dq and scx_chaos builds - Include lib/dhq.h in types.h scx_chaos Compatibility: - Update enqueue promise handling to recognize DHQ type - Error message updated to mention both ATQs and DHQs not supported Benefits: - Cache affinity: Tasks stay on origin LLC (strand) - Controlled migration: max_imbalance prevents migration storms - Race-free: Atomic affinity handling eliminates migration-disabled errors - Work conservation: Cross-strand stealing when priority demands - Scalable: Lock contention distributed across DHQ strands Signed-off-by: Daniel Hodges <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
scx_p2dq: Add DHQ support and fix migration-disabled task errors
Integrate Double Helix Queue (DHQ) as an alternative to ATQ for LLC-aware task migration.
DHQ Integration:
Strand-Specific DHQ Operations:
Use scx_dhq_peek_strand() and scx_dhq_pop_strand() instead of generic operations to ensure CPUs only consume from their designated strand. This preserves cache locality and prevents load imbalance.
Data Structure Changes:
Configuration:
Build System:
scx_chaos Compatibility:
Benefits:
Signed-off-by: Daniel Hodges [email protected]