Skip to content

Conversation

sjmonson
Copy link
Collaborator

Unify the scheduling method used for async and synchronous modes in preparation for multi-turn conversation support. This change with also significantly reduce the number PIDs used by GuideLLM at high concurrency.

Relates to #196

@sjmonson sjmonson force-pushed the feat/unified_scheduler branch from 17e1be7 to b7a2eaf Compare September 4, 2025 18:26
@sjmonson sjmonson requested a review from Copilot September 4, 2025 18:27
Copilot

This comment was marked as outdated.

@sjmonson sjmonson force-pushed the feat/unified_scheduler branch 2 times, most recently from 330ddf5 to 681d505 Compare September 4, 2025 19:02
@sjmonson sjmonson requested a review from Copilot September 4, 2025 19:03
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Unifies the scheduling methods for both async and synchronous modes by removing the synchronous mode and standardizing on the asynchronous approach. This change is made in preparation for multi-turn conversation support and significantly reduces process IDs at high concurrency.

  • Removed the separate synchronous scheduling path and unified on the asynchronous approach
  • Moved type definitions from scheduler-specific module to a more general request types module
  • Updated the scheduling strategy to use a consistent start time across all processes

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/guidellm/scheduler/worker.py Removed synchronous process loop, unified on async scheduling with updated parameters
src/guidellm/scheduler/strategy.py Added start_time field and updated request_times methods to use consistent timing
src/guidellm/scheduler/scheduler.py Removed sync/async branching logic, updated queue handling and process management
src/guidellm/scheduler/result.py Moved WorkerProcessRequest and WorkerProcessResult dataclasses from worker module
src/guidellm/scheduler/queues.py New helper module for queue type imports and MPQueues dataclass
src/guidellm/request/types.py Formatting update to all list
src/guidellm/request/loader.py Added type hints to abstract methods
src/guidellm/config.py Updated max_worker_processes calculation and added scheduler_start_delay setting
src/guidellm/benchmark/benchmarker.py Updated import path for RequestT and ResponseT types
src/guidellm/benchmark/aggregator.py Updated import path for RequestT and ResponseT types
src/guidellm/scheduler/init.py Removed exports that moved to other modules
src/guidellm/request/init.py Added RequestT and ResponseT to exports

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@sjmonson sjmonson force-pushed the feat/unified_scheduler branch from 681d505 to 8bdfc83 Compare September 4, 2025 19:06
@sjmonson sjmonson force-pushed the feat/unified_scheduler branch from 8bdfc83 to 89b501f Compare September 4, 2025 19:06
@markurtz markurtz self-requested a review September 5, 2025 14:30
markurtz
markurtz previously approved these changes Sep 5, 2025
Copy link
Collaborator

@jaredoconnell jaredoconnell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@sjmonson sjmonson merged commit ad9513f into main Sep 5, 2025
17 checks passed
@sjmonson sjmonson deleted the feat/unified_scheduler branch September 5, 2025 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants