Allow running scheduler and server without worker

[mccormickt]

Feature Request

Is your feature request related to a problem? Please describe.

The current arguments to start only allow you to run the server or worker individually, the server and scheduler together, or all components together. You cannot run the scheduler and server together without the worker. I would think this could be a common pattern to scale workers horizontally while using a single server + scheduler to dispatch them work and the restriction doesn't seem intentional as far as I can tell.

Describe the solution you'd like

Separation of CLI arguments to the start command to allow for all combinations of the components - server, worker, and scheduler - to be able to run together.

Describe alternatives you've considered

We've implemented this with individual CLI arguments for start, deprecating the --server-and-worker arg. Since there is also a scheduler subcommand, perhaps a more holistic change could address that as well.

[kaplanelad]

Hey ey @mccormickt, thanks for opening this valid use case.

The scheduler is fully independent from the worker (runs as its own tokio task, executes tasks as subprocesses, not through the job queue), so there's no technical reason they must be coupled.

Would adding a composable --scheduler flag solve your use case?

[mccormickt]

Hey ey @mccormickt, thanks for opening this valid use case.

The scheduler is fully independent from the worker (runs as its own tokio task, executes tasks as subprocesses, not through the job queue), so there's no technical reason they must be coupled.

Would adding a composable --scheduler flag solve your use case?

Absolutely. I'll open a PR from our patch that did exactly that, and we can discuss the approach.