Dynamic scheduled tasks by cupatea · Pull Request #553 · rails/solid_queue

cupatea · 2025-04-23T19:58:48Z

Fixes #186

Add resque-scheduler style dynamic schedules feature, allowing you to add or remove recurring tasks at runtime without touching your static config file.

What’s new:

Dynamic scope (static: false) on SolidQueue::RecurringTask model.
Renamed methods/vars in SolidQueue::Scheduler::RecurringSchedule to distinguish static vs. dynamic schedules.
On boot, @configured_tasks now includes static and dynamic tasks.
Added SolidQueue::Scheduler::RecurringSchedule.update_scheduled_tasks:
- Schedules any new dynamic tasks added since the last run.
- Unschedule any dynamic tasks removed from the database.
- Refreshes @configured_tasks to match the current DB state.
SolidQueue::Configuration no longer requires a non-blank static config file - pure dynamic scheduling is now supported.
SolidQueue::Scheduler watches for changes after launch and updates its metadata so the running process always reflects the true set of recurring tasks.

Tests verify that adding or dropping dynamic tasks at runtime correctly updates what’s scheduled.

cupatea · 2025-06-12T08:49:32Z

Hi @rosa 👋, could you please take a look at this PR when you have a moment? Thanks so much!

rosa · 2025-06-17T20:00:11Z

Hey @cupatea, thanks for this! It's a good start, but it needs a few changes. The main ones are:

We need to make the sleep interval for the scheduler configurable, as a sort of polling interval, to reload dynamic tasks.
The way to use this needs to be a bit friendlier, without users having to go and figure out the details to create a SolidQueue::RecurringTask record. Something like SolidQueue.schedule_recurring_task could work, and it needs to be documented.
We'd need the ability to delete dynamically scheduled tasks as well, by task ID. If the task is static, this could simply raise an error.

And then some other more specific changes that I'll note in the code.

rosa · 2025-06-17T20:01:06Z

lib/solid_queue/configuration.rb


      def invalid_tasks
-        recurring_tasks.select(&:invalid?)
+        static_recurring_tasks.select(&:invalid?)


This doesn't need to change names. It's clear the tasks here are static since this comes from the recurring.yml configuration. We don't need to rename anything here.

rosa · 2025-06-17T20:14:06Z

lib/solid_queue/scheduler.rb

+          recurring_schedule.update_scheduled_tasks.tap do |updated_tasks|
+            if updated_tasks.any?
+              process.update_columns(metadata: metadata.compact)
+            end


This code is mixing actions at very different levels, making it aware of details it shouldn't need to know, like how to update the metadata for its registered process record, or whether the recurring schedule changed. It should change, perhaps to something like

recurring_schedule.reload! if recurring_schedule.changed? refresh_registered_process end

And refresh_registered_process would go in SolidQueue::Processes::Registrable.

Hi @rosa do you think this Dynamic scheduled tasks will merge by the end of this year 2026? thanks

@lrjbrual oh yes, for sure.

cupatea · 2025-06-19T15:11:48Z

Hi @rosa, thank you for the feedback! I think it's ready for the second round of review

bruno-berchielli · 2025-08-21T14:45:38Z

@cupatea , in the docs files you mention SolidQueue.delete_recurring_task but I think it should be SolidQueue.destroy_recurring_task, also, you might consider using the key instead of id, because that's is what probably gonna be easily available on the app level, no?
SolidQueue::RecurringTask.find_by(key: solid_queue_key)&.destroy!

cupatea · 2025-08-21T16:11:57Z

@cupatea , in the docs files you mention SolidQueue.delete_recurring_task but I think it should be SolidQueue.destroy_recurring_task, also, you might consider using the key instead of id, because that's is what probably gonna be easily available on the app level, no? SolidQueue::RecurringTask.find_by(key: solid_queue_key)&.destroy!

Thanks for noticing, on a way to fix that!
And yes, using a key instead of ID makes more sense to me as well. To prevent deleting a static task, this method will raise a record not found error in case of specifying a static task key (or if you try deleting a task that does not exist)

Two assertions were using assert value, message instead of assert_equal/assert_empty, meaning they always passed regardless of the actual metadata content. Fixed to use assert_empty. Also fixed a typo ("unschedule" -> "unscheduled").

DB queries in reload! (dynamic_tasks.where.not(...), RecurringTask.pluck(:key)) were not wrapped in the app executor, which could cause connection management issues. Wrapped in wrap_in_app_executor.

empty? checked stale configured_tasks (lib/solid_queue/scheduler/recurring_schedule.rb) -- configured_tasks was set once in initialize and never updated with dynamic tasks. This meant empty? could return true even when dynamic tasks existed, causing the scheduler to exit prematurely in inline mode. Changed empty? to check scheduled_tasks.empty? && dynamic_tasks.none?.

Changes not cleared after consumption (lib/solid_queue/scheduler.rb, lib/solid_queue/scheduler/recurring_schedule.rb) -- Added clear_changes method and call it in the scheduler's run loop after refresh_registered_process, preventing stale change state from persisting.

Remove extra blank lines, column-aligned hashes, and inline comments on test assertions. Revert unrelated Gemfile.lock platform addition. Filter :static from schedule_task options since dynamic tasks are always non-static by definition. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Dynamic tasks require explicit `dynamic_tasks: true` in the scheduler config. Without it, the scheduler behaves as before — no extra DB queries, no polling for dynamic changes. Default polling interval changed from 1s to 5s. Rename schedule_task/unschedule_task to schedule_recurring_task/ unschedule_recurring_task so the API clearly communicates these are recurring tasks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…tasks_enabled Have from_configuration read static from options instead of hardcoding it. Callers that create static tasks (YAML config) pass static: true via reverse_merge. schedule_recurring_task passes static: false directly. Rename the config key from dynamic_tasks to dynamic_tasks_enabled for clarity. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Keep the longer sleep interval when dynamic tasks are disabled. Add tests for `dynamic_tasks_enabled` opt-in in configuration and for verifying dynamic tasks are ignored when not enabled.

We don't need to track dynamic task changes, we can just reload the metadata in every case. If it hasn't changed, Rails won't issue any new update to the process record. Also, we can just use `dynamic_tasks` everywhere, as an empty AR relation if dynamic tasks are disabled, avoiding extra queries but keeping the code simpler.

rosa · 2026-02-25T15:15:19Z

Hey @cupatea, sorry for the delay here. I'm finally working towards Solid Queue 2.0 with the plan of including this. I've pushed a bunch of changes. Some are stylistic and simplifications, but I've also changed the default behaviour. I think this feature should be opt-in, disabled by default, as people might not need dynamic tasks at all and could save the extra queries to the DB.

We need the dynamic task keys there and were doing a new query every time. We only need to do a query when explicitly reloading them.

cupatea · 2026-02-25T15:54:51Z

Hey @cupatea, sorry for the delay here. I'm finally working towards Solid Queue 2.0 with the plan of including this. I've pushed a bunch of changes. Some are stylistic and simplifications, but I've also changed the default behaviour. I think this feature should be opt-in, disabled by default, as people might not need dynamic tasks at all and could save the extra queries to the DB.

Thank you!

I noticed something in app/models/solid_queue/recurring_task.rb:40.
It looks like this should use **options.merge(static: false) instead of reverse_merge, since we want to ensure the static option is always overridden

rosa · 2026-02-25T17:47:44Z

I noticed something in app/models/solid_queue/recurring_task.rb:40.
It looks like this should use **options.merge(static: false) instead of reverse_merge, since we want to ensure the > static option is always overridden

Ohh, totally! My bad. I had it backwards before, switched it around, but forgot to update the method. Thanks!

We want to set that value overriding whatever we get in options, so it should be merged into options, not the other way around. Co-Authored-By: Vladyslav Davydenko <vladyslav@hey.com>

rosa · 2026-02-25T18:53:14Z

This should be ready to go but I'd like to test it a bit better before merging. Have you been using this already in production by any chance, @cupatea @lrjbrual?

cupatea · 2026-02-25T21:05:11Z

This should be ready to go but I'd like to test it a bit better before merging. Have you been using this already in production by any chance, @cupatea @lrjbrual?

Not really - I migrated from Resque to this and deployed to the test server, but I haven’t deployed it to production yet.
Our production is pretty light in terms of jobs number, so it's more for getting rid of Redis dependency

cupatea mentioned this pull request Apr 23, 2025

Dynamic scheduled tasks #186

Open

cupatea force-pushed the feature/dynamic_recurring_tasks branch 3 times, most recently from ebb8716 to ebd9629 Compare April 24, 2025 08:54

rosa reviewed Jun 17, 2025

View reviewed changes

cupatea force-pushed the feature/dynamic_recurring_tasks branch from ad943cc to 6a883a7 Compare June 19, 2025 15:03

cupatea requested a review from rosa June 19, 2025 15:11

cupatea force-pushed the feature/dynamic_recurring_tasks branch from 6a883a7 to c754746 Compare July 12, 2025 09:28

cupatea force-pushed the feature/dynamic_recurring_tasks branch from c754746 to 214a7f6 Compare August 14, 2025 17:06

cupatea force-pushed the feature/dynamic_recurring_tasks branch from 017c34e to dd72f10 Compare November 24, 2025 09:27

cupatea force-pushed the feature/dynamic_recurring_tasks branch from dd72f10 to 62bfa37 Compare February 12, 2026 22:33

cupatea and others added 13 commits February 24, 2026 21:58

Feature: Dynamic scheduled tasks

9347dbd

Extract polling_interval to scheduler configuration

3a23087

Fix abstraction for RecurringSchedule and Process

3a13962

Add create and destroy recurring task helpers

d1c0efc

Update README with Recurring tasks info

77ff523

Use task key instead of id

f1028a0

Fix mismatches in readme

7dd67c8

Use correct asserts in tests

eef747f

Two assertions were using assert value, message instead of assert_equal/assert_empty, meaning they always passed regardless of the actual metadata content. Fixed to use assert_empty. Also fixed a typo ("unschedule" -> "unscheduled").

Add missing platform to Gemfile.lock

8e1be8b

Move some Scheduler::RecurringSchedule methods to private

5ba7384

Wrap reload in app executor

dd67d58

DB queries in reload! (dynamic_tasks.where.not(...), RecurringTask.pluck(:key)) were not wrapped in the app executor, which could cause connection management issues. Wrapped in wrap_in_app_executor.

cupatea and others added 12 commits February 24, 2026 21:58

Add test for enqueuing the job

99ba202

Improve create_recurring_task - use RecurringTask .from_configuration

79fca8d

Fix names for public methods

7ca2eb7

Prevent stale configured_tasks value

890dabe

Use wrap_in_app_executor for refresh_registered_process

abbf54e

Update tests and README

4ec36d6

Refactor scheduler loop and add missing tests

b012535

Keep the longer sleep interval when dynamic tasks are disabled. Add tests for `dynamic_tasks_enabled` opt-in in configuration and for verifying dynamic tasks are ignored when not enabled.

Clean a bit top-level SolidQueue methods to manage dynamic tasks

b4f7c99

rosa force-pushed the feature/dynamic_recurring_tasks branch from 62bfa37 to b4f7c99 Compare February 25, 2026 15:13

Avoid extra queries for dynamic tasks for process metadata and procline

7ded710

We need the dynamic task keys there and were doing a new query every time. We only need to do a query when explicitly reloading them.

Merge right value of static correctly

3b490ad

We want to set that value overriding whatever we get in options, so it should be merged into options, not the other way around. Co-Authored-By: Vladyslav Davydenko <vladyslav@hey.com>

Conversation

cupatea commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cupatea commented Jun 12, 2025

Uh oh!

rosa commented Jun 17, 2025

Uh oh!

rosa Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

rosa Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

lrjbrual Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

rosa Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

cupatea commented Jun 19, 2025

Uh oh!

bruno-berchielli commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cupatea commented Aug 21, 2025

Uh oh!

rosa commented Feb 25, 2026

Uh oh!

cupatea commented Feb 25, 2026

Uh oh!

rosa commented Feb 25, 2026

Uh oh!

rosa commented Feb 25, 2026

Uh oh!

cupatea commented Feb 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cupatea commented Apr 23, 2025 •

edited

Loading

bruno-berchielli commented Aug 21, 2025 •

edited

Loading