Skip to content

Link claimed_executions to processes via process_name #601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

rosa
Copy link
Member

@rosa rosa commented Jul 21, 2025

That is, instead of linking them via the standard process_id (claimed_execution.process_id and process.id).

The reason for this change is that we're exploring the use of Solid Queue in a multi-tenanted setup, where we'll have an individual database per tenant, but a shared supervisor, dispatcher, scheduler, and workers for multiple tenants. We'll keep a supervisor-local database separated from the tenanted DBs, with just the solid_queue_processes table. The supervisor will control the processes stored in that table in that local DB, whereas each tenant will use its own database with the remaining tables, which will contain the jobs and the job metadata.

The consequence of this is that we can no longer rely on finding a process's in-flight jobs via its id, because the id can be (and most likely will be) duplicated across supervisors' local databases. The name, however, is randomly generated via SecureRandom.hex(10), which, with a very high likelihood, guarantees unique names across supervisors' local DBs. In this way, if a process is killed, leaving in-flight jobs (claimed executions) behind, and we fail over a different supervisor with a different local DB, that supervisor will be able to know which claimed executions are orphaned.

This requires a new migration, which is added here, but it's not enforced. The code still works without running the new migration, it just emits a deprecation warning like this:

DEPRECATION WARNING: Solid Queue has pending database migrations. To get the new migration files, run:
  rails solid_queue:update
And then:
  rails db:migrate
These migrations will be required after version 2.0

This also includes a new solid_queue:update command to facilitate installing this migration and future migrations.

cc @flavorjones

unless connection.column_exists?(:solid_queue_claimed_executions, :process_name)
add_column :solid_queue_claimed_executions, :process_name, :string
add_index :solid_queue_claimed_executions, :process_name
end
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This migration is idempotent because it is already included in the setup schema, for people who are installing Solid Queue for the first time. If there's a new update in the future with a new migration, people should be able to run all of them without any issues.

@rosa rosa force-pushed the claimed-executions-by-process-name branch from bc62970 to b18e146 Compare July 22, 2025 15:46
rosa added 11 commits July 22, 2025 17:48
And include it in the host application's deprecators.
It's been a long time already since version 1.0 went out.
To be used as `bin/rails solid_queue:update` to copy new migrations.

The new migration links claimed executions to processes with
`process_name`, and replaces the unique index on processes on
`supervisor_id, name` with just `name`, as we'll want the name to
uniquely identify a process because we'll rely on that to release
claimed executions, independently from the supervisor. Even though it
was really unlikely to have a collision on name because these are set
randomly with `SecureRandom.hex(10)`, in this way we guarantee it.

fix
People installing Solid Queue for the first time will get the final
schema. Migrations are idempotent so they can later update and run
migrations without any problems.
We can only use this if new migrations have been run. If not, we just
emit a deprecation warning and continue as before.
In the deprecation warning and upgrade instructions.
@rosa rosa force-pushed the claimed-executions-by-process-name branch from b18e146 to a06ef6b Compare July 22, 2025 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant