Add support for job dependencies in pg boss

[klesgidis]

Summary

Add first-class job dependencies to pg-boss — a job can declare dependsOn parent jobs and won't be fetched until all parents complete. Enables fan-in and pipeline patterns without application-level orchestration.

What changed

• dependsOn option on send() / insert() — dependent jobs are inserted as blocked = true, invisible to fetch
• job_dependency table — normalized parent/child tracking with efficient indexes for both directions
• Automatic unblocking — CTE in completeJobs checks and unblocks children when all parents are done (no-op for queues without dependencies)
• blocking flag on parent jobs — lets the completion CTE short-circuit
• Updated job_i5 fetch index — adds AND NOT blocked at the index level, zero cost for non-dependent jobs
• getDependencies() / getDependents() — query the dependency graph
• Validation, cleanup, migration v30→v31 — all additive, no breaking changes

Usage

const jobA = await boss.send('process-data', { file: '1.csv' })
const jobB = await boss.send('process-data', { file: '2.csv' })

// jobC won't start until both jobA and jobB complete
const jobC = await boss.send('aggregate-results', { output: 'report.csv' }, {
  dependsOn: [
    { name: 'process-data', id: jobA },
    { name: 'process-data', id: jobB }
  ]
})

// query the graph
const parents = await boss.getDependencies('aggregate-results', jobC)
const children = await boss.getDependents('process-data', jobA)

Resolves #745

[coveralls]

coverage: 100.0%. remained the same
when pulling fb7a8e7 on klesgidis:master
into a827ec4 on timgit:master.

[timgit]

Thanks for submitting this. I have a few change requests on the implementation.

When creating a "dep graph", "flow", etc., there is the possibility of race conditions, and I'm wondering what weaknesses this introduces into how to support issues that may come up.

Since the job creation is not batched into a transaction, there is the possibility that a dependent job is completed before a dependency on it is created. I'm not sure what complexity this creates, but one mitigation I noticed after reading Bull's flow docs is they require all payloads/jobs in the same request. If we constrain ourselves to a single batch request, we could assign blocking and blocked simultaneously without requiring another round trip.

My other primary concern is how to mitigate excessive querying that starts with the join in the children_to_check CTE, since the where condition won't prevent hitting the dependency table. I don't have a quick suggestion to this yet, but I'm thinking about how that query could become opt-in somehow.

[klesgidis]

I agree with both of your concerns.

For the first one:
I'll replace the per-job dependsOn option on send() with a dedicated createFlow() method that accepts the full set of jobs and their dependency edges in a single call. Internally it wraps everything in one transaction.

For the second:
For the children_to_check CTE cost, I'll make it structurally opt-in at the queue level rather than relying solely on the blocking per-job filter.

Add a has_dependencies boolean (default false) to the queue table. It gets set to true automatically inside the createFlow() transaction when a dependency touches a queue.
The queue cache (already used by getQueueCache()) picks up the flag, so complete() knows at call time whether to use the lean query (just the results CTE) or the full query with children_to_check and unblocked.

Queues that never participate in dependencies never see the extra CTEs — zero overhead, not even in the query plan.

WDYT?

[timgit]

I like your thought of using the queue and maybe also the cache. That would address a noisy neighbor use case. It does have the downside of affecting all jobs in that queue, however.

Are you imagining a situation where you either only use deps on a queue for every job or never use them? My first thought when you opened the issue was "some jobs in this queue might have deps and some won't".

[klesgidis]

Are you imagining a situation where you either only use deps on a queue for every job or never use them? My first thought when you opened the issue was "some jobs in this queue might have deps and some won't".

Yes, my requirement is some jobs would have deps and some not. Do you have any other idea?

[timgit]

Add a has_dependencies boolean (default false) to the queue table. It gets set to true automatically inside the createFlow() transaction when a dependency touches a queue.

This will require resetting the cache across all connected instances. I like the concept, but in a distributed setup, we'd have a race condition issue to deal with.

I'm still looking at alternatives to bypassing the downstream CTE. One option is to inspect the results to see if flows were involved, then have the client issue another query to process them. It's not as elegant as a single command, for sure. A pg function could also be created that would be able to run this in a single transaction via control flow.

[klesgidis]

I find the pg function a nice idea. Do you want me to approach this way and discuss after this? The function would be:

-- 1. Complete the jobs
-- 2. Check if any completed job was a parent
-- 3. Only if needed, run the dependency check + unblock the affected parents

[klesgidis]

Moved here #777