-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Labels
Description
Summary or User Story
The new audit failed in prod with Connection reset by peer (redis://figgy-web-prod1.princeton.edu:6379/1) (RedisClient::ConnectionError)
We should try getting a count of IDs, then splitting off sub-jobs that add preservation check jobs to the batch. if we make sure the ids query is sorted we can use an offset and limit to get the right chunk of ids from the database.
Acceptance Criteria
- the batch creation is broken up so that all preservation check jobs aren't enqueued at once.
First step
check sidekiq docs https://github.com/sidekiq/sidekiq/wiki/Batches#huge-batches