Skip to content

Rework preservation audit batch creation #6975

@hackartisan

Description

@hackartisan

Summary or User Story

The new audit failed in prod with Connection reset by peer (redis://figgy-web-prod1.princeton.edu:6379/1) (RedisClient::ConnectionError)

We should try getting a count of IDs, then splitting off sub-jobs that add preservation check jobs to the batch. if we make sure the ids query is sorted we can use an offset and limit to get the right chunk of ids from the database.

Acceptance Criteria

  • the batch creation is broken up so that all preservation check jobs aren't enqueued at once.

First step

check sidekiq docs https://github.com/sidekiq/sidekiq/wiki/Batches#huge-batches

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions