Store state court documents under the RECAP path layout

[albertisfu]

Describe the issue.

Documents from models that inherit from AbstractStateDocument (FloridaDocument, NYCoADocument, and TexasDocument) are currently stored in S3 using state_pdf_path from cl/search/state/shared.py:

us/state/<state_code>/<court_id>/gov.<state_code>.<court_id>.<slug><ext>

We want to move these files to the same path structure we use for RECAP documents, using a directory per docket:

recap/gov.uscourts.<court_id>.<cl_docket_id>/gov.uscourts.<court_id>.<cl_docket_id>.<x>.<x><ext>

Where <cl_docket_id> is the CourtListener docket PK, not the PACER case ID.

Document naming

For RECAP, we use <document_number>.<attachment_number> for the <x>.<x> part.

State docket entries don't have equivalent fields. Florida uses UUIDs, New York has entry_index, and Texas has sequence_number.

We need something that is unique within the docket. I think we can use:

<docket_entry_id>.<document_id>

using the CL PKs.

This needs to be unique because in the follow-up migration we'll do raw S3 copies, where the incrementing storage won't protect us from collisions.

Tasks

• Agree on the <x>.<x> naming.
• Update AbstractStateDocument.state_pdf_path to use the new path.
• Keep the current extension handling since Texas and New York can have non-PDF files.
• Update the path tests for the three state document models.
• Deploy this before starting the migration in the follow-up issue, so new documents stop being stored under us/state/.

Notes

• This issue only changes the path for new files. Existing files will be moved in a separate issue.
• The recap/ prefix will now be shared between RECAP and state documents.

Related Issue

No response

Screenshots

No response

What type of issue is this?

Research

[albertisfu]

@Brennan-Chesley-FLP and @mlissner, do you have any thoughts on using <docket_entry_id>.<document_id> to replace the <document_number>.<attachment_number> part that we use in RECAP for state content, where we don't have document numbers?

For SCOTUS, we can keep <document_number>.<attachment_number> since we do have those values.

[Brennan-Chesley-FLP]

I don't think we should use PKs in the document names if we can avoid it. If we do that, we are committing to either a move/copy in S3, a complicated download dance, or a Django model that doesn't point at anything until we download it. For NYCoA, it would be nice to go with something like recap/gov.uscourts.ny.<docket_number>/gov.uscourts.ny.<docket_number>.<filename/title>.<sha256[0:16]><ext> . This gives us an ID that's derivable without first creating database entries, an ID that's pinned to content, and should work generally for scrapers that can't gather meaningful docket ids from the court site. The dance is just: temporary local stream ->(sha256 compute)-> upload to S3 (which we'll have to do in most cases where we want to conserve memory anyhow).

[albertisfu]

Got it! So what you want to achieve here is having future state work download content directly to S3 using a final path and filename that can be derived solely from the scraped data. Ideally, we should be able to determine the document's final key from the scraped data and store it directly in filepath_local, with no additional movement or renaming during the ingestion process, right?

For NYCoA, we don't have a unique identifier for documents, so using the SHA-256 makes sense since we already have it available from the scraped data.

For Texas and Florida, however, we do have UUIDs that are already part of the document filenames, for example:

https://storage.courtlistener.com/us/state/tx/txctapp15/gov.tx.txctapp15.f1616114-bf2e-4417-aa1a-92b8baffbad4-cc9b9580-fd80-47db-adbc-0bcaff9bb176.pdf

https://storage.courtlistener.com/us/state/fl/fladistctapp3/gov.fl.fladistctapp3.record-on-appeal-07c6b0e3-eda9-4945-95de-2bca1fd3243f.pdf

So I was wondering if we could use those UUIDs since we already have them.

Another issue I see is using docket_number in the key. For instance, in Texas we have docket numbers containing commas, such as:

WR-82,851-01

https://www.courtlistener.com/admin/search/docket/72998721/

I'm guessing numbers like this could be problematic in URLs.

What about using docket_number_core instead, which is a cleaned version of the docket number? In this case, it would be wr8285101.

If we do that, though, we'd need to move the docket-number cleaning logic to scrape time, right?

So, for Florida and Texas, we could have something like:

recap/gov.uscourts.<court_id>.<cl_docket_number_core>/gov.uscourts.<court_id>.<cl_docket_number_core>.uuid<ext>

We could also use the SHA-256 for Texas and Florida, but that would require fetching each file to generate the SHA-256 since it's not currently available in the model. That would make the migration process significantly more complex and time-consuming than simply doing a server-side copy to the new path. I'm trying to avoid that since we'd need to move around 3.5 million documents.

So maybe we could end up with something like this:

• Use docket_number_core if you think generating it at scrape time is a good approach.
• Use the document UUID when it's available in the scraped metadata.
• If the metadata doesn't provide a UUID for the document, use the SHA-256 instead.

What do you think?

[Brennan-Chesley-FLP]

Are there particular things we're hoping to capture with this naming change? It can be nice to have things normalized, but are there properties that we're shooting for in that normalization?

• They need to be valid S3 paths.
• They should be derivable from scraped data.

Do we care about S3 factors?

• We might care about S3 prefix rate limits?
• We might care about S3 prefixes for searching?

Given a file that a user has downloaded what do we want to be able to get back from that?

• The court and docket that it came from?
• Differentiation between files on the same docket?
• Hint to the contents/subject of the file?
• A version stamp in the filename for differentiating between versions of "the same file"?

Given a download URL, are we looking to add anything additional to what we can infer?

[albertisfu]

Well, we want to keep the base /recap/ directory and this structure:

/recap/gov.uscourts.<court_id>.<docket_identifier>/

This way, we can preserve the redirect functionality to the CL docket page. For instance, if you remove the filename and just go to:

https://storage.courtlistener.com/recap/gov.uscourts.casd.855944/

you'll be redirected to the CL docket page.

In RECAP, this works using the court_id and pacer_case_id, so we can look up the docket and redirect to its page. See: redirect_docket_recap.

In SCOTUS, Texas, and probably many other future states, we don't have a unique docket identifier.

So, to preserve the redirect, we first need the court_id and a docket identifier. This can't be the docket_number, since it may contain unsafe characters, as described in valid S3 paths.

So it looks like the only option left is to use docket_number_core, since it can be derived at scrape time, right?

For the filename, I think we just want it to look similar to what we have for RECAP:

gov.uscourts.<court_id>.<docket_number_core>.<x>.<x><ext>

If we want to keep filenames normalized for state content, it looks like sha256 is the only option that we can ensure will always be available, right? As we won't always have file UUIDs like we do in Texas or Florida, or document numbers like we do in SCOTUS.