Spooky README Caching Issue

[toolCHAINZ]

Current Behavior

I've run into what I assume must be a caching issue.

I recently published a new crate named crackers, sourced from this git repo: https://github.com/toolCHAINZ/crackers.

rust-lang/crates.io-index@16d57ab
https://crates.io/crates/crackers/0.1.1

Notice that the crates.io page for the crate has a README that is not at all related to the one in the repository.

Judging from the contents of the README, it seems to me like someone squatted this crate name, it was freed up at some point, but some metadata of the old version is still hanging out in some cache?

Expected Behavior

I expected to see the readme from my github repo (or nothing if I messed up the configuration in my Cargo.toml)

Steps To Reproduce

No response

Environment

• Browser:
• OS:

Anything else?

No response

[toolCHAINZ]

Maybe related to #10887?

[Turbo87]

my best guess of what happened is:

• the crackers crate was deleted by our support team before we were
  • tracking deleted crate names and
  • deleting associated files from S3 (instead of only the records from the database)
• this left the rendered readme (and crate file) on S3
• you published the crackers crate without a readme at first
• since your crate contained no readme the previous rendered readme was not overwritten (but the crate file was correctly overwritten)
• you fixed the crate readme via toolCHAINZ/crackers@4940d25 so v0.1.2 contains the correct readme

since these days we delete all related files from S3 when a crate is deleted, this shouldn't happen anymore. to fix this particular case we could manually delete the problematic readme from S3.

any thoughts?

[LawnGnome]

any thoughts?

That feels like what happened to me as well. I think deleting the file from S3 and invalidating the CDN caches should fix this.

[toolCHAINZ]

Agreed, the explanation makes sense, so clearing the cache seems reasonable? I wonder how many crate names there are potentially with this problem? Good to know that it's only very old squatted names.

[LawnGnome]

OK, so it turns out that we have 993 READMEs that are dangling, of which 32 would be the incorrect, active README for an actual crate version published on crates.io today. In practice, most — but definitely not all — of these are simple cases where a crate version was deleted long ago and the author then re-published the same crate version with whatever fix they wanted.

Unsurprisingly, the newest of those READMEs is about two years old, which corresponds nicely with when we improved the deletion worker job to clean up S3 more effectively.

What this also means is that we have up to 961 crate files in S3 that are also supposed to be inaccessible (total versions - versions that have since been re-published).

So, my proposed course of action (cc: @Turbo87 in particular, and @rust-lang/crates-io in general):

1. Delete all 993 READMEs.
2. Delete all 961 crate files, excluding the 32 (crate, version) pairs that correspond to currently published crate versions.
3. Invalidate all those paths in the CDNs.

Any objections?

(I'm going to keep the list of (crate, version) pairs private for now — I think it would take an imaginative adversary to figure out how to weaponise this somehow for a supply chain attack, but just in case, let's keep it under our hats until we've finished our mitigations.)

[Turbo87]

sounds good to me. I thought we had already done that in the past though, so I'm a bit surprised that the number is that high 🤔