Update storage limits documentation to reflect new file size recommendations #2037
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| hub_limits: | ||
| max_file_size_gb: 50 # absolute hard stop enforced by LFS | ||
| recommended_file_size_gb: 20 # best-practice shard size | ||
| max_file_size_gb: 200 # absolute hard stop enforced by LFS |
There was a problem hiding this comment.
Good catch! I've just checked and we still reference LFS in quite a few places, so I'll make a separate PR to tidy those.
There was a problem hiding this comment.
in the product/platform wordings we moved to "Large Files". cc @Pierrci our terminology expert
| - **File size**: In the case of uploading large files (e.g. model weights), we strongly recommend splitting them **into chunks of around 50GB each**. | ||
| There are a few reasons for this: | ||
| - Uploading and downloading smaller files is much easier both for you and the other users. Connection issues can always | ||
| happen when streaming data and smaller files avoid resuming from the beginning in case of errors. |
There was a problem hiding this comment.
but also update this part no?
In all cases no single LFS file will be able to be >50GB. I.e. 50GB is the hard limit for single file size.
There was a problem hiding this comment.
(i think we said new file limit= 200GB)
There was a problem hiding this comment.
Maybe:
In the case of uploading large files (e.g. model weights), we strongly recommend splitting them into chunks <200GB each.
works?
Update storage size recommendations