From d569693552b351d8d1b3b0eaf51194a9fe1938eb Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Wed, 21 May 2025 14:31:00 -0400 Subject: [PATCH 01/33] PEP 694: Abstract file upload mechanisms --- peps/pep-0694.rst | 415 +++++++++++----------------------------------- 1 file changed, 94 insertions(+), 321 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index db2ef5fdb24..8ff8647a740 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -24,7 +24,7 @@ with standardization, the upload API provides additional useful features such as * artifacts which can be overwritten and replaced, until a session is published; -* asynchronous and "chunked", resumable file uploads, for more efficient use of network bandwidth; +* flexible file upload mechanisms for index operators; * detailed status on the state of artifact uploads; @@ -49,10 +49,10 @@ In addition, there are a number of major issues with the legacy API: * It is fully synchronous, which forces requests to be held open both for the upload itself, and while the index processes the uploaded file to determine success or failure. -* It does not support any mechanism for resuming an upload. With the largest default file size on - PyPI being around 1GB in size, requiring the entire upload to complete successfully means - bandwidth is wasted when such uploads experience a network interruption while the request is in - progress. +* It does not support any mechanism for parallelizing or resuming an upload. With the largest + default file size on PyPI being around 1GB in size, requiring the entire upload to complete + successfully means bandwidth is wasted when such uploads experience a network interruption while + the request is in progress. * The atomic unit of operation is a single file. This is problematic when a release logically includes an sdist and multiple binary wheels, leading to race conditions where consumers get @@ -78,7 +78,7 @@ In addition, there are a number of major issues with the legacy API: to claim a project namespace. The new upload API proposed in this PEP solves all of these problems, providing for a much more -flexible, bandwidth friendly approach, with better error reporting, a better release testing +flexible approach, with better error reporting, a better release testing experience, and atomic and simultaneous publishing of all release artifacts. @@ -139,11 +139,6 @@ methods. Upload 2.0 API Specification ============================ -This PEP draws inspiration from the `Resumable Uploads for HTTP `_ internet draft, -however there are significant differences. This is largely due to the unique nature of Python -package releases (i.e. metadata, multiple related artifacts, etc.), and the support for an upload -session and release stages. Where it makes sense to adopt details of the draft, this PEP does so. - This PEP traces the root cause of most of the issues with the existing API to be roughly two things: - The metadata is submitted alongside the file, rather than being parsed from the @@ -156,7 +151,8 @@ To address these issues, this PEP proposes a multi-request workflow, which at a these steps: #. Initiate an upload session, creating a release stage. -#. Upload the file(s) to that stage as part of the upload session. +#. Initiate file-upload session(s) to that stage as part of the upload session. +#. Complete the file-upload session(s). #. Complete the upload session, publishing or discarding the stage. #. Optionally check the status of an upload session. @@ -183,6 +179,8 @@ Specifically for PyPI, this PEP proposes to implement the root endpoint at being tested, and will be blessed as permanent after sufficient testing with live projects. + + .. _session-create: Create an Upload Session @@ -308,8 +306,8 @@ Session Links For the ``links`` key in the success JSON, the following sub-keys are valid: ``upload`` - The endpoint session clients will use to initiate :ref:`uploads ` for each file to - be included in this session. + The endpoint session clients will use to initiate a :ref:`file-upload session ` + for each file to be included in this session. ``stage`` The endpoint where this staged release can be :ref:`previewed ` prior to @@ -379,7 +377,8 @@ request body has the following JSON format: "filename": "foo-1.0.tar.gz", "size": 1000, "hashes": {"sha256": "...", "blake2b": "..."}, - "metadata": "..." + "metadata": "...", + "mechanisms": ["http-post-application-octet-stream"] } @@ -406,6 +405,10 @@ Besides the standard ``meta`` key, the request JSON has the following additional If given, this is a string value containing the file's `core metadata `_. +``mechanisms`` (**optional**) + If given this string value contains the file-upload mechanisms supported by the client. + The server **MAY** use this list to inform the contents of ``mechanisms`` in its response. + Servers **MAY** use the data provided in this request to do some sanity checking prior to allowing the file to be uploaded. These checks may include, but are not limited to: @@ -415,195 +418,78 @@ the file to be uploaded. These checks may include, but are not limited to: - checking if the contents of the ``metadata``, if provided, are valid. -If the server determines that upload should proceed, it will return a ``201 Created`` response, with -an empty body, and a ``Location`` header pointing to the URL that the file content should be -uploaded to. The :ref:`status ` of the session will also include the filename in -the ``files`` mapping, with the above ``Location`` URL included in under the ``link`` sub-key. -If the server determines the upload cannot proceed, it **MUST** return a ``409 Conflict``. The -server **MAY** allow parallel uploads of files, but is not required to. - - -.. IMPORTANT:: - - The `IETF draft `_ calls this the URL of the `upload resource - `_, and this PEP uses that nomenclature as well. - -.. _ietf-upload-resource: https://www.ietf.org/archive/id/draft-ietf-httpbis-resumable-upload-05.html#name-upload-creation-2 - - -.. _upload-contents: - -Upload File Contents -++++++++++++++++++++ - -The actual file contents are uploaded by issuing a ``POST`` request to the upload resource URL -[#fn-location]_. The client may either upload the entire file in a single request, or it may opt -for "chunked" upload where the file contents are split into multiple requests, as described below. - -.. IMPORTANT:: - - The protocol defined in this PEP differs from the `IETF draft `_ in a few ways: - - * For chunked uploads, the `second and subsequent chunks `_ are uploaded - using a ``POST`` request instead of ``PATCH`` requests. Similarly, this PEP uses - ``application/octet-stream`` for the ``Content-Type`` headers for all chunks. - - * No ``Upload-Draft-Interop-Version`` header is required. - - * Some of the server responses are different. - -.. _ietf-upload-append: https://www.ietf.org/archive/id/draft-ietf-httpbis-resumable-upload-05.html#name-upload-append-2 - - -When uploading the entire file in a single request, the request **MUST** include the following -headers (e.g. for a 100,000 byte file): - -.. code-block:: email - - Content-Length: 100000 - Content-Type: application/octet-stream - Upload-Length: 100000 - Upload-Complete: ?1 - -The body of this request contains all 100,000 bytes of the unencoded raw binary data. - -``Content-Length`` - The number of file bytes contained in the body of *this* request. - -``Content-Type`` - **MUST** be ``application/octet-stream``. - -``Upload-Length`` - Indicates the total number of bytes that will be uploaded for this file. For single-request - uploads this will always be equal to ``Content-Length``, but these values will likely differ for - chunked uploads. This value **MUST** equal the number of bytes given in the ``size`` field of - the file upload initiation request. +If the server determines that upload should proceed, it will return a ``202 Accepted`` response, with +the response body below. The :ref:`status ` of the session will also include the filename in the ``files`` mapping. If the server determines the upload cannot proceed, it **MUST** return +a ``409 Conflict``. The server **MAY** allow parallel uploads of files, but is not required to. -``Upload-Complete`` - A flag indicating whether more chunks are coming for this file. For single-request uploads, the - value of this header **MUST** be ``?1``. +.. _upload-session-response: -If the upload completes successfully, the server **MUST** respond with a ``201 Created`` status. -The response body has no content. - -If this single-request upload fails, the entire file must be resent in another single HTTP request. -This is the recommended, preferred format for file uploads since fewer requests are required. - -As an example, if the client was to upload a 100,000 byte file, the headers would look like: - -.. code-block:: email - - Content-Length: 100000 - Content-Type: application/octet-stream - Upload-Length: 100000 - Upload-Complete: ?1 - -Clients can opt to upload the file in multiple chunks. Because the upload resource URL provided in -the metadata response will be unique per file, clients **MUST** use the given upload resource URL -for all chunks. Clients upload file chunks by sending multiple ``POST`` requests to this URL, with -one request per chunk. - -For chunked uploads, the ``Content-Length`` is equal to the size in bytes of the chunk that is -currently being sent. The client **MUST** include a ``Upload-Offset`` header which indicates the -byte offset that the content included in this chunk's request starts at, and an ``Upload-Complete`` -header with the value ``?0``. For the first chunk, the ``Upload-Offset`` header **MUST** be set to -``0``. As with single-request uploads, the ``Content-Type`` header is ``application/octet-stream`` -and the body is the raw, unencoded bytes of the chunk. - -For example, if uploading a 100,000 byte file in 1000 byte chunks, the first chunk's request headers -would be: - -.. code-block:: email - - Content-Length: 1000 - Content-Type: application/octet-stream - Upload-Offset: 0 - Upload-Length: 100000 - Upload-Complete: ?0 - -For the second chunk representing bytes 1000 through 1999, include the following headers: - -.. code-block:: email - - Content-Length: 1000 - Content-Type: application/octet-stream - Upload-Offset: 1000 - Upload-Length: 100000 - Upload-Complete: ?0 - -These requests would continue sequentially until the last chunk is ready to be uploaded. - -For each successful chunk, the server **MUST** respond with a ``202 Accepted`` header, except for -the final chunk, which **MUST** be either: - -* ``201 Created`` if the server accepts and processes the last chunk synchronously, completing the - file upload. -* ``202 Accepted`` if the server accepts the last chunk, but must process it asynchronously. In - this case, the client should query the :ref:`session status ` periodically until - the uploaded :ref:`file status ` transitions to ``complete``. +Response Body ++++++++++++++ -The final chunk of data **MUST** include the ``Upload-Complete: ?1`` header, since at that point the -entire file has been uploaded. +The successful response includes the following JSON content: -With both chunked and non-chunked uploads, once completed successfully, the file **MUST NOT** be -publicly visible in the repository, but merely staged until the upload session is :ref:`completed -`. If the server supports :ref:`previews `, the file **MUST** be -visible at the ``stage`` :ref:`URL `. Partially uploaded chunked files **SHOULD -NOT** be visible at the ``stage`` URL. +.. code-block:: json -The following constraints are placed on uploads regardless of whether they are single chunk or -multiple chunks: + { + "meta": { + "api-version": "2.0" + }, + "links": { + "file-upload-session": "..." + }, + "upload-session-id": "", + "mechanisms": { + "http-post-application-octet-stream": { + "url": "..." + } + } + } -- A client **MUST NOT** perform multiple ``POST`` requests in parallel for the same file to avoid - race conditions and data loss or corruption. -- If the offset provided in ``Upload-Offset`` is not ``0`` and does not correctly specify the byte - offset of the next chunk in an incomplete upload, then the server **MUST** respond with a ``409 - Conflict``. This means that a client **MUST NOT** upload chunks out of order. +Besides the ``meta`` key, which has the same format as the request JSON, the success response has +the following keys: -- Once a file upload has completed successfully, you may initiate another upload for that file, - which **once completed**, will replace that file. This is possible until the entire session is - completed, at which point no further file uploads (either creating or replacing a session file) - are accepted. I.e. once a session is published, the files included in that release are immutable - [#fn-immutable]_. +``upload-session-id`` + The unique file-upload session identifier. +``mechanisms`` + A mapping containing the mechanism identifier that the server supports, to a mapping + containing details necessary to execute each mechanism. -Resume an Upload -++++++++++++++++ -To resume an upload, you first have to know how much of the file's contents the server has already -received. If this is not already known, a client can make a ``HEAD`` request to the upload resource -URL. +File Upload Session Completion +++++++++++++++++++++++++++++++ -The server **MUST** respond with a ``204 No Content`` response, with an ``Upload-Offset`` header -that indicates what offset the client should continue uploading from. If the server has not received -any data, then this would be ``0``, if it has received 1007 bytes then it would be ``1007``. For -this example, the full response headers would look like: +To complete a file upload session, which indicates that the file upload mechanism has been executed +and did not produce an error, a client issues a ``POST`` to the ``file-upload-session`` link in the +file upload session creation response body. -.. code-block:: email +The JSON body of this requests looks like: - Upload-Offset: 1007 - Upload-Complete: ?0 - Cache-Control: no-store +.. code-block:: json + { + "meta": { + "api-version": "2.0" + }, + "action": "complete", + } -Once the client has retrieved the offset that they need to start from, they can upload the rest of -the file as described above, either in a single request containing all of the remaining bytes, or in -multiple chunks as per the above protocol. -.. _cancel-an-upload: Canceling and Deleting File Uploads +++++++++++++++++++++++++++++++++++ -A client can cancel an in-progress upload for a file, or delete a file that has been completely -uploaded. In both cases, the client performs this by issuing a ``DELETE`` request to the upload -resource URL of the file they want to delete. +A client can cancel an in-progress upload session for a file, or delete a file that has been +completely uploaded. In both cases, the client performs this by issuing a ``DELETE`` request to +the file upload session URL of the file they want to delete. A successful deletion request **MUST** response with a ``204 No Content``. -Once canceled or deleted, a client **MUST NOT** assume that the previous upload resource URL can be reused. +Once canceled or deleted, a client **MUST NOT** assume that the previous file upload session resource +or associated file upload mechanisms can be reused. Replacing a Partially or Fully Uploaded File @@ -652,7 +538,7 @@ The JSON body of this request looks like: "meta": { "api-version": "2.0" }, - ":action": "extend", + "action": "extend", "extend-for": 3600 } @@ -702,22 +588,18 @@ The JSON body of this request looks like: "meta": { "api-version": "2.0" }, - ":action": "publish", + "action": "publish", } -If the server is able to immediately complete the session, it may do so and return a ``201 Created`` -response. If it is unable to immediately complete the session (for instance, if it needs to do -processing that may take longer than reasonable in a single HTTP request), then it may return a -``202 Accepted`` response. - -In either case, the server should include a ``Location`` header pointing back to the session status -URL, and if the server returned a ``202 Accepted``, the client may poll that URL to watch for the -status to change. +If the server is able to immediately complete the file upload session, it may do so and return a +``201 Created`` response. If it is unable to immediately complete the file upload session +(for instance, if it needs to do validation that may take longer than reasonable in a single HTTP +request), then it may return a ``202 Accepted`` response. -If a session is published that has no staged files, the operation is effectively a no-op, except -where a new project name is being reserved. In this case, the new project is created, reserved, and -owned by the user that created the session. +In either case, the server should include a ``Location`` header pointing back to the file upload +session status URL, and if the server returned a ``202 Accepted``, the client may poll that URL to +watch for the status to change. If an error occurs, the appropriate ``4xx`` code should be returned, as described in the :ref:`session-errors` section. @@ -781,15 +663,6 @@ URL can be passed to installers such as ``pip`` by setting the `--extra-index-ur `__ flag to this value. Multiple stages can even be previewed by repeating this flag with multiple values. -In the future, it may be valuable to include something like a ``Stage-Token`` header to the `Simple -Repository API `_ -requests or the :pep:`691` JSON-based Simple API, with the value from the ``session-token`` sub-key -of the JSON response to the session creation request. Multiple ``Stage-Token`` headers could be -allowed, and installers could support enabling stage previews by adding a ``--staged `` or -similarly named option to set the ``Stage-Token`` header at the command line. This feature is not -currently support, nor proposed by this PEP, though it could be proposed by a separate PEP in the -future. - In either case, the index will return views that expose the staged releases to the installer tool, making them available to download and install into virtual environments built for that last-mile testing. The former option allows for existing installers to preview staged releases with no @@ -832,6 +705,24 @@ Besides the standard ``meta`` key, this has the following top level keys: The ``message`` and ``source`` strings do not have any specific meaning, and are intended for human interpretation to aid in diagnosing underlying issue. +File Upload Mechanisms +---------------------- + +File Upload Mechanisms, with the exception of ``http-post-application-octet-stream`` are left as an +implementation detail specific to each server. Servers **MUST** implement a +``http-post-application-octet-stream`` mechanism as a fallback if no server specific implementations +exist. + +A given server may implement an arbitrary number of mechanisms and is responsible for documenting +their usage. Implemenatations **SHOULD** be prefixed with a string that clearly identifies the +server and is unique from other well known servers or implementations. + +If a server intendes to match the behavior of another server's implementation, it **MAY** respond +with that implementation's file upload mechanism name. + +All implementations of this PEP **MUST** implement the ``http-post-application-octet-stream`` file +upload mechanism. + Content Types ------------- @@ -840,8 +731,8 @@ Like :pep:`691`, this PEP proposes that all requests and responses from this upl standard content type that describes what the content is, what version of the API it represents, and what serialization format has been used. -This standard request content type applies to all requests *except* for :ref:`file upload requests -` which, since they contain only binary data, is always ``application/octet-stream``. +This standard request content type applies to all requests *except* for requests to execute +a File Upload Mechanism, which will be specified by the documentation for that mechanism. The structure of the ``Content-Type`` header for all other requests is: @@ -888,36 +779,6 @@ the legacy upload API to be (responsibly) deprecated and removed at some point i future deprecation planning is explicitly out of scope for *this* PEP. -Is this Resumable Upload protocol based on anything? ----------------------------------------------------- - -Yes! - -It's actually based on the protocol specified in an `active internet draft `_, where the -authors took what they learned implementing `tus `_ to provide the idea of -resumable uploads in a wholly generic, standards based way. - -.. _ietf-draft: https://www.ietf.org/archive/id/draft-ietf-httpbis-resumable-upload-05.html - -This PEP deviates from that spec in several ways, as described in the body of the proposal. This -decision was made for a few reasons: - -- The ``104 Upload Resumption Supported`` is the only part of that draft which does not rely - entirely on things that are already supported in the existing standards, since it was adding a new - informational status. - -- Many clients and web frameworks don't support ``1xx`` informational responses in a very good way, - if at all, adding it would complicate implementation for very little benefit. - -- The purpose of the ``104 Upload Resumption Supported`` support is to allow clients to determine - that an arbitrary endpoint that they're interacting with supports resumable uploads. Since this - PEP is mandating support for that in servers, clients can just assume that the server they are - interacting with supports it, which makes using it unneeded. - -- In theory, if the support for ``1xx`` responses got resolved and the draft gets accepted with it - in, we can add that in at a later date without changing the overall flow of the API. - - Can I use the upload 2.0 API to reserve a project name? ------------------------------------------------------- @@ -945,90 +806,6 @@ However, the ability to preview stages before they're published does complicate this proposal. We could defer this feature for later, although if we do, we should still keep the optional ``nonce`` for token generation, in order to be easily future proof. - -Multipart Uploads vs tus ------------------------- - -This PEP currently bases the actual uploading of files on an `internet draft `_ -(originally designed by `tus.io `__) that supports resumable file uploads. - -That protocol requires a few things: - -- That if clients don't upload the entire file in one shot, that they have to submit the chunks - serially, and in the correct order, with all but the final chunk having a ``Upload-Complete: ?0`` - header. - -- Resumption of an upload is essentially just querying the server to see how much data they've - gotten, then sending the remaining bytes (either as a single request, or in chunks). - -- The upload implicitly is completed when the server successfully gets all of the data from the - client. - -This has the benefit that if a client doesn't care about resuming their download, it can essentially -ignore the protocol. Clients can just ``POST`` the file to the file upload URL, and if it doesn't -succeed, they can just ``POST`` the whole file again. - -The other benefit is that even if clients do want to support resumption, unless they *need* to -resume the download, they can still just ``POST`` the file. - -Another, possibly theoretical benefit is that for hashing the uploaded files, the serial chunks -requirement means that the server can maintain hashing state between requests, update it for each -request, then write that file back to storage. Unfortunately this isn't actually possible to do with -Python's `hashlib `__ standard library module. -There are some libraries third party libraries, such as `Rehash -`__ that do implement the necessary APIs, but they don't -support every hash that ``hashlib`` does (e.g. ``blake2`` or ``sha3`` at the time of writing). - -We might also need to reconstitute the download for processing anyways to do things like extract -metadata, etc from it, which would make it a moot point. - -The downside is that there is no ability to parallelize the upload of a single file because each -chunk has to be submitted serially. - -AWS S3 has a similar API, and most blob stores have copied it either wholesale or something like it -which they call multipart uploading. - -The basic flow for a multipart upload is: - -#. Initiate a multipart upload to get an upload ID. -#. Break your file up into chunks, and upload each one of them individually. -#. Once all chunks have been uploaded, finalize the upload. This is the step where any errors would - occur. - -Such multipart uploads do not directly support resuming an upload, but it allows clients to control -the "blast radius" of failure by adjusting the size of each part they upload, and if any of the -parts fail, they only have to resend those specific parts. The trade-off is that it allows for more -parallelism when uploading a single file, allowing clients to maximize their bandwidth using -multiple threads to send the file data. - -We wouldn't need an explicit step (1), because our session would implicitly initiate a multipart -upload for each file. - -There are downsides to this though: - -- Clients have to do more work on every request to have something resembling resumable uploads. They - would *have* to break the file up into multiple parts rather than just making a single POST - request, and only needing to deal with the complexity if something fails. - -- Clients that don't care about resumption at all still have to deal with the third explicit step, - though they could just upload the file all as a single part. (S3 works around this by having - another API for one shot uploads, but the PEP authors place a high value on having a single API - for uploading any individual file.) - -- Verifying hashes gets somewhat more complicated. AWS implements hashing multipart uploads by - hashing each part, then the overall hash is just a hash of those hashes, not of the content - itself. Since PyPI needs to know the actual hash of the file itself anyway, we would have to - reconstitute the file, read its content, and hash it once it's been fully uploaded, though it - could still use the hash of hashes trick for checksumming the upload itself. - -The PEP authors lean towards ``tus`` style resumable uploads, due to them being simpler to use, -easier to imp;lement, and more consistent, with the main downside being that multi-threaded -performance is theoretically left on the table. - -One other possible benefit of the S3 style multipart uploads is that you don't have to try and do -any sort of protection against parallel uploads, since they're just supported. That alone might -erase most of the server side implementation simplification. - .. rubric:: Footnotes .. [#fn-action] Obsolete ``:action`` values ``submit``, ``submit_pkg_info``, and ``doc_upload`` are @@ -1046,10 +823,6 @@ erase most of the server side implementation simplification. .. [#fn-immutable] Published files may still be yanked (i.e. :pep:`592`) or `deleted `__ as normal. -.. [#fn-location] Or the URL given in the ``Location`` header in the response to the file upload - initiation request, i.e. the metadata upload request; both of these links **MUST** - be the same. - Copyright ========= From 8e98c9b23d2574d1aa4727ef42e3ec0489a35b0c Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Wed, 21 May 2025 16:18:05 -0400 Subject: [PATCH 02/33] lint --- peps/pep-0694.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 8ff8647a740..7897b6e700c 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -469,7 +469,7 @@ The JSON body of this requests looks like: .. code-block:: json - { + { "meta": { "api-version": "2.0" }, From daf06f47b28201a06d9bff68738d715197b1c23d Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Wed, 21 May 2025 16:22:25 -0400 Subject: [PATCH 03/33] no warnings allowed --- peps/pep-0694.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 7897b6e700c..1b96e387686 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -477,7 +477,7 @@ The JSON body of this requests looks like: } - +.. _cancel-an-upload: Canceling and Deleting File Uploads +++++++++++++++++++++++++++++++++++ From 5ef177e590973f9ed8680e113c992069901ff0b7 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Thu, 22 May 2025 08:51:17 -0400 Subject: [PATCH 04/33] refinements --- peps/pep-0694.rst | 104 +++++++++++++++++++++++++++++++--------------- 1 file changed, 70 insertions(+), 34 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 1b96e387686..5f9fc36248d 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -152,7 +152,8 @@ these steps: #. Initiate an upload session, creating a release stage. #. Initiate file-upload session(s) to that stage as part of the upload session. -#. Complete the file-upload session(s). +#. Execute file upload mechanism for the file-upload session(s). +#. Complete the file-upload session(s), marking them as executed or canceled. #. Complete the upload session, publishing or discarding the stage. #. Optionally check the status of an upload session. @@ -179,8 +180,6 @@ Specifically for PyPI, this PEP proposes to implement the root endpoint at being tested, and will be blessed as permanent after sufficient testing with live projects. - - .. _session-create: Create an Upload Session @@ -251,6 +250,7 @@ The successful response includes the following JSON content: "upload": "...", "session": "...", }, + "mechanisms": ["http-post-application-octet-stream"], "session-token": "", "valid-for": 604800, "status": "pending", @@ -268,6 +268,9 @@ the following keys: A dictionary mapping :ref:`keys to URLs ` related to this session, the details of which are provided below. +``mechanisms`` + A list of file-upload mechanisms supported by the server. + ``session-token`` If the index supports :ref:`previewing staged releases `, this key will contain the unique :ref:`"session token" ` that can be provided to installers in order to @@ -330,13 +333,9 @@ The ``files`` key contains a mapping from the names of the files uploaded in thi sub-mapping with the following keys: ``status`` - A string with valid values ``partial``, ``pending``, ``complete``, and ``error``. If a file - upload has not seen an ``Upload-Complete: ?1`` header, then ``partial`` will be returned. If - ``Upload-Complete: ?1`` resulted in a ``202 Accepted``, then ``pending`` will be returned until - asynchronous processing of the last chunk and the full file has been completed. If a ``201 - Created`` was returned, or the last chunk processing is finished, ``complete`` will be returned. + A string with valid values ``pending``, ``processing``, ``complete``, ``error``, and ``canceled``. If there was an error during upload, then clients should not assume the file is in any usable - state, ``error`` will be returned and it's best to :ref:`cancel or delete ` + state, ``error`` will be returned and it's best to :ref:`cancel or delete ` the file and start over. This action would remove the file name from the ``files`` key of the :ref:`session status response body `. @@ -378,7 +377,7 @@ request body has the following JSON format: "size": 1000, "hashes": {"sha256": "...", "blake2b": "..."}, "metadata": "...", - "mechanisms": ["http-post-application-octet-stream"] + "mechanism": "http-post-application-octet-stream" } @@ -401,14 +400,17 @@ Besides the standard ``meta`` key, the request JSON has the following additional Multiple hashes may be passed at a time, but all hashes provided **MUST** be valid for the file. +``mechanism`` (**required**) + The file-upload mechanisms the client intends to use for this file. + This mechanism **SHOULD** be chosen from the list of mechanisms advertised in the `session response body + `_. + A client **MAY** send a mechanism that is not advertised in cases where server operators have + documented a new or up-coming mechanism that is available for use on a "pre-release" basis. + ``metadata`` (**optional**) If given, this is a string value containing the file's `core metadata `_. -``mechanisms`` (**optional**) - If given this string value contains the file-upload mechanisms supported by the client. - The server **MAY** use this list to inform the contents of ``mechanisms`` in its response. - Servers **MAY** use the data provided in this request to do some sanity checking prior to allowing the file to be uploaded. These checks may include, but are not limited to: @@ -421,11 +423,13 @@ the file to be uploaded. These checks may include, but are not limited to: If the server determines that upload should proceed, it will return a ``202 Accepted`` response, with the response body below. The :ref:`status ` of the session will also include the filename in the ``files`` mapping. If the server determines the upload cannot proceed, it **MUST** return a ``409 Conflict``. The server **MAY** allow parallel uploads of files, but is not required to. +If the server cannot proceed with an upload because the ``mechanism`` supplied by the client is not supported +it **MUST** return a ``422 Unprocessable Entity``. -.. _upload-session-response: +.. _file-upload-session-response: -Response Body -+++++++++++++ +File Upload Session Response Body ++++++++++++++++++++++++++++++++++ The successful response includes the following JSON content: @@ -436,10 +440,12 @@ The successful response includes the following JSON content: "api-version": "2.0" }, "links": { + "session": "...", "file-upload-session": "..." }, - "upload-session-id": "", - "mechanisms": { + "status": "pending", + "valid-for": 3600, + "mechanism": { "http-post-application-octet-stream": { "url": "..." } @@ -450,13 +456,32 @@ The successful response includes the following JSON content: Besides the ``meta`` key, which has the same format as the request JSON, the success response has the following keys: -``upload-session-id`` - The unique file-upload session identifier. +``links`` + A dictionary mapping :ref:`keys to URLs ` related to this session, + the details of which are provided below. -``mechanisms`` - A mapping containing the mechanism identifier that the server supports, to a mapping - containing details necessary to execute each mechanism. +``mechanism`` + A mapping containing the supported mechanism identifier negotiated by the client and server, + to a mapping containing details necessary to execute the mechanism. +.. _file-upload-session-links: + +Session Links ++++++++++++++ + +For the ``links`` key in the success JSON, the following sub-keys are valid: + +``session`` + The endpoint where actions for the parent session can be performed. + +``file-upload-session`` + The endpoint where actions for this file-upload-session can be performed. + including :ref:`canceling and discarding the file upload session `, + :ref:`querying the current file upload session status `, + and :ref:`requesting an extension of the file upload session lifetime ` + (*if* the server supports it). + +.. _file-upload-session-completion: File Upload Session Completion ++++++++++++++++++++++++++++++ @@ -477,7 +502,14 @@ The JSON body of this requests looks like: } -.. _cancel-an-upload: +After receiving this requests the server **MAY** perform additional asynchronous processing on the file, +for instance to verify its hashes or contents. +If the processing is required to complete before an upload session can be published, +the status of the file upload session can be set to ``processing`` until such processing is complete, +reaches an error state, or the file upload session is canceled. + + +.. _file-upload-session-cancelation: Canceling and Deleting File Uploads +++++++++++++++++++++++++++++++++++ @@ -499,7 +531,7 @@ To replace a session file, the file upload **MUST** have been previously complet deleted. It is not possible to replace a file if the upload for that file is in-progress. To replace a session file, clients should :ref:`cancel and delete the in-progress upload -` by issuing a ``DELETE`` to the upload resource URL for the file they want to +` by issuing a ``DELETE`` to the upload resource URL for the file they want to replace. After this, the new file upload can be initiated by beginning the entire :ref:`file upload ` sequence over again. This means providing the metadata request again to retrieve a new upload resource URL. Client **MUST NOT** assume that the previous upload resource URL can be @@ -511,13 +543,16 @@ reused after deletion. Session Status ~~~~~~~~~~~~~~ -At any time, a client can query the status of the session by issuing a ``GET`` request to the -``session`` :ref:`link ` given in the :ref:`session creation response body -`. +At any time, a client can query the status of a session by issuing a ``GET`` request to the +``session`` :ref:`link ` or ``file-upload-session`` :ref:`link ` +given in the :ref:`session creation response body ` +or :ref:`file upload session creation response body `, +respectively. -The server will respond to this ``GET`` request with the same :ref:`response ` -that they got when they initially created the upload session, except with any changes to ``status``, -``valid-for``, or ``files`` reflected. +The server will respond to this ``GET`` request with the same :ref:`session response ` +or :ref:`file upload session creation response body `, +that they got when they initially created the upload session or file upload session, +except with any changes to ``status``, ``valid-for``, or ``files`` reflected. .. _session-extension: @@ -527,8 +562,9 @@ Session Extension Servers **MAY** allow clients to extend sessions, but the overall lifetime and number of extensions allowed is left to the server. To extend a session, a client issues a ``POST`` request to the -``session`` :ref:`link ` given in the :ref:`session creation response body -`. +``session`` :ref:`link ` or ``file-upload-session`` :ref:`link ` +given in the :ref:`session creation response body ` +or :ref:`file upload session creation response body `, respectively. The JSON body of this request looks like: From 7f1e741e69d327a3f7cca4e87825690981e992dc Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Wed, 28 May 2025 14:03:40 -0400 Subject: [PATCH 05/33] Update peps/pep-0694.rst --- peps/pep-0694.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 5f9fc36248d..c6f95405c94 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -746,7 +746,7 @@ File Upload Mechanisms File Upload Mechanisms, with the exception of ``http-post-application-octet-stream`` are left as an implementation detail specific to each server. Servers **MUST** implement a -``http-post-application-octet-stream`` mechanism as a fallback if no server specific implementations +``http-post-application-octet-stream`` mechanism. This serves as a fallback if no server specific implementations exist. A given server may implement an arbitrary number of mechanisms and is responsible for documenting From f8469cffcdae89afdfa7d303cda45d7cb488c9fa Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 30 May 2025 06:29:32 -0400 Subject: [PATCH 06/33] remove reference to a specific URL for PyPI --- peps/pep-0694.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index c6f95405c94..13eeb70167d 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -175,11 +175,6 @@ All URLs described here are relative to the "root endpoint", which may be locate the url structure of a domain. For example, the root endpoint could be ``https://upload.example.com/``, or ``https://example.com/upload/``. -Specifically for PyPI, this PEP proposes to implement the root endpoint at -``https://upload.pypi.org/2.0``. This root URL will be considered provisional while the feature is -being tested, and will be blessed as permanent after sufficient testing with live projects. - - .. _session-create: Create an Upload Session From 05d7fc202b7b967d8bdff20dcf96159f6896e5dd Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 30 May 2025 06:38:12 -0400 Subject: [PATCH 07/33] clarify that resumable/parallel uploads are supported but not defined in this pep --- peps/pep-0694.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 13eeb70167d..4e5534ac8d5 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -77,9 +77,11 @@ In addition, there are a number of major issues with the legacy API: * Creation of new projects requires the uploading of at least one file, leading to "stub" uploads to claim a project namespace. -The new upload API proposed in this PEP solves all of these problems, providing for a much more -flexible approach, with better error reporting, a better release testing -experience, and atomic and simultaneous publishing of all release artifacts. +The new upload API proposed in this PEP provides a solution to all of these problems, +providing for a much more flexible approach, with support for servers to +implement resumable and parallel uploads via mechanisms, +better error reporting, better release testing experience, +and atomic and simultaneous publishing of all release artifacts. Legacy API From 2ef077cd355941faee8412b7eb6163fc1e894cf2 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 30 May 2025 07:33:38 -0400 Subject: [PATCH 08/33] attempt to specify http-post-application-octet-stream mechanism --- peps/pep-0694.rst | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 4e5534ac8d5..262d026b35c 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -444,7 +444,8 @@ The successful response includes the following JSON content: "valid-for": 3600, "mechanism": { "http-post-application-octet-stream": { - "url": "..." + "file_url": "..." + "attestations_url": "..." } } } @@ -756,6 +757,36 @@ with that implementation's file upload mechanism name. All implementations of this PEP **MUST** implement the ``http-post-application-octet-stream`` file upload mechanism. +``http-post-application-octet-stream`` Mechanism +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``http-post-application-octet-stream`` mechansism **MUST** be supported servers which +implement this PEP. + +A client executes this mechanism by submitting a ``POST`` request to the ``file_url`` returned in the +``http-post-application-octet-stream`` map of the ``mechanism`` map of the +:ref:`file upload session creation response body ` like: + +.. code-block:: text + + Content-Type: application/octet-stream + + + +Servers **MAY** support uploading of digital attestations for files (see :pep:`740`). +This support will be indicated by inclusion of an ``attestations_url`` key in the +``http-post-application-octet-stream`` map of the ``mechanism`` map of the +:ref:`file upload session creation response body `. + +To upload an attestation, a client submits a ``POST`` request to the ``attestations_url`` +containing a JSON array of :pep:`attestation objects <740#attestation-objects>` like: + +.. code-block:: text + + Content-Type: application/json + + [{"version": 1, "verification_material": {...}, "envelope": {...}},...] + Content Types ------------- From a2bfb56edd828349a48ed3a6a00a3ddc300501fa Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 30 May 2025 07:40:29 -0400 Subject: [PATCH 09/33] if attestations are going to be uploaded, do it before completion of file upload session --- peps/pep-0694.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 262d026b35c..279e7ca667f 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -777,6 +777,8 @@ Servers **MAY** support uploading of digital attestations for files (see :pep:`7 This support will be indicated by inclusion of an ``attestations_url`` key in the ``http-post-application-octet-stream`` map of the ``mechanism`` map of the :ref:`file upload session creation response body `. +Attestations **MUST** be uploaded to the ``attestations_url`` before +:ref:`file upload session completion `. To upload an attestation, a client submits a ``POST`` request to the ``attestations_url`` containing a JSON array of :pep:`attestation objects <740#attestation-objects>` like: From 442d37f9923f591ce70e450a2c266e024d7be275 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 30 May 2025 07:40:57 -0400 Subject: [PATCH 10/33] i'm in it now --- peps/pep-0694.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 279e7ca667f..ec472c623d9 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -1,6 +1,6 @@ PEP: 694 Title: Upload 2.0 API for Python Package Indexes -Author: Barry Warsaw , Donald Stufft +Author: Barry Warsaw , Donald Stufft , Ee Durbin Discussions-To: https://discuss.python.org/t/pep-694-upload-2-0-api-for-python-package-repositories/16879 Status: Draft Type: Standards Track From 924a27d0ede174b9c6e7ea14e7373298c5a80660 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 30 May 2025 08:05:40 -0400 Subject: [PATCH 11/33] try to un-wonk content-type per feedback --- peps/pep-0694.rst | 61 +++++++++++++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 23 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index ec472c623d9..a0f8f9078f6 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -182,10 +182,12 @@ the url structure of a domain. For example, the root endpoint could be Create an Upload Session ~~~~~~~~~~~~~~~~~~~~~~~~ -A release starts by creating a new upload session. To create the session, a client submits a ``POST`` request -to the root URL, with a payload that looks like: +A release starts by creating a new upload session. To create the session, a client submits a +``POST`` request to the root URL like: -.. code-block:: json +.. code-block:: text + + Content-Type: application/vnd.pypi.upload.v2+json { "meta": { @@ -234,9 +236,11 @@ requests. Response Body +++++++++++++ -The successful response includes the following JSON content: +The successful response includes the following content: -.. code-block:: json +.. code-block:: text + + Content-Type: application/vnd.pypi.upload.v2+json { "meta": { @@ -362,9 +366,11 @@ After creating the session, the ``upload`` endpoint from the response's :ref:`se to those URLs from one session to the next. To initiate a file upload, a client first sends a ``POST`` request to the ``upload`` URL. The -request body has the following JSON format: +request looks like: + +.. code-block:: text -.. code-block:: json + Content-Type: application/vnd.pypi.upload.v2+json { "meta": { @@ -428,9 +434,11 @@ it **MUST** return a ``422 Unprocessable Entity``. File Upload Session Response Body +++++++++++++++++++++++++++++++++ -The successful response includes the following JSON content: +The successful response includes the following: + +.. code-block:: text -.. code-block:: json + Content-Type: application/vnd.pypi.upload.v2+json { "meta": { @@ -446,7 +454,8 @@ The successful response includes the following JSON content: "http-post-application-octet-stream": { "file_url": "..." "attestations_url": "..." - } + }, + ... } } @@ -488,9 +497,11 @@ To complete a file upload session, which indicates that the file upload mechanis and did not produce an error, a client issues a ``POST`` to the ``file-upload-session`` link in the file upload session creation response body. -The JSON body of this requests looks like: +The requests looks like: -.. code-block:: json +.. code-block:: text + + Content-Type: application/vnd.pypi.upload.v2+json { "meta": { @@ -564,9 +575,11 @@ allowed is left to the server. To extend a session, a client issues a ``POST`` given in the :ref:`session creation response body ` or :ref:`file upload session creation response body `, respectively. -The JSON body of this request looks like: +The request looks like: + +.. code-block:: text -.. code-block:: json + Content-Type: application/vnd.pypi.upload.v2+json { "meta": { @@ -614,9 +627,11 @@ To complete a session and publish the files that have been included in it, a cli ``POST`` request to the ``session`` :ref:`link ` given in the :ref:`session creation response body `. -The JSON body of this request looks like: +The request looks like: + +.. code-block:: text -.. code-block:: json + Content-Type: application/vnd.pypi.upload.v2+json { "meta": { @@ -709,9 +724,11 @@ experience, but the details of this are left to installer tool maintainers. Errors ------ -All error responses that contain content will have a body that looks like: +All error responses that contain content look like: -.. code-block:: json +.. code-block:: text + + Content-Type: application/vnd.pypi.upload.v2+json { "meta": { @@ -809,6 +826,9 @@ The structure of the ``Content-Type`` header for all other requests is: Since minor API version differences should never be disruptive, only the major version is included in the content type; the version number is prefixed with a ``v``. +The minor API version specified in the `.meta.api-version` JSON key of client requests +**MUST** match the `Content-Type` header for major version. + Unlike :pep:`691`, this PEP does not change the existing *legacy* ``1.0`` upload API in any way, so servers are required to host the new API described in this PEP at a different endpoint than the existing upload API. @@ -818,11 +838,6 @@ defined in this PEP **MUST** include a ``Content-Type`` header value of: - ``application/vnd.pypi.upload.v2+json``. -As with :pep:`691`, a special "meta" version is supported named ``latest``, the purpose of which is -to allow clients to request the latest version implemented by the server, without having to know -ahead of time what that version is. It is recommended however, that clients be explicit about what -versions they support. - Similar to :pep:`691`, this PEP also standardizes on using server-driven content negotiation to allow clients to request different versions or serialization formats, which includes the ``format`` part of the content type. However, since this PEP expects the existing legacy ``1.0`` upload API to From 7f0798b94a7cf02b86930972b66a8c987f37402d Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 30 May 2025 08:10:59 -0400 Subject: [PATCH 12/33] lint --- peps/pep-0694.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index a0f8f9078f6..b6b6a6ce132 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -826,8 +826,8 @@ The structure of the ``Content-Type`` header for all other requests is: Since minor API version differences should never be disruptive, only the major version is included in the content type; the version number is prefixed with a ``v``. -The minor API version specified in the `.meta.api-version` JSON key of client requests -**MUST** match the `Content-Type` header for major version. +The minor API version specified in the ``.meta.api-version`` JSON key of client requests +**MUST** match the ``Content-Type`` header for major version. Unlike :pep:`691`, this PEP does not change the existing *legacy* ``1.0`` upload API in any way, so servers are required to host the new API described in this PEP at a different endpoint than the From 3effee201b7404121b3b1c5c65ea106dbc2b030e Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 30 May 2025 08:51:42 -0400 Subject: [PATCH 13/33] clarify file upload mechanism details --- peps/pep-0694.rst | 45 +++++++++++++++++++++++++++++++-------------- 1 file changed, 31 insertions(+), 14 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index b6b6a6ce132..ef998ec1def 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -159,6 +159,7 @@ these steps: #. Complete the upload session, publishing or discarding the stage. #. Optionally check the status of an upload session. +.. _versioning: Versioning ---------- @@ -759,26 +760,42 @@ interpretation to aid in diagnosing underlying issue. File Upload Mechanisms ---------------------- -File Upload Mechanisms, with the exception of ``http-post-application-octet-stream`` are left as an -implementation detail specific to each server. Servers **MUST** implement a -``http-post-application-octet-stream`` mechanism. This serves as a fallback if no server specific implementations -exist. +Servers **MUST** implement :ref:`required file upload mechansisms `. +Such mechanisms serve as a fallback if no server specific implementations exist. -A given server may implement an arbitrary number of mechanisms and is responsible for documenting -their usage. Implemenatations **SHOULD** be prefixed with a string that clearly identifies the -server and is unique from other well known servers or implementations. +Each major version of the Upload API **MUST** specify at least one required File Upload Mechanism. -If a server intendes to match the behavior of another server's implementation, it **MAY** respond +New required mechanisms **MUST NOT** be added and existing required mechanisms **MUST NOT** be removed +without an update to the :ref:`major version `. + +A given server **MAY** implement an arbitrary number of server specific mechanisms +and is responsible for documenting their usage. +Server specific implementations **MUST** be prefixed with ``vnd-`` +and **SHOULD** further contain a string that clearly identifies the server operator +and is unique from other well known servers or implementations. + +For example: + +====================================== ================ ======================================================== +File Upload Mechanism string Server Operator Mechanism description +====================================== ================ ======================================================== +``vnd-pypi-s3multipart-presigned`` PyPI S3 multipart upload via pre-signed URL +``vnd-pypi-fetch`` PyPI File delivered by instructing server to fetch from a URL +``vnd-acmecorp-fetch`` Acme Corp File delivered by instructing server to fetch from a URL +``vnd-acmecorp-postal`` Acme Corp File delivered via postal mail +``vnd-madscience-quantumentanglement`` Mad Science Labs Upload via quantum entanglement +====================================== ================ ======================================================== + +If a server intends to match the behavior of another server's implementation, it **MAY** respond with that implementation's file upload mechanism name. -All implementations of this PEP **MUST** implement the ``http-post-application-octet-stream`` file -upload mechanism. +.. _required-file-upload-mechanisms: -``http-post-application-octet-stream`` Mechanism -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Required File Upload Mechanisms +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``http-post-application-octet-stream`` mechansism **MUST** be supported servers which -implement this PEP. +``http-post-application-octet-stream`` +++++++++++++++++++++++++++++++++++++++ A client executes this mechanism by submitting a ``POST`` request to the ``file_url`` returned in the ``http-post-application-octet-stream`` map of the ``mechanism`` map of the From c05324287ad4aa1989385b101012d32d3cd8fbad Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 30 May 2025 11:59:25 -0400 Subject: [PATCH 14/33] Fix typo Co-authored-by: Donald Stufft --- peps/pep-0694.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index ef998ec1def..76b15543a7f 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -843,7 +843,7 @@ The structure of the ``Content-Type`` header for all other requests is: Since minor API version differences should never be disruptive, only the major version is included in the content type; the version number is prefixed with a ``v``. -The minor API version specified in the ``.meta.api-version`` JSON key of client requests +The major API version specified in the ``.meta.api-version`` JSON key of client requests **MUST** match the ``Content-Type`` header for major version. Unlike :pep:`691`, this PEP does not change the existing *legacy* ``1.0`` upload API in any way, so From 847b2bd7acab90f6de9517327e2d6d31b5123030 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Wed, 11 Jun 2025 05:58:48 -0400 Subject: [PATCH 15/33] address general content feedback from review --- peps/pep-0694.rst | 52 ++++++++++++++++++++++++++++------------------- 1 file changed, 31 insertions(+), 21 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 76b15543a7f..7fad2470c9d 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -14,22 +14,23 @@ Post-History: `27-Jun-2022 `__; +* "staging" a release, which can be used to test uploads before publicly publishing them, + without the need for `test.pypi.org `__; * artifacts which can be overwritten and replaced, until a session is published; -* flexible file upload mechanisms for index operators; - * detailed status on the state of artifact uploads; * new project creation without requiring the uploading of an artifact. +* a protocol to extend the supported upload mechanisms in the future without requiring a full PEP; + these can be standardized and recommended for all indexes, or be index-specific; + Once this new upload API is adopted, the existing legacy API can be deprecated, however this PEP does not propose a deprecation schedule for the legacy API. @@ -77,11 +78,12 @@ In addition, there are a number of major issues with the legacy API: * Creation of new projects requires the uploading of at least one file, leading to "stub" uploads to claim a project namespace. -The new upload API proposed in this PEP provides a solution to all of these problems, -providing for a much more flexible approach, with support for servers to -implement resumable and parallel uploads via mechanisms, -better error reporting, better release testing experience, +The new upload API proposed in this PEP provides an immediate solution to many of these problems, +and defines a flexible mechanism for future support of the other problems by extension. +Indexes implementing this API will provide better error reporting, +better release testing experience, and atomic and simultaneous publishing of all release artifacts. +In the future indexes can implement resumable and parallel uploads via extensions. Legacy API @@ -154,8 +156,9 @@ these steps: #. Initiate an upload session, creating a release stage. #. Initiate file-upload session(s) to that stage as part of the upload session. -#. Execute file upload mechanism for the file-upload session(s). -#. Complete the file-upload session(s), marking them as executed or canceled. +#. Negotiate the specific file upload mechanism to use between client and server. +#. Execute file upload mechanism for the file-upload session(s) using the negotiated mechanism. +#. Complete the file-upload session(s), marking them as completed or canceled. #. Complete the upload session, publishing or discarding the stage. #. Optionally check the status of an upload session. @@ -178,6 +181,8 @@ All URLs described here are relative to the "root endpoint", which may be locate the url structure of a domain. For example, the root endpoint could be ``https://upload.example.com/``, or ``https://example.com/upload/``. +The choice of the root endpoint is left up to the index operator. + .. _session-create: Create an Upload Session @@ -271,7 +276,8 @@ the following keys: which are provided below. ``mechanisms`` - A list of file-upload mechanisms supported by the server. + A list of file-upload mechanisms supported by the server, sorted in server-preferred order. + At least one value is required. ``session-token`` If the index supports :ref:`previewing staged releases `, this key will contain @@ -427,8 +433,7 @@ the file to be uploaded. These checks may include, but are not limited to: If the server determines that upload should proceed, it will return a ``202 Accepted`` response, with the response body below. The :ref:`status ` of the session will also include the filename in the ``files`` mapping. If the server determines the upload cannot proceed, it **MUST** return a ``409 Conflict``. The server **MAY** allow parallel uploads of files, but is not required to. -If the server cannot proceed with an upload because the ``mechanism`` supplied by the client is not supported -it **MUST** return a ``422 Unprocessable Entity``. +If the server cannot proceed with an upload because the ``mechanism`` supplied by the client is not supported it **MUST** return a ``422 Unprocessable Entity``. .. _file-upload-session-response: @@ -511,13 +516,18 @@ The requests looks like: "action": "complete", } +After receiving this requests the server **MAY** perform additional asynchronous processing +on the file, for instance to verify its hashes or contents. -After receiving this requests the server **MAY** perform additional asynchronous processing on the file, -for instance to verify its hashes or contents. -If the processing is required to complete before an upload session can be published, -the status of the file upload session can be set to ``processing`` until such processing is complete, -reaches an error state, or the file upload session is canceled. +If the file upload session requires no further processing, the server **MUST** respond with a +``200 OK`` and the status of the file upload session **MUST** be set to set to ``complete``. +If such processing is required to complete before an upload session can be published, +the server **MUST** respond with a ``202 Accepted`` and the status +of the file upload session **MUST** be set to ``processing`` until such processing is complete, +reaches an error state, or the file upload session is canceled. +Clients can query the file upload session :ref:`status ` by issuing a GET request to +the ``file-upload-session`` :ref:`link `. .. _file-upload-session-cancelation: @@ -760,7 +770,7 @@ interpretation to aid in diagnosing underlying issue. File Upload Mechanisms ---------------------- -Servers **MUST** implement :ref:`required file upload mechansisms `. +Servers **MUST** implement :ref:`required file upload mechanisms `. Such mechanisms serve as a fallback if no server specific implementations exist. Each major version of the Upload API **MUST** specify at least one required File Upload Mechanism. From 64fdafc03e0f192a91d48079b2b05fdea14ae2bc Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Wed, 11 Jun 2025 05:59:12 -0400 Subject: [PATCH 16/33] bubble content-type handling up so that it can be assumed througouth --- peps/pep-0694.rst | 121 +++++++++++++++++++++------------------------- 1 file changed, 55 insertions(+), 66 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 7fad2470c9d..98e0e5b8f36 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -173,6 +173,52 @@ PEP does not modify the legacy API in any way. The API proposed in this PEP therefor has the version number ``2.0``. +Content Types +------------- + +Like :pep:`691`, this PEP proposes that all requests and responses from this upload API will have a +standard content type that describes what the content is, what version of the API it represents, and +what serialization format has been used. + +This standard request content type applies to all requests *except* for requests to execute +a File Upload Mechanism, which will be specified by the documentation for that mechanism. + +The structure of the ``Content-Type`` header for all other requests is: + +.. code-block:: text + + application/vnd.pypi.upload.$version+$format + +Since minor API version differences should never be disruptive, only the major version is included +in the content type; the version number is prefixed with a ``v``. + +The major API version specified in the ``.meta.api-version`` JSON key of client requests +**MUST** match the ``Content-Type`` header for major version. + +Unlike :pep:`691`, this PEP does not change the existing *legacy* ``1.0`` upload API in any way, so +servers are required to host the new API described in this PEP at a different endpoint than the +existing upload API. + +Since JSON is the only defined request format defined in this PEP, all non-file-upload requests +defined in this PEP **MUST** include a ``Content-Type`` header value of: + +- ``application/vnd.pypi.upload.v2+json``. + +Similar to :pep:`691`, this PEP also standardizes on using server-driven content negotiation to +allow clients to request different versions or serialization formats, which includes the ``format`` +part of the content type. However, since this PEP expects the existing legacy ``1.0`` upload API to +exist at a different endpoint, and this PEP currently only provides for JSON serialization, this +mechanism is not particularly useful. Clients only have a single version and serialization they can +request. However clients **SHOULD** be prepared to handle content negotiation gracefully in the case +that additional formats or versions are added in the future. + +Unless otherwise specified, all HTTP requests and responses in this document are assumed to include +the HTTP header: + +.. code-block:: text + + Content-Type: application/vnd.pypi.upload.v2+json + Root Endpoint ------------- @@ -191,9 +237,7 @@ Create an Upload Session A release starts by creating a new upload session. To create the session, a client submits a ``POST`` request to the root URL like: -.. code-block:: text - - Content-Type: application/vnd.pypi.upload.v2+json +.. code-block:: json { "meta": { @@ -244,9 +288,7 @@ Response Body The successful response includes the following content: -.. code-block:: text - - Content-Type: application/vnd.pypi.upload.v2+json +.. code-block:: json { "meta": { @@ -375,9 +417,7 @@ to those URLs from one session to the next. To initiate a file upload, a client first sends a ``POST`` request to the ``upload`` URL. The request looks like: -.. code-block:: text - - Content-Type: application/vnd.pypi.upload.v2+json +.. code-block:: json { "meta": { @@ -442,9 +482,7 @@ File Upload Session Response Body The successful response includes the following: -.. code-block:: text - - Content-Type: application/vnd.pypi.upload.v2+json +.. code-block:: json { "meta": { @@ -460,8 +498,7 @@ The successful response includes the following: "http-post-application-octet-stream": { "file_url": "..." "attestations_url": "..." - }, - ... + } } } @@ -505,9 +542,7 @@ file upload session creation response body. The requests looks like: -.. code-block:: text - - Content-Type: application/vnd.pypi.upload.v2+json +.. code-block:: json { "meta": { @@ -588,9 +623,7 @@ or :ref:`file upload session creation response body `. The request looks like: -.. code-block:: text - - Content-Type: application/vnd.pypi.upload.v2+json +.. code-block:: json { "meta": { @@ -737,9 +768,7 @@ Errors All error responses that contain content look like: -.. code-block:: text - - Content-Type: application/vnd.pypi.upload.v2+json +.. code-block:: json { "meta": { @@ -834,46 +863,6 @@ containing a JSON array of :pep:`attestation objects <740#attestation-objects>` [{"version": 1, "verification_material": {...}, "envelope": {...}},...] -Content Types -------------- - -Like :pep:`691`, this PEP proposes that all requests and responses from this upload API will have a -standard content type that describes what the content is, what version of the API it represents, and -what serialization format has been used. - -This standard request content type applies to all requests *except* for requests to execute -a File Upload Mechanism, which will be specified by the documentation for that mechanism. - -The structure of the ``Content-Type`` header for all other requests is: - -.. code-block:: text - - application/vnd.pypi.upload.$version+$format - -Since minor API version differences should never be disruptive, only the major version is included -in the content type; the version number is prefixed with a ``v``. - -The major API version specified in the ``.meta.api-version`` JSON key of client requests -**MUST** match the ``Content-Type`` header for major version. - -Unlike :pep:`691`, this PEP does not change the existing *legacy* ``1.0`` upload API in any way, so -servers are required to host the new API described in this PEP at a different endpoint than the -existing upload API. - -Since JSON is the only defined request format defined in this PEP, all non-file-upload requests -defined in this PEP **MUST** include a ``Content-Type`` header value of: - -- ``application/vnd.pypi.upload.v2+json``. - -Similar to :pep:`691`, this PEP also standardizes on using server-driven content negotiation to -allow clients to request different versions or serialization formats, which includes the ``format`` -part of the content type. However, since this PEP expects the existing legacy ``1.0`` upload API to -exist at a different endpoint, and this PEP currently only provides for JSON serialization, this -mechanism is not particularly useful. Clients only have a single version and serialization they can -request. However clients **SHOULD** be prepared to handle content negotiation gracefully in the case -that additional formats or versions are added in the future. - - FAQ === From 2d634d7a9be8181711010f8db2123b15be0b0baa Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Wed, 11 Jun 2025 06:27:41 -0400 Subject: [PATCH 17/33] restructure document to un-twist Upload Sessions and File Upload Sessions --- peps/pep-0694.rst | 344 ++++++++++++++++++++++++---------------------- 1 file changed, 178 insertions(+), 166 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 98e0e5b8f36..cd9c81b0335 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -154,12 +154,14 @@ This PEP traces the root cause of most of the issues with the existing API to be To address these issues, this PEP proposes a multi-request workflow, which at a high level involves these steps: -#. Initiate an upload session, creating a release stage. -#. Initiate file-upload session(s) to that stage as part of the upload session. -#. Negotiate the specific file upload mechanism to use between client and server. -#. Execute file upload mechanism for the file-upload session(s) using the negotiated mechanism. -#. Complete the file-upload session(s), marking them as completed or canceled. -#. Complete the upload session, publishing or discarding the stage. +#. Initiate an :ref:`Upload session `, creating a release stage. +#. Initiate :ref:`File Upload Session(s) ` to that stage + as part of the upload session. +#. Negotiate the specific :ref:`File Upload Mechanism ` to use + between client and server. +#. Execute File Upload Mechanism for the File Upload Session(s) using the negotiated mechanism(s). +#. Complete the File Upload Session(s), marking them as completed or canceled. +#. Complete the Upload Session, publishing or discarding the stage. #. Optionally check the status of an upload session. .. _versioning: @@ -229,6 +231,45 @@ the url structure of a domain. For example, the root endpoint could be The choice of the root endpoint is left up to the index operator. +.. _session-errors: + +Errors +------ + +All error responses that contain content look like: + +.. code-block:: json + + { + "meta": { + "api-version": "2.0" + }, + "message": "...", + "errors": [ + { + "source": "...", + "message": "..." + } + ] + } + +Besides the standard ``meta`` key, this has the following top level keys: + +``message`` + A singular message that encapsulates all errors that may have happened on this + request. + +``errors`` + An array of specific errors, each of which contains a ``source`` key, which is a string that + indicates what the source of the error is, and a ``message`` key for that specific error. + +The ``message`` and ``source`` strings do not have any specific meaning, and are intended for human +interpretation to aid in diagnosing underlying issue. + + +Upload Session +-------------- + .. _session-create: Create an Upload Session @@ -359,7 +400,7 @@ Session Links For the ``links`` key in the success JSON, the following sub-keys are valid: ``upload`` - The endpoint session clients will use to initiate a :ref:`file-upload session ` + The endpoint session clients will use to initiate a :ref:`file-upload session ` for each file to be included in this session. ``stage`` @@ -391,7 +432,7 @@ sub-mapping with the following keys: ``link`` The *absolute* URL that the client should use to reference this specific file. This URL is used - to retrieve, replace, or delete the :ref:`referenced file `. If a ``nonce`` was + to retrieve, replace, or delete the :ref:`referenced file `. If a ``nonce`` was provided, this URL **MUST** be obfuscated with a non-guessable token as described in the :ref:`session token ` section. @@ -401,13 +442,112 @@ sub-mapping with the following keys: If a second session is created for the same name-version pair while a session for that pair is in the ``pending`` state, then the server **MUST** return the JSON status response for the already -existing session, along with the ``200 Ok`` status code rather than creating a new, empty session. +existing session, along with the ``200 OK`` status code rather than creating a new, empty session. + + +.. _publish-session: + +Complete an Upload Session +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To complete a session and publish the files that have been included in it, a client issues a +``POST`` request to the ``session`` :ref:`link ` given in the :ref:`session creation +response body `. + +The request looks like: + +.. code-block:: json + + { + "meta": { + "api-version": "2.0" + }, + "action": "publish", + } + + +If the server is able to immediately complete the Upload Session, it may do so and return a +``201 Created`` response. If it is unable to immediately complete the Upload Session +(for instance, if it needs to do validation that may take longer than reasonable in a single HTTP +request), then it may return a ``202 Accepted`` response. + +In either case, the server should include a ``Location`` header pointing back to the Upload Session +status URL, and if the server returned a ``202 Accepted``, the client may poll that URL to +watch for the status to change. + +If an error occurs, the appropriate ``4xx`` code should be returned, as described in the +:ref:`session-errors` section. + +.. _session-cancellation: + +Session Cancellation +~~~~~~~~~~~~~~~~~~~~ + +To cancel an Upload Session, a client issues a ``DELETE`` request to the ``session`` :ref:`link +` given in the :ref:`session creation response body `. The server +then marks the session as canceled, and **SHOULD** purge any data that was uploaded as part of that +session. Future attempts to access that session URL or any of the upload session URLs **MUST** +return a ``404 Not Found``. + +To prevent dangling sessions, servers may also choose to cancel timed-out sessions on their own +accord. It is recommended that servers expunge their sessions after no less than a week, but each +server may choose their own schedule. Servers **MAY** support client-directed :ref:`session +extensions `. + + +.. _session-token: + +Session Token +~~~~~~~~~~~~~ + +When creating an Upload Session, clients can provide a ``nonce`` in the +:ref:`initial session creation request `. +This nonce is a string with arbitrary content. The ``nonce`` is +optional, and if omitted, is equivalent to providing an empty string. + +In order to support previewing of staged uploads, the package ``name`` and ``version``, along with +this ``nonce`` are used as input into a hashing algorithm to produce a unique "session token". This +session token is valid for the life of the session (i.e., until it is completed, either by +cancellation or publishing), and can be provided to supporting installers to gain access to the +staged release. + +The use of the ``nonce`` allows clients to decide whether they want to obscure the visibility of +their staged releases or not, and there can be good reasons for either choice. For example, if a CI +system wants to upload some wheels for a new release, and wants to allow independent validation of a +stage before it's published, the client may opt for not including a nonce. On the other hand, if a +client would like to pre-seed a release which it publishes atomically at the time of a public +announcement, that client will likely opt for providing a nonce. + +The `SHA256 algorithm `_ is used to +turn these inputs into a unique token, in the order ``name``, ``version``, ``nonce``, using the +following Python code as an example: + +.. code-block:: python + + from hashlib import sha256 + + def gentoken(name: bytes, version: bytes, nonce: bytes = b''): + h = sha256() + h.update(name) + h.update(version) + h.update(nonce) + return h.hexdigest() + +It should be evident that if no ``nonce`` is provided in the :ref:`session creation request +`, then the preview token is easily guessable from the package name and version +number alone. Clients can elect to omit the ``nonce`` (or set it to the empty string themselves) if +they want to allow previewing from anybody without access to the preview token. By providing a +non-empty ``nonce``, clients can elect for security-through-obscurity, but this does not protect +staged files behind any kind of authentication. -.. _file-uploads: +File Upload Session +------------------- -File Upload -~~~~~~~~~~~ +.. _file-upload-session: + +Create a File Upload Session +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After creating the session, the ``upload`` endpoint from the response's :ref:`session links ` mapping is used to begin the upload of new files into that session. Clients @@ -477,8 +617,8 @@ If the server cannot proceed with an upload because the ``mechanism`` supplied b .. _file-upload-session-response: -File Upload Session Response Body -+++++++++++++++++++++++++++++++++ +Response Body ++++++++++++++ The successful response includes the following: @@ -533,8 +673,8 @@ For the ``links`` key in the success JSON, the following sub-keys are valid: .. _file-upload-session-completion: -File Upload Session Completion -++++++++++++++++++++++++++++++ +Complete a File Upload Session +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To complete a file upload session, which indicates that the file upload mechanism has been executed and did not produce an error, a client issues a ``POST`` to the ``file-upload-session`` link in the @@ -551,23 +691,25 @@ The requests looks like: "action": "complete", } -After receiving this requests the server **MAY** perform additional asynchronous processing -on the file, for instance to verify its hashes or contents. +If the server is able to immediately complete the File Upload Session, it may do so and return a +``201 Created`` response and set the status of the File Upload Session to ``complete``. +If it is unable to immediately complete the File Upload Session +(for instance, if it needs to do validation that may take longer than reasonable in a single HTTP +request), then it may return a ``202 Accepted`` response +and set the status of the File Upload Session to ``processing``. + +In either case, the server should include a ``Location`` header pointing back to the File Upload +Session status URL, and if the server returned a ``202 Accepted``, the client may poll that URL to +watch for the status to change. -If the file upload session requires no further processing, the server **MUST** respond with a -``200 OK`` and the status of the file upload session **MUST** be set to set to ``complete``. +If an error occurs, the appropriate ``4xx`` code should be returned, as described in the +:ref:`session-errors` section. -If such processing is required to complete before an upload session can be published, -the server **MUST** respond with a ``202 Accepted`` and the status -of the file upload session **MUST** be set to ``processing`` until such processing is complete, -reaches an error state, or the file upload session is canceled. -Clients can query the file upload session :ref:`status ` by issuing a GET request to -the ``file-upload-session`` :ref:`link `. .. _file-upload-session-cancelation: -Canceling and Deleting File Uploads -+++++++++++++++++++++++++++++++++++ +Cancellation and Deletion +~~~~~~~~~~~~~~~~~~~~~~~~~ A client can cancel an in-progress upload session for a file, or delete a file that has been completely uploaded. In both cases, the client performs this by issuing a ``DELETE`` request to @@ -580,7 +722,7 @@ or associated file upload mechanisms can be reused. Replacing a Partially or Fully Uploaded File -++++++++++++++++++++++++++++++++++++++++++++ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To replace a session file, the file upload **MUST** have been previously completed, canceled, or deleted. It is not possible to replace a file if the upload for that file is in-progress. @@ -588,15 +730,15 @@ deleted. It is not possible to replace a file if the upload for that file is in To replace a session file, clients should :ref:`cancel and delete the in-progress upload ` by issuing a ``DELETE`` to the upload resource URL for the file they want to replace. After this, the new file upload can be initiated by beginning the entire :ref:`file upload -` sequence over again. This means providing the metadata request again to retrieve a -new upload resource URL. Client **MUST NOT** assume that the previous upload resource URL can be -reused after deletion. +` sequence over again. This means providing the metadata request again to +retrieve a new upload resource URL. +Clients **MUST NOT** assume that the previous upload resource URL can be reused after deletion. .. _session-status: Session Status -~~~~~~~~~~~~~~ +-------------- At any time, a client can query the status of a session by issuing a ``GET`` request to the ``session`` :ref:`link ` or ``file-upload-session`` :ref:`link ` @@ -613,7 +755,7 @@ except with any changes to ``status``, ``valid-for``, or ``files`` reflected. .. _session-extension: Session Extension -~~~~~~~~~~~~~~~~~ +----------------- Servers **MAY** allow clients to extend sessions, but the overall lifetime and number of extensions allowed is left to the server. To extend a session, a client issues a ``POST`` request to the @@ -644,107 +786,10 @@ If the server refuses to extend the session for the requested number of seconds, success response, and the ``valid-for`` key will simply include the number of seconds remaining in the current session. - -.. _session-cancellation: - -Session Cancellation -~~~~~~~~~~~~~~~~~~~~ - -To cancel an entire session, a client issues a ``DELETE`` request to the ``session`` :ref:`link -` given in the :ref:`session creation response body `. The server -then marks the session as canceled, and **SHOULD** purge any data that was uploaded as part of that -session. Future attempts to access that session URL or any of the upload session URLs **MUST** -return a ``404 Not Found``. - -To prevent dangling sessions, servers may also choose to cancel timed-out sessions on their own -accord. It is recommended that servers expunge their sessions after no less than a week, but each -server may choose their own schedule. Servers **MAY** support client-directed :ref:`session -extensions `. - - -.. _publish-session: - -Session Completion -~~~~~~~~~~~~~~~~~~ - -To complete a session and publish the files that have been included in it, a client issues a -``POST`` request to the ``session`` :ref:`link ` given in the :ref:`session creation -response body `. - -The request looks like: - -.. code-block:: json - - { - "meta": { - "api-version": "2.0" - }, - "action": "publish", - } - - -If the server is able to immediately complete the file upload session, it may do so and return a -``201 Created`` response. If it is unable to immediately complete the file upload session -(for instance, if it needs to do validation that may take longer than reasonable in a single HTTP -request), then it may return a ``202 Accepted`` response. - -In either case, the server should include a ``Location`` header pointing back to the file upload -session status URL, and if the server returned a ``202 Accepted``, the client may poll that URL to -watch for the status to change. - -If an error occurs, the appropriate ``4xx`` code should be returned, as described in the -:ref:`session-errors` section. - - -.. _session-token: - -Session Token -~~~~~~~~~~~~~ - -When creating a session, clients can provide a ``nonce`` in the :ref:`initial session creation -request ` . This nonce is a string with arbitrary content. The ``nonce`` is -optional, and if omitted, is equivalent to providing an empty string. - -In order to support previewing of staged uploads, the package ``name`` and ``version``, along with -this ``nonce`` are used as input into a hashing algorithm to produce a unique "session token". This -session token is valid for the life of the session (i.e., until it is completed, either by -cancellation or publishing), and can be provided to supporting installers to gain access to the -staged release. - -The use of the ``nonce`` allows clients to decide whether they want to obscure the visibility of -their staged releases or not, and there can be good reasons for either choice. For example, if a CI -system wants to upload some wheels for a new release, and wants to allow independent validation of a -stage before it's published, the client may opt for not including a nonce. On the other hand, if a -client would like to pre-seed a release which it publishes atomically at the time of a public -announcement, that client will likely opt for providing a nonce. - -The `SHA256 algorithm `_ is used to -turn these inputs into a unique token, in the order ``name``, ``version``, ``nonce``, using the -following Python code as an example: - -.. code-block:: python - - from hashlib import sha256 - - def gentoken(name: bytes, version: bytes, nonce: bytes = b''): - h = sha256() - h.update(name) - h.update(version) - h.update(nonce) - return h.hexdigest() - -It should be evident that if no ``nonce`` is provided in the :ref:`session creation request -`, then the preview token is easily guessable from the package name and version -number alone. Clients can elect to omit the ``nonce`` (or set it to the empty string themselves) if -they want to allow previewing from anybody without access to the preview token. By providing a -non-empty ``nonce``, clients can elect for security-through-obscurity, but this does not protect -staged files behind any kind of authentication. - - .. _staged-preview: Stage Previews -~~~~~~~~~~~~~~ +-------------- The ability to preview staged releases before they are published is an important feature of this PEP, enabling an additional level of last-mile testing before the release is available to the @@ -761,40 +806,7 @@ changes, although perhaps in a less user-friendly way. The latter option can be experience, but the details of this are left to installer tool maintainers. -.. _session-errors: - -Errors ------- - -All error responses that contain content look like: - -.. code-block:: json - - { - "meta": { - "api-version": "2.0" - }, - "message": "...", - "errors": [ - { - "source": "...", - "message": "..." - } - ] - } - -Besides the standard ``meta`` key, this has the following top level keys: - -``message`` - A singular message that encapsulates all errors that may have happened on this - request. - -``errors`` - An array of specific errors, each of which contains a ``source`` key, which is a string that - indicates what the source of the error is, and a ``message`` key for that specific error. - -The ``message`` and ``source`` strings do not have any specific meaning, and are intended for human -interpretation to aid in diagnosing underlying issue. +.. _file-upload-mechanisms: File Upload Mechanisms ---------------------- From 6fea5f9203a5bc222c91e638a510fee314aa1ea3 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Wed, 11 Jun 2025 06:53:51 -0400 Subject: [PATCH 18/33] naming things --- peps/pep-0694.rst | 75 +++++++++++++++++++++++++++++------------------ 1 file changed, 46 insertions(+), 29 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index cd9c81b0335..7ca7293e109 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -340,7 +340,7 @@ The successful response includes the following content: "upload": "...", "session": "...", }, - "mechanisms": ["http-post-application-octet-stream"], + "mechanisms": ["http-post-bytes"], "session-token": "", "valid-for": 604800, "status": "pending", @@ -567,7 +567,7 @@ request looks like: "size": 1000, "hashes": {"sha256": "...", "blake2b": "..."}, "metadata": "...", - "mechanism": "http-post-application-octet-stream" + "mechanism": "http-post-bytes" } @@ -635,7 +635,7 @@ The successful response includes the following: "status": "pending", "valid-for": 3600, "mechanism": { - "http-post-application-octet-stream": { + "http-post-bytes": { "file_url": "..." "attestations_url": "..." } @@ -816,40 +816,20 @@ Such mechanisms serve as a fallback if no server specific implementations exist. Each major version of the Upload API **MUST** specify at least one required File Upload Mechanism. -New required mechanisms **MUST NOT** be added and existing required mechanisms **MUST NOT** be removed +New required mechanisms **MUST NOT** be added +and existing required mechanisms **MUST NOT** be removed without an update to the :ref:`major version `. -A given server **MAY** implement an arbitrary number of server specific mechanisms -and is responsible for documenting their usage. -Server specific implementations **MUST** be prefixed with ``vnd-`` -and **SHOULD** further contain a string that clearly identifies the server operator -and is unique from other well known servers or implementations. - -For example: - -====================================== ================ ======================================================== -File Upload Mechanism string Server Operator Mechanism description -====================================== ================ ======================================================== -``vnd-pypi-s3multipart-presigned`` PyPI S3 multipart upload via pre-signed URL -``vnd-pypi-fetch`` PyPI File delivered by instructing server to fetch from a URL -``vnd-acmecorp-fetch`` Acme Corp File delivered by instructing server to fetch from a URL -``vnd-acmecorp-postal`` Acme Corp File delivered via postal mail -``vnd-madscience-quantumentanglement`` Mad Science Labs Upload via quantum entanglement -====================================== ================ ======================================================== - -If a server intends to match the behavior of another server's implementation, it **MAY** respond -with that implementation's file upload mechanism name. - .. _required-file-upload-mechanisms: Required File Upload Mechanisms ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -``http-post-application-octet-stream`` -++++++++++++++++++++++++++++++++++++++ +``http-post-bytes`` ++++++++++++++++++++ A client executes this mechanism by submitting a ``POST`` request to the ``file_url`` returned in the -``http-post-application-octet-stream`` map of the ``mechanism`` map of the +``http-post-bytes`` map of the ``mechanism`` map of the :ref:`file upload session creation response body ` like: .. code-block:: text @@ -860,7 +840,7 @@ A client executes this mechanism by submitting a ``POST`` request to the ``file_ Servers **MAY** support uploading of digital attestations for files (see :pep:`740`). This support will be indicated by inclusion of an ``attestations_url`` key in the -``http-post-application-octet-stream`` map of the ``mechanism`` map of the +``http-post-bytes`` map of the ``mechanism`` map of the :ref:`file upload session creation response body `. Attestations **MUST** be uploaded to the ``attestations_url`` before :ref:`file upload session completion `. @@ -875,6 +855,43 @@ containing a JSON array of :pep:`attestation objects <740#attestation-objects>` [{"version": 1, "verification_material": {...}, "envelope": {...}},...] +.. _server-specific-file-upload-mechanisms: + +Server Specific File Upload Mechanisms +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A given server **MAY** implement an arbitrary number of server specific mechanisms +and is responsible for documenting their usage. + +A server specific implementation file upload mechanism identifier has three parts: + +.. code-block:: text + + -- + +Server specific implementations **MUST** use ``vnd`` as their ``prefix``. +The ``operator identifier`` **SHOULD** clearly identify the server operator, +be unique from other well known indexes, +and contain only alphanumeric characters ``[a-z0-9]``. +The ``implementation identifier`` **SHOULD** concisely describe the underlying implementation +and contain only alphanumeric characters ``[a-z0-9]`` and ``-``. + +For example: + +====================================== ================ ========================================================================= +File Upload Mechanism string Server Operator Mechanism description +====================================== ================ ========================================================================= +``vnd-pypi-s3multipart-presigned`` PyPI S3 multipart upload via pre-signed URL +``vnd-pypi-http-fetch`` PyPI File delivered by instructing server to fetch from a URL via HTTP request +``vnd-acmecorp-http-fetch`` Acme Corp File delivered by instructing server to fetch from a URL via HTTP request +``vnd-acmecorp-postal`` Acme Corp File delivered via postal mail +``vnd-madscience-quantumentanglement`` Mad Science Labs Upload via quantum entanglement +====================================== ================ ========================================================================= + +If a server intends to match the behavior of another server's implementation, it **MAY** respond +with that implementation's file upload mechanism name. + + FAQ === From 86710f1e5c89c27c6ae6046f90609a0c6fd24f40 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Wed, 11 Jun 2025 07:22:35 -0400 Subject: [PATCH 19/33] flatten the `mechanism` value of the file upload session response --- peps/pep-0694.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 7ca7293e109..01a72658326 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -635,10 +635,9 @@ The successful response includes the following: "status": "pending", "valid-for": 3600, "mechanism": { - "http-post-bytes": { - "file_url": "..." - "attestations_url": "..." - } + "identifier": "http-post-bytes", + "file_url": "...", + "attestations_url": "..." } } @@ -651,8 +650,10 @@ the following keys: the details of which are provided below. ``mechanism`` - A mapping containing the supported mechanism identifier negotiated by the client and server, - to a mapping containing details necessary to execute the mechanism. + A mapping containing the necessary details for the supported mechanism + as negotiated by the client and server. + This mapping **MUST** contain a key ``identifier`` which maps to + the identifier string for the File Upload Mechanism. .. _file-upload-session-links: From 5fbf005899188b0bdce36244f2c02e50dd3dcd13 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Wed, 11 Jun 2025 07:57:02 -0400 Subject: [PATCH 20/33] Rename "Upload Session" to "Publishing Session" --- peps/pep-0694.rst | 170 ++++++++++++++++++++++++---------------------- 1 file changed, 87 insertions(+), 83 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 01a72658326..5c361d0cb26 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -17,7 +17,8 @@ Abstract This PEP proposes an extensible API for uploading files to a Python package index such as PyPI. Along with standardization, the upload API provides additional useful features such as support for: -* an upload session, which can be used to simultaneously publish all wheels in a package release; +* an publishing session, which can be used to simultaneously publish + all wheels in a package release; * "staging" a release, which can be used to test uploads before publicly publishing them, without the need for `test.pypi.org `__; @@ -154,15 +155,15 @@ This PEP traces the root cause of most of the issues with the existing API to be To address these issues, this PEP proposes a multi-request workflow, which at a high level involves these steps: -#. Initiate an :ref:`Upload session `, creating a release stage. +#. Initiate an :ref:`Publishing Session `, creating a release stage. #. Initiate :ref:`File Upload Session(s) ` to that stage - as part of the upload session. + as part of the Publishing Session. #. Negotiate the specific :ref:`File Upload Mechanism ` to use between client and server. #. Execute File Upload Mechanism for the File Upload Session(s) using the negotiated mechanism(s). #. Complete the File Upload Session(s), marking them as completed or canceled. -#. Complete the Upload Session, publishing or discarding the stage. -#. Optionally check the status of an upload session. +#. Complete the Publishing Session, publishing or discarding the stage. +#. Optionally check the status of a Publishing Session. .. _versioning: @@ -267,15 +268,17 @@ The ``message`` and ``source`` strings do not have any specific meaning, and are interpretation to aid in diagnosing underlying issue. -Upload Session --------------- +.. _publishing-session: -.. _session-create: +Publishing Session +------------------ -Create an Upload Session -~~~~~~~~~~~~~~~~~~~~~~~~ +.. _publishing-session-create: + +Create a Publishing Session +~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A release starts by creating a new upload session. To create the session, a client submits a +A release starts by creating a new Publishing Session. To create the session, a client submits a ``POST`` request to the root URL like: .. code-block:: json @@ -303,7 +306,7 @@ The request includes the following top-level keys: The version of the project that this session is attempting to add files to. ``nonce`` (**optional**) - An additional client-side string input to the :ref:`"session token" ` + An additional client-side string input to the :ref:`"Publishing Session Token" ` algorithm. Details are provided below, but if this key is omitted, it is equivalent to passing the empty string. @@ -322,7 +325,7 @@ with the same credentials, otherwise a ``403 Forbidden`` will be returned on tho requests. -.. _session-response: +.. _publishing-session-response: Response Body +++++++++++++ @@ -355,7 +358,7 @@ Besides the ``meta`` key, which has the same format as the request JSON, the suc the following keys: ``links`` - A dictionary mapping :ref:`keys to URLs ` related to this session, the details of + A dictionary mapping :ref:`keys to URLs ` related to this session, the details of which are provided below. ``mechanisms`` @@ -364,7 +367,7 @@ the following keys: ``session-token`` If the index supports :ref:`previewing staged releases `, this key will contain - the unique :ref:`"session token" ` that can be provided to installers in order to + the unique :ref:`"session token" ` that can be provided to installers in order to preview the staged release before it's published. If the index does *not* support stage previewing, this key **MUST** be omitted. @@ -384,7 +387,7 @@ the following keys: ``files`` A mapping containing the filenames that have been uploaded to this session, to a mapping - containing details about each :ref:`file referenced in this session `. + containing details about each :ref:`file referenced in this session `. ``notices`` An optional key that points to an array of human-readable informational notices that the server @@ -392,15 +395,15 @@ the following keys: to any particular file in the session. -.. _session-links: +.. _publishing-session-links: -Session Links -+++++++++++++ +Publishing Session Links +++++++++++++++++++++++++ For the ``links`` key in the success JSON, the following sub-keys are valid: ``upload`` - The endpoint session clients will use to initiate a :ref:`file-upload session ` + The endpoint session clients will use to initiate a :ref:`File Upload Session ` for each file to be included in this session. ``stage`` @@ -410,15 +413,15 @@ For the ``links`` key in the success JSON, the following sub-keys are valid: ``session`` The endpoint where actions for this session can be performed, including :ref:`publishing this - session `, :ref:`canceling and discarding the session `, + session `, :ref:`canceling and discarding the session `, :ref:`querying the current session status `, and :ref:`requesting an extension of the session lifetime ` (*if* the server supports it). -.. _session-files: +.. _publishing-session-files: -Session Files -+++++++++++++ +Publishing Session Files +++++++++++++++++++++++++ The ``files`` key contains a mapping from the names of the files uploaded in this session to a sub-mapping with the following keys: @@ -428,13 +431,13 @@ sub-mapping with the following keys: If there was an error during upload, then clients should not assume the file is in any usable state, ``error`` will be returned and it's best to :ref:`cancel or delete ` the file and start over. This action would remove the file name from the ``files`` key of the - :ref:`session status response body `. + :ref:`session status response body `. ``link`` The *absolute* URL that the client should use to reference this specific file. This URL is used to retrieve, replace, or delete the :ref:`referenced file `. If a ``nonce`` was provided, this URL **MUST** be obfuscated with a non-guessable token as described in the - :ref:`session token ` section. + :ref:`Publishing Session Token ` section. ``notices`` An optional key with similar format and semantics as the ``notices`` session key, except that @@ -445,14 +448,14 @@ the ``pending`` state, then the server **MUST** return the JSON status response existing session, along with the ``200 OK`` status code rather than creating a new, empty session. -.. _publish-session: +.. _publishing-session-completion: -Complete an Upload Session -~~~~~~~~~~~~~~~~~~~~~~~~~~ +Complete a Publishing Session +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To complete a session and publish the files that have been included in it, a client issues a -``POST`` request to the ``session`` :ref:`link ` given in the :ref:`session creation -response body `. +``POST`` request to the ``session`` :ref:`link ` given in the :ref:`session creation +response body `. The request looks like: @@ -466,27 +469,27 @@ The request looks like: } -If the server is able to immediately complete the Upload Session, it may do so and return a -``201 Created`` response. If it is unable to immediately complete the Upload Session +If the server is able to immediately complete the Publishing Session, it may do so and return a +``201 Created`` response. If it is unable to immediately complete the Publishing Session (for instance, if it needs to do validation that may take longer than reasonable in a single HTTP request), then it may return a ``202 Accepted`` response. -In either case, the server should include a ``Location`` header pointing back to the Upload Session +In either case, the server should include a ``Location`` header pointing back to the Publishing Session status URL, and if the server returned a ``202 Accepted``, the client may poll that URL to watch for the status to change. If an error occurs, the appropriate ``4xx`` code should be returned, as described in the :ref:`session-errors` section. -.. _session-cancellation: +.. _publishing-session-cancellation: -Session Cancellation -~~~~~~~~~~~~~~~~~~~~ +Cancellation +~~~~~~~~~~~~ -To cancel an Upload Session, a client issues a ``DELETE`` request to the ``session`` :ref:`link -` given in the :ref:`session creation response body `. The server +To cancel a Publishing Session, a client issues a ``DELETE`` request to the ``session`` :ref:`link +` given in the :ref:`session creation response body `. The server then marks the session as canceled, and **SHOULD** purge any data that was uploaded as part of that -session. Future attempts to access that session URL or any of the upload session URLs **MUST** +session. Future attempts to access that session URL or any of the Publishing Session URLs **MUST** return a ``404 Not Found``. To prevent dangling sessions, servers may also choose to cancel timed-out sessions on their own @@ -495,13 +498,13 @@ server may choose their own schedule. Servers **MAY** support client-directed : extensions `. -.. _session-token: +.. _publishing-session-token: -Session Token -~~~~~~~~~~~~~ +Publishing Session Token +~~~~~~~~~~~~~~~~~~~~~~~~ -When creating an Upload Session, clients can provide a ``nonce`` in the -:ref:`initial session creation request `. +When creating a Publishing Session, clients can provide a ``nonce`` in the +:ref:`initial session creation request `. This nonce is a string with arbitrary content. The ``nonce`` is optional, and if omitted, is equivalent to providing an empty string. @@ -534,9 +537,9 @@ following Python code as an example: return h.hexdigest() It should be evident that if no ``nonce`` is provided in the :ref:`session creation request -`, then the preview token is easily guessable from the package name and version +`, then the session token is easily guessable from the package name and version number alone. Clients can elect to omit the ``nonce`` (or set it to the empty string themselves) if -they want to allow previewing from anybody without access to the preview token. By providing a +they want to allow previewing from anybody without access to the session token. By providing a non-empty ``nonce``, clients can elect for security-through-obscurity, but this does not protect staged files behind any kind of authentication. @@ -549,8 +552,8 @@ File Upload Session Create a File Upload Session ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -After creating the session, the ``upload`` endpoint from the response's :ref:`session links -` mapping is used to begin the upload of new files into that session. Clients +After creating a Publishing Session, the ``upload`` endpoint from the response's :ref:`session links +` mapping is used to begin the upload of new files into that session. Clients **MUST** use the provided ``upload`` URL and **MUST NOT** assume there is any pattern or commonality to those URLs from one session to the next. @@ -592,8 +595,8 @@ Besides the standard ``meta`` key, the request JSON has the following additional ``mechanism`` (**required**) The file-upload mechanisms the client intends to use for this file. - This mechanism **SHOULD** be chosen from the list of mechanisms advertised in the `session response body - `_. + This mechanism **SHOULD** be chosen from the list of mechanisms advertised in the + :ref:`Publishing Session response body `. A client **MAY** send a mechanism that is not advertised in cases where server operators have documented a new or up-coming mechanism that is available for use on a "pre-release" basis. @@ -629,7 +632,7 @@ The successful response includes the following: "api-version": "2.0" }, "links": { - "session": "...", + "publishing-session": "...", "file-upload-session": "..." }, "status": "pending", @@ -657,19 +660,19 @@ the following keys: .. _file-upload-session-links: -Session Links -+++++++++++++ +File Upload Session Links ++++++++++++++++++++++++++ For the ``links`` key in the success JSON, the following sub-keys are valid: -``session`` - The endpoint where actions for the parent session can be performed. +``publishing-session`` + The endpoint where actions for the parent Publishing Session can be performed. ``file-upload-session`` The endpoint where actions for this file-upload-session can be performed. - including :ref:`canceling and discarding the file upload session `, - :ref:`querying the current file upload session status `, - and :ref:`requesting an extension of the file upload session lifetime ` + including :ref:`canceling and discarding the File Upload Session `, + :ref:`querying the current File Upload Session status `, + and :ref:`requesting an extension of the File Upload Session lifetime ` (*if* the server supports it). .. _file-upload-session-completion: @@ -677,9 +680,9 @@ For the ``links`` key in the success JSON, the following sub-keys are valid: Complete a File Upload Session ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To complete a file upload session, which indicates that the file upload mechanism has been executed +To complete a File Upload Session, which indicates that the file upload mechanism has been executed and did not produce an error, a client issues a ``POST`` to the ``file-upload-session`` link in the -file upload session creation response body. +File Upload Session creation response body. The requests looks like: @@ -712,13 +715,13 @@ If an error occurs, the appropriate ``4xx`` code should be returned, as describe Cancellation and Deletion ~~~~~~~~~~~~~~~~~~~~~~~~~ -A client can cancel an in-progress upload session for a file, or delete a file that has been +A client can cancel an in-progress File Upload Session, or delete a file that has been completely uploaded. In both cases, the client performs this by issuing a ``DELETE`` request to -the file upload session URL of the file they want to delete. +the File Upload Session URL of the file they want to delete. A successful deletion request **MUST** response with a ``204 No Content``. -Once canceled or deleted, a client **MUST NOT** assume that the previous file upload session resource +Once canceled or deleted, a client **MUST NOT** assume that the previous File Upload Session resource or associated file upload mechanisms can be reused. @@ -742,14 +745,14 @@ Session Status -------------- At any time, a client can query the status of a session by issuing a ``GET`` request to the -``session`` :ref:`link ` or ``file-upload-session`` :ref:`link ` -given in the :ref:`session creation response body ` -or :ref:`file upload session creation response body `, +``publishing-session`` :ref:`link ` or ``file-upload-session`` :ref:`link ` +given in the :ref:`session creation response body ` +or :ref:`File Upload Session creation response body `, respectively. -The server will respond to this ``GET`` request with the same :ref:`session response ` -or :ref:`file upload session creation response body `, -that they got when they initially created the upload session or file upload session, +The server will respond to this ``GET`` request with the same :ref:`Publishing Session creation response body ` +or :ref:`File Upload Session creation response body `, +that they got when they initially created the Publishing Session or File Upload Session, except with any changes to ``status``, ``valid-for``, or ``files`` reflected. @@ -760,9 +763,9 @@ Session Extension Servers **MAY** allow clients to extend sessions, but the overall lifetime and number of extensions allowed is left to the server. To extend a session, a client issues a ``POST`` request to the -``session`` :ref:`link ` or ``file-upload-session`` :ref:`link ` -given in the :ref:`session creation response body ` -or :ref:`file upload session creation response body `, respectively. +``publishing-session`` :ref:`link ` or ``file-upload-session`` :ref:`link ` +given in the :ref:`Publishing Session creation response body ` +or :ref:`File Upload Session creation response body `, respectively. The request looks like: @@ -779,9 +782,10 @@ The request looks like: The number of seconds specified is just a suggestion to the server for the number of additional seconds to extend the current session. For example, if the client wants to extend the current session for another hour, ``extend-for`` would be ``3600``. Upon successful extension, the server -will respond with the same :ref:`response ` that they got when they initially -created the upload session, except with any changes to ``status``, ``valid-for``, or ``files`` -reflected. +will respond with the same :ref:`Publishing Session creation response body ` +or :ref:`File Upload Session creation response body `, +that they got when they initially created the Publishing Session or File Upload Session, +except with any changes to ``status``, ``valid-for``, or ``files`` reflected. If the server refuses to extend the session for the requested number of seconds, it still returns a success response, and the ``valid-for`` key will simply include the number of seconds remaining in @@ -795,8 +799,8 @@ Stage Previews The ability to preview staged releases before they are published is an important feature of this PEP, enabling an additional level of last-mile testing before the release is available to the public. Indexes **MAY** provide this functionality through the URL provided in the ``stage`` -sub-key of the :ref:`links key ` returned when the session is created. The ``stage`` -URL can be passed to installers such as ``pip`` by setting the `--extra-index-url +sub-key of the :ref:`links key ` returned when the Publishing Session is created. +The ``stage`` URL can be passed to installers such as ``pip`` by setting the `--extra-index-url `__ flag to this value. Multiple stages can even be previewed by repeating this flag with multiple values. @@ -831,7 +835,7 @@ Required File Upload Mechanisms A client executes this mechanism by submitting a ``POST`` request to the ``file_url`` returned in the ``http-post-bytes`` map of the ``mechanism`` map of the -:ref:`file upload session creation response body ` like: +:ref:`File Upload Session creation response body ` like: .. code-block:: text @@ -842,9 +846,9 @@ A client executes this mechanism by submitting a ``POST`` request to the ``file_ Servers **MAY** support uploading of digital attestations for files (see :pep:`740`). This support will be indicated by inclusion of an ``attestations_url`` key in the ``http-post-bytes`` map of the ``mechanism`` map of the -:ref:`file upload session creation response body `. +:ref:`File Upload Session creation response body `. Attestations **MUST** be uploaded to the ``attestations_url`` before -:ref:`file upload session completion `. +:ref:`File Upload Session completion `. To upload an attestation, a client submits a ``POST`` request to the ``attestations_url`` containing a JSON array of :pep:`attestation objects <740#attestation-objects>` like: @@ -912,8 +916,8 @@ Can I use the upload 2.0 API to reserve a project name? Yes! If you're not ready to upload files to make a release, you can still reserve a project name (assuming of course that the name doesn't already exist). -To do this, :ref:`create a new session `, then :ref:`publish the session -` without uploading any files. While the ``version`` key is required in the JSON +To do this, :ref:`create a new Publishing Session `, then :ref:`publish the session +` without uploading any files. While the ``version`` key is required in the JSON body of the create session request, you can simply use the placeholder version number ``"0.0.0"``. The user that created the session will become the owner of the new project. From 413937723d8e91b315b7b5add3f307f462c44cce Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Wed, 11 Jun 2025 08:13:32 -0400 Subject: [PATCH 21/33] re-flow the doc into (mostly) lines <100 --- peps/pep-0694.rst | 243 +++++++++++++++++++++++++++------------------- 1 file changed, 141 insertions(+), 102 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 5c361d0cb26..c07a09feada 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -101,8 +101,9 @@ The existing upload API lives at a base URL. For PyPI, that URL is currently ``https://upload.pypi.org/legacy/``. Clients performing uploads specify the API they want to call by adding an ``:action`` URL parameter with a value of ``file_upload``. [#fn-action]_ -The legacy API also has a ``protocol_version`` parameter, in theory allowing new versions of the API -to be defined. In practice this has never happened, and the value is always ``1``. +The legacy API also has a ``protocol_version`` parameter, +in theory allowing new versions of the API to be defined. +In practice this has never happened, and the value is always ``1``. Thus, the effective upload API on PyPI is: ``https://upload.pypi.org/legacy/?:action=file_upload&protocol_version=1``. @@ -113,8 +114,8 @@ Encoding The data to be submitted is submitted as a ``POST`` request with the content type of ``multipart/form-data``. This reflects the legacy API's historical nature, which was originally -designed not as an API, but rather as a web form on the initial PyPI implementation, with client code -written to programmatically submit that form. +designed not as an API, but rather as a web form on the initial PyPI implementation, +with client code written to programmatically submit that form. Content @@ -123,8 +124,8 @@ Content Roughly speaking, the metadata contained within the package is submitted as parts where the content disposition is ``form-data``, and the metadata key is the name of the field. The names of these various pieces of metadata are not documented, and they sometimes, but not always match the names -used in the ``METADATA`` files for package artifacts. The case rarely matches, and the ``form-data`` -to ``METADATA`` conversion is inconsistent. +used in the ``METADATA`` files for package artifacts. +The case rarely matches, and the ``form-data`` to ``METADATA`` conversion is inconsistent. The upload artifact file itself is sent as a ``application/octet-stream`` part with the name of ``content``, and if there is a PGP signature attached, then it will be included as a @@ -180,8 +181,8 @@ Content Types ------------- Like :pep:`691`, this PEP proposes that all requests and responses from this upload API will have a -standard content type that describes what the content is, what version of the API it represents, and -what serialization format has been used. +standard content type that describes what the content is, what version of the API it represents, +and what serialization format has been used. This standard request content type applies to all requests *except* for requests to execute a File Upload Mechanism, which will be specified by the documentation for that mechanism. @@ -198,8 +199,8 @@ in the content type; the version number is prefixed with a ``v``. The major API version specified in the ``.meta.api-version`` JSON key of client requests **MUST** match the ``Content-Type`` header for major version. -Unlike :pep:`691`, this PEP does not change the existing *legacy* ``1.0`` upload API in any way, so -servers are required to host the new API described in this PEP at a different endpoint than the +Unlike :pep:`691`, this PEP does not change the existing *legacy* ``1.0`` upload API in any way, +so servers are required to host the new API described in this PEP at a different endpoint than the existing upload API. Since JSON is the only defined request format defined in this PEP, all non-file-upload requests @@ -209,11 +210,12 @@ defined in this PEP **MUST** include a ``Content-Type`` header value of: Similar to :pep:`691`, this PEP also standardizes on using server-driven content negotiation to allow clients to request different versions or serialization formats, which includes the ``format`` -part of the content type. However, since this PEP expects the existing legacy ``1.0`` upload API to -exist at a different endpoint, and this PEP currently only provides for JSON serialization, this -mechanism is not particularly useful. Clients only have a single version and serialization they can -request. However clients **SHOULD** be prepared to handle content negotiation gracefully in the case -that additional formats or versions are added in the future. +part of the content type. However, since this PEP expects the existing legacy ``1.0`` upload API +to exist at a different endpoint, and this PEP currently only provides for JSON serialization, this +mechanism is not particularly useful. +Clients only have a single version and serialization they can request. +However clients **SHOULD** be prepared to handle content negotiation gracefully +in the case that additional formats or versions are added in the future. Unless otherwise specified, all HTTP requests and responses in this document are assumed to include the HTTP header: @@ -306,23 +308,26 @@ The request includes the following top-level keys: The version of the project that this session is attempting to add files to. ``nonce`` (**optional**) - An additional client-side string input to the :ref:`"Publishing Session Token" ` - algorithm. Details are provided below, but if this key is omitted, it is equivalent - to passing the empty string. + An additional client-side string input to the + :ref:`"Publishing Session Token" ` algorithm. + Details are provided below, but if this key is omitted, + it is equivalent to passing the empty string. Upon successful session creation, the server returns a ``201 Created`` response. If an error occurs, the appropriate ``4xx`` code will be returned, as described in the :ref:`session-errors` section. -If a session is created for a project which has no previous release, then the index **MAY** reserve -the project name before the session is published, however it **MUST NOT** be possible to navigate to -that project using the "regular" (i.e. :ref:`unstaged `) access protocols, *until* -the stage is published. If this first-release stage gets canceled, then the index **SHOULD** delete -the project record, as if it were never uploaded. +If a session is created for a project which has no previous release, +then the index **MAY** reserve the project name before the session is published, +however it **MUST NOT** be possible to navigate to that project using +the "regular" (i.e. :ref:`unstaged `) access protocols, +*until* the stage is published. +If this first-release stage gets canceled, +then the index **SHOULD** delete the project record, as if it were never uploaded. -The session is owned by the user that created it, and all subsequent requests **MUST** be performed -with the same credentials, otherwise a ``403 Forbidden`` will be returned on those subsequent -requests. +The session is owned by the user that created it, +and all subsequent requests **MUST** be performed with the same credentials, +otherwise a ``403 Forbidden`` will be returned on those subsequent requests. .. _publishing-session-response: @@ -358,18 +363,18 @@ Besides the ``meta`` key, which has the same format as the request JSON, the suc the following keys: ``links`` - A dictionary mapping :ref:`keys to URLs ` related to this session, the details of - which are provided below. + A dictionary mapping :ref:`keys to URLs ` related to this session, + the details of which are provided below. ``mechanisms`` A list of file-upload mechanisms supported by the server, sorted in server-preferred order. At least one value is required. ``session-token`` - If the index supports :ref:`previewing staged releases `, this key will contain - the unique :ref:`"session token" ` that can be provided to installers in order to - preview the staged release before it's published. If the index does *not* support stage - previewing, this key **MUST** be omitted. + If the index supports :ref:`previewing staged releases `, + this key will contain the unique :ref:`"session token" ` + that can be provided to installers in order to preview the staged release before it's published. + If the index does *not* support stage previewing, this key **MUST** be omitted. ``valid-for`` An integer representing how long, in seconds, until the server itself will expire this session, @@ -378,8 +383,9 @@ the following keys: :ref:`extended `. The session **SHOULD** live at least this much longer unless the client itself has canceled or published the session. Servers **MAY** choose to *increase* this time, but should never *decrease* it, except naturally through the passage of - time. Clients can query the :ref:`session status ` to get time remaining in the - session. + time. + Clients can query the :ref:`session status ` + to get time remaining in the session. ``status`` A string that contains one of ``pending``, ``published``, ``error``, or ``canceled``, @@ -412,10 +418,12 @@ For the ``links`` key in the success JSON, the following sub-keys are valid: the index does not support previewing staged releases, this key **MUST** be omitted. ``session`` - The endpoint where actions for this session can be performed, including :ref:`publishing this - session `, :ref:`canceling and discarding the session `, - :ref:`querying the current session status `, and :ref:`requesting an extension - of the session lifetime ` (*if* the server supports it). + The endpoint where actions for this session can be performed, + including :ref:`publishing this session `, + :ref:`canceling and discarding the session `, + :ref:`querying the current session status `, + and :ref:`requesting an extension of the session lifetime ` + (*if* the server supports it). .. _publishing-session-files: @@ -427,16 +435,21 @@ The ``files`` key contains a mapping from the names of the files uploaded in thi sub-mapping with the following keys: ``status`` - A string with valid values ``pending``, ``processing``, ``complete``, ``error``, and ``canceled``. - If there was an error during upload, then clients should not assume the file is in any usable - state, ``error`` will be returned and it's best to :ref:`cancel or delete ` - the file and start over. This action would remove the file name from the ``files`` key of the + A string with valid values + ``pending``, ``processing``, ``complete``, ``error``, and ``canceled``. + If there was an error during upload, + then clients should not assume the file is in any usable state, + ``error`` will be returned and it's best to + :ref:`cancel or delete ` the file and start over. + This action would remove the file name from the ``files`` key of the :ref:`session status response body `. ``link`` - The *absolute* URL that the client should use to reference this specific file. This URL is used - to retrieve, replace, or delete the :ref:`referenced file `. If a ``nonce`` was - provided, this URL **MUST** be obfuscated with a non-guessable token as described in the + The *absolute* URL that the client should use to reference this specific file. + This URL is used to retrieve, replace, or delete + the :ref:`referenced file `. + If a ``nonce`` was provided, this URL **MUST** be obfuscated + with a non-guessable token as described in the :ref:`Publishing Session Token ` section. ``notices`` @@ -454,8 +467,8 @@ Complete a Publishing Session ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To complete a session and publish the files that have been included in it, a client issues a -``POST`` request to the ``session`` :ref:`link ` given in the :ref:`session creation -response body `. +``POST`` request to the ``session`` :ref:`link ` +given in the :ref:`session creation response body `. The request looks like: @@ -474,9 +487,10 @@ If the server is able to immediately complete the Publishing Session, it may do (for instance, if it needs to do validation that may take longer than reasonable in a single HTTP request), then it may return a ``202 Accepted`` response. -In either case, the server should include a ``Location`` header pointing back to the Publishing Session -status URL, and if the server returned a ``202 Accepted``, the client may poll that URL to -watch for the status to change. +In either case, the server should include a ``Location`` header pointing back to +the Publishing Session status URL, +and if the server returned a ``202 Accepted``, +the client may poll that URL to watch for the status to change. If an error occurs, the appropriate ``4xx`` code should be returned, as described in the :ref:`session-errors` section. @@ -486,11 +500,13 @@ If an error occurs, the appropriate ``4xx`` code should be returned, as describe Cancellation ~~~~~~~~~~~~ -To cancel a Publishing Session, a client issues a ``DELETE`` request to the ``session`` :ref:`link -` given in the :ref:`session creation response body `. The server -then marks the session as canceled, and **SHOULD** purge any data that was uploaded as part of that -session. Future attempts to access that session URL or any of the Publishing Session URLs **MUST** -return a ``404 Not Found``. +To cancel a Publishing Session, a client issues a ``DELETE`` request to +the ``session`` :ref:`link ` +given in the :ref:`session creation response body `. +The server then marks the session as canceled, and **SHOULD** purge any data that was uploaded +as part of that session. +Future attempts to access that session URL or any of the Publishing Session URLs +**MUST** return a ``404 Not Found``. To prevent dangling sessions, servers may also choose to cancel timed-out sessions on their own accord. It is recommended that servers expunge their sessions after no less than a week, but each @@ -509,17 +525,20 @@ This nonce is a string with arbitrary content. The ``nonce`` is optional, and if omitted, is equivalent to providing an empty string. In order to support previewing of staged uploads, the package ``name`` and ``version``, along with -this ``nonce`` are used as input into a hashing algorithm to produce a unique "session token". This -session token is valid for the life of the session (i.e., until it is completed, either by -cancellation or publishing), and can be provided to supporting installers to gain access to the -staged release. - -The use of the ``nonce`` allows clients to decide whether they want to obscure the visibility of -their staged releases or not, and there can be good reasons for either choice. For example, if a CI -system wants to upload some wheels for a new release, and wants to allow independent validation of a -stage before it's published, the client may opt for not including a nonce. On the other hand, if a -client would like to pre-seed a release which it publishes atomically at the time of a public -announcement, that client will likely opt for providing a nonce. +this ``nonce`` are used as input into a hashing algorithm to produce a unique "session token". +This session token is valid for the life of the session +(i.e., until it is completed, either by cancellation or publishing), +and can be provided to supporting installers to gain access to the staged release. + +The use of the ``nonce`` allows clients to decide whether they want to +obscure the visibility of their staged releases or not, +and there can be good reasons for either choice. +For example, if a CI system wants to upload some wheels for a new release, +and wants to allow independent validation of a stage before it's published, +the client may opt for not including a nonce. +On the other hand, if a client would like to pre-seed a release which it publishes atomically +at the time of a public announcement, +that client will likely opt for providing a nonce. The `SHA256 algorithm `_ is used to turn these inputs into a unique token, in the order ``name``, ``version``, ``nonce``, using the @@ -536,12 +555,14 @@ following Python code as an example: h.update(nonce) return h.hexdigest() -It should be evident that if no ``nonce`` is provided in the :ref:`session creation request -`, then the session token is easily guessable from the package name and version -number alone. Clients can elect to omit the ``nonce`` (or set it to the empty string themselves) if -they want to allow previewing from anybody without access to the session token. By providing a -non-empty ``nonce``, clients can elect for security-through-obscurity, but this does not protect -staged files behind any kind of authentication. +It should be evident that if no ``nonce`` is provided in the +:ref:`session creation request `, +then the session token is easily guessable from the package name and version number alone. +Clients can elect to omit the ``nonce`` (or set it to the empty string themselves) +if they want to allow previewing from anybody without access to the session token. +By providing a non-empty ``nonce``, +clients can elect for security-through-obscurity, +but this does not protect staged files behind any kind of authentication. File Upload Session @@ -552,13 +573,14 @@ File Upload Session Create a File Upload Session ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -After creating a Publishing Session, the ``upload`` endpoint from the response's :ref:`session links -` mapping is used to begin the upload of new files into that session. Clients -**MUST** use the provided ``upload`` URL and **MUST NOT** assume there is any pattern or commonality -to those URLs from one session to the next. +After creating a Publishing Session, the ``upload`` endpoint from the response's +:ref:`session links ` mapping +is used to begin the upload of new files into that session. +Clients **MUST** use the provided ``upload`` URL and +**MUST NOT** assume there is any pattern or commonality to those URLs from one session to the next. -To initiate a file upload, a client first sends a ``POST`` request to the ``upload`` URL. The -request looks like: +To initiate a file upload, a client first sends a ``POST`` request to the ``upload`` URL. +The request looks like: .. code-block:: json @@ -613,10 +635,16 @@ the file to be uploaded. These checks may include, but are not limited to: - checking if the contents of the ``metadata``, if provided, are valid. -If the server determines that upload should proceed, it will return a ``202 Accepted`` response, with -the response body below. The :ref:`status ` of the session will also include the filename in the ``files`` mapping. If the server determines the upload cannot proceed, it **MUST** return -a ``409 Conflict``. The server **MAY** allow parallel uploads of files, but is not required to. -If the server cannot proceed with an upload because the ``mechanism`` supplied by the client is not supported it **MUST** return a ``422 Unprocessable Entity``. +If the server determines that upload should proceed, it will return a ``202 Accepted`` response, +with the response body below. +The :ref:`status ` of the session will also include +the filename in the ``files`` mapping. +If the server determines the upload cannot proceed, +it **MUST** return a ``409 Conflict``. +The server **MAY** allow parallel uploads of files, but is not required to. +If the server cannot proceed with an upload because +the ``mechanism`` supplied by the client is not supported +it **MUST** return a ``422 Unprocessable Entity``. .. _file-upload-session-response: @@ -721,8 +749,10 @@ the File Upload Session URL of the file they want to delete. A successful deletion request **MUST** response with a ``204 No Content``. -Once canceled or deleted, a client **MUST NOT** assume that the previous File Upload Session resource -or associated file upload mechanisms can be reused. +Once canceled or deleted, a client **MUST NOT** assume that +the previous File Upload Session resource +or associated file upload mechanisms +can be reused. Replacing a Partially or Fully Uploaded File @@ -731,11 +761,12 @@ Replacing a Partially or Fully Uploaded File To replace a session file, the file upload **MUST** have been previously completed, canceled, or deleted. It is not possible to replace a file if the upload for that file is in-progress. -To replace a session file, clients should :ref:`cancel and delete the in-progress upload -` by issuing a ``DELETE`` to the upload resource URL for the file they want to -replace. After this, the new file upload can be initiated by beginning the entire :ref:`file upload -` sequence over again. This means providing the metadata request again to -retrieve a new upload resource URL. +To replace a session file, clients should +:ref:`cancel and delete the in-progress upload ` by +issuing a ``DELETE`` to the upload resource URL for the file they want to replace. +After this, the new file upload can be initiated by beginning +the entire :ref:`file upload ` sequence over again. +This means providing the metadata request again to retrieve a new upload resource URL. Clients **MUST NOT** assume that the previous upload resource URL can be reused after deletion. @@ -745,12 +776,14 @@ Session Status -------------- At any time, a client can query the status of a session by issuing a ``GET`` request to the -``publishing-session`` :ref:`link ` or ``file-upload-session`` :ref:`link ` +``publishing-session`` :ref:`link ` +or ``file-upload-session`` :ref:`link ` given in the :ref:`session creation response body ` or :ref:`File Upload Session creation response body `, respectively. -The server will respond to this ``GET`` request with the same :ref:`Publishing Session creation response body ` +The server will respond to this ``GET`` request with the same +:ref:`Publishing Session creation response body ` or :ref:`File Upload Session creation response body `, that they got when they initially created the Publishing Session or File Upload Session, except with any changes to ``status``, ``valid-for``, or ``files`` reflected. @@ -763,9 +796,11 @@ Session Extension Servers **MAY** allow clients to extend sessions, but the overall lifetime and number of extensions allowed is left to the server. To extend a session, a client issues a ``POST`` request to the -``publishing-session`` :ref:`link ` or ``file-upload-session`` :ref:`link ` +``publishing-session`` :ref:`link ` +or ``file-upload-session`` :ref:`link ` given in the :ref:`Publishing Session creation response body ` -or :ref:`File Upload Session creation response body `, respectively. +or :ref:`File Upload Session creation response body `, +respectively. The request looks like: @@ -782,7 +817,8 @@ The request looks like: The number of seconds specified is just a suggestion to the server for the number of additional seconds to extend the current session. For example, if the client wants to extend the current session for another hour, ``extend-for`` would be ``3600``. Upon successful extension, the server -will respond with the same :ref:`Publishing Session creation response body ` +will respond with the same +:ref:`Publishing Session creation response body ` or :ref:`File Upload Session creation response body `, that they got when they initially created the Publishing Session or File Upload Session, except with any changes to ``status``, ``valid-for``, or ``files`` reflected. @@ -799,7 +835,8 @@ Stage Previews The ability to preview staged releases before they are published is an important feature of this PEP, enabling an additional level of last-mile testing before the release is available to the public. Indexes **MAY** provide this functionality through the URL provided in the ``stage`` -sub-key of the :ref:`links key ` returned when the Publishing Session is created. +sub-key of the :ref:`links key ` returned when +the Publishing Session is created. The ``stage`` URL can be passed to installers such as ``pip`` by setting the `--extra-index-url `__ flag to this value. Multiple stages can even be previewed by repeating this flag with multiple values. @@ -833,8 +870,8 @@ Required File Upload Mechanisms ``http-post-bytes`` +++++++++++++++++++ -A client executes this mechanism by submitting a ``POST`` request to the ``file_url`` returned in the -``http-post-bytes`` map of the ``mechanism`` map of the +A client executes this mechanism by submitting a ``POST`` request to the ``file_url`` +returned in the ``http-post-bytes`` map of the ``mechanism`` map of the :ref:`File Upload Session creation response body ` like: .. code-block:: text @@ -906,8 +943,8 @@ Does this mean PyPI is planning to drop support for the existing upload API? At this time PyPI does not have any specific plans to drop support for the existing upload API. Unlike with :pep:`691` there are significant benefits to doing so, so it is likely that support for -the legacy upload API to be (responsibly) deprecated and removed at some point in the future. Such -future deprecation planning is explicitly out of scope for *this* PEP. +the legacy upload API to be (responsibly) deprecated and removed at some point in the future. +Such future deprecation planning is explicitly out of scope for *this* PEP. Can I use the upload 2.0 API to reserve a project name? @@ -916,9 +953,11 @@ Can I use the upload 2.0 API to reserve a project name? Yes! If you're not ready to upload files to make a release, you can still reserve a project name (assuming of course that the name doesn't already exist). -To do this, :ref:`create a new Publishing Session `, then :ref:`publish the session -` without uploading any files. While the ``version`` key is required in the JSON -body of the create session request, you can simply use the placeholder version number ``"0.0.0"``. +To do this, +:ref:`create a new Publishing Session `, +then :ref:`publish the session ` without uploading any files. +While the ``version`` key is required in the JSON body of the create session request, +you can simply use the placeholder version number ``"0.0.0"``. The user that created the session will become the owner of the new project. From 98e7929bb8239d8c983a92d4ef66b23224de5c8d Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 11 Jul 2025 08:53:18 -0400 Subject: [PATCH 22/33] Apply suggestions from code review Co-authored-by: Barry Warsaw --- peps/pep-0694.rst | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index c07a09feada..9ab96bad64a 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -17,7 +17,7 @@ Abstract This PEP proposes an extensible API for uploading files to a Python package index such as PyPI. Along with standardization, the upload API provides additional useful features such as support for: -* an publishing session, which can be used to simultaneously publish +* a publishing session, which can be used to simultaneously publish all wheels in a package release; * "staging" a release, which can be used to test uploads before publicly publishing them, @@ -620,7 +620,7 @@ Besides the standard ``meta`` key, the request JSON has the following additional This mechanism **SHOULD** be chosen from the list of mechanisms advertised in the :ref:`Publishing Session response body `. A client **MAY** send a mechanism that is not advertised in cases where server operators have - documented a new or up-coming mechanism that is available for use on a "pre-release" basis. + documented a new or upcoming mechanism that is available for use on a "pre-release" basis. ``metadata`` (**optional**) If given, this is a string value containing the file's `core metadata @@ -684,7 +684,7 @@ the following keys: A mapping containing the necessary details for the supported mechanism as negotiated by the client and server. This mapping **MUST** contain a key ``identifier`` which maps to - the identifier string for the File Upload Mechanism. + the identifier string for the chosen File Upload Mechanism. .. _file-upload-session-links: @@ -747,7 +747,7 @@ A client can cancel an in-progress File Upload Session, or delete a file that ha completely uploaded. In both cases, the client performs this by issuing a ``DELETE`` request to the File Upload Session URL of the file they want to delete. -A successful deletion request **MUST** response with a ``204 No Content``. +A successful deletion request **MUST** respond with a ``204 No Content``. Once canceled or deleted, a client **MUST NOT** assume that the previous File Upload Session resource @@ -841,11 +841,11 @@ The ``stage`` URL can be passed to installers such as ``pip`` by setting the `-- `__ flag to this value. Multiple stages can even be previewed by repeating this flag with multiple values. -In either case, the index will return views that expose the staged releases to the installer tool, +If supported, the index will return views that expose the staged releases to the installer tool, making them available to download and install into virtual environments built for that last-mile -testing. The former option allows for existing installers to preview staged releases with no -changes, although perhaps in a less user-friendly way. The latter option can be a better user -experience, but the details of this are left to installer tool maintainers. +testing. This option allows existing installers to preview staged releases with no +changes to the installer tool required. +The details of this user experience are left to installer tool maintainers. .. _file-upload-mechanisms: @@ -870,6 +870,8 @@ Required File Upload Mechanisms ``http-post-bytes`` +++++++++++++++++++ +Upload API version 2.0 compliant servers **MUST** support the ``http-post-bytes`` mechanism. + A client executes this mechanism by submitting a ``POST`` request to the ``file_url`` returned in the ``http-post-bytes`` map of the ``mechanism`` map of the :ref:`File Upload Session creation response body ` like: @@ -930,7 +932,7 @@ File Upload Mechanism string Server Operator Mechanism description ``vnd-madscience-quantumentanglement`` Mad Science Labs Upload via quantum entanglement ====================================== ================ ========================================================================= -If a server intends to match the behavior of another server's implementation, it **MAY** respond +If a server intends to precisely match the behavior of another server's implementation, it **MAY** respond with that implementation's file upload mechanism name. From 1ff2415c64cc8fb5cf173b5edd8625d17e11f261 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 11 Jul 2025 09:00:35 -0400 Subject: [PATCH 23/33] re-order response codes in session status --- peps/pep-0694.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 9ab96bad64a..32290b56b1d 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -639,12 +639,12 @@ If the server determines that upload should proceed, it will return a ``202 Acce with the response body below. The :ref:`status ` of the session will also include the filename in the ``files`` mapping. -If the server determines the upload cannot proceed, -it **MUST** return a ``409 Conflict``. -The server **MAY** allow parallel uploads of files, but is not required to. If the server cannot proceed with an upload because the ``mechanism`` supplied by the client is not supported it **MUST** return a ``422 Unprocessable Entity``. +If the server determines the upload cannot proceed, +it **MUST** return a ``409 Conflict``. +The server **MAY** allow parallel uploads of files, but is not required to. .. _file-upload-session-response: From e946d41e93dc9c9ecca8410963c62c97b43d9f56 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Fri, 11 Jul 2025 12:01:17 -0700 Subject: [PATCH 24/33] Update peps/pep-0694.rst Co-authored-by: Ee Durbin --- peps/pep-0694.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 32290b56b1d..6e51378291a 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -860,7 +860,7 @@ Each major version of the Upload API **MUST** specify at least one required File New required mechanisms **MUST NOT** be added and existing required mechanisms **MUST NOT** be removed -without an update to the :ref:`major version `. +without an update to the :ref:`major version `. Any server-specific or experimental mechanisms added or removed **MUST NOT** change the major or minor version number of this specification. .. _required-file-upload-mechanisms: From 191c9fea032d65817f56fe673942fbe747453002 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 25 Jul 2025 06:22:30 -0400 Subject: [PATCH 25/33] Update verbiage in rationale section. --- peps/pep-0694.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 6e51378291a..540b6eee176 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -79,13 +79,13 @@ In addition, there are a number of major issues with the legacy API: * Creation of new projects requires the uploading of at least one file, leading to "stub" uploads to claim a project namespace. -The new upload API proposed in this PEP provides an immediate solution to many of these problems, -and defines a flexible mechanism for future support of the other problems by extension. -Indexes implementing this API will provide better error reporting, -better release testing experience, +The new upload API proposed in this PEP provides ways to solve all of these problems, +either directly or through an extensible approach, +allowing servers to implement features such as resumable and parallel uploads. +This upload API this PEP proposes provides +better error reporting, +a more robust release testing experience, and atomic and simultaneous publishing of all release artifacts. -In the future indexes can implement resumable and parallel uploads via extensions. - Legacy API ========== From dcb682b503c397e9b1149f827f3647438eddbc5e Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 25 Jul 2025 06:39:43 -0400 Subject: [PATCH 26/33] specify that Retry-After header should be used to manage polling of the file upload session status --- peps/pep-0694.rst | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 540b6eee176..5f8513aa5fd 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -672,6 +672,8 @@ The successful response includes the following: } } +A `Retry-After` response header **MUST** be present +to indicate to clients when they should next poll for an updated status. Besides the ``meta`` key, which has the same format as the request JSON, the success response has the following keys: @@ -731,8 +733,14 @@ request), then it may return a ``202 Accepted`` response and set the status of the File Upload Session to ``processing``. In either case, the server should include a ``Location`` header pointing back to the File Upload -Session status URL, and if the server returned a ``202 Accepted``, the client may poll that URL to -watch for the status to change. +Session status URL. + +Servers **MUST** allow clients to poll the File Upload Session status URL +to watch for the status to change. +If the server responds with a ``202 Accepted``, +clients may poll the File Upload Session status URL to watch for the status to change. +Clients **SHOULD** respect the `Retry-After` header value +of the File Upload Session status response. If an error occurs, the appropriate ``4xx`` code should be returned, as described in the :ref:`session-errors` section. From acbc1b141a8dc5c30215487cafab43e75eb0694e Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 25 Jul 2025 06:54:09 -0400 Subject: [PATCH 27/33] stage previews are optionally supported by this PEP, and thus are not an open question --- peps/pep-0694.rst | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 5f8513aa5fd..b9153fbada5 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -972,20 +972,6 @@ you can simply use the placeholder version number ``"0.0.0"``. The user that created the session will become the owner of the new project. -Open Questions -============== - -Defer Stage Previews --------------------- - -:ref:`Stage previews ` are an important and useful feature for testing new version -wheel uploads before they are published. They'd allow us to effectively decommission -``test.pypi.org``, which has well-known deficiencies. - -However, the ability to preview stages before they're published does complicate the protocol and -this proposal. We could defer this feature for later, although if we do, we should still keep the -optional ``nonce`` for token generation, in order to be easily future proof. - .. rubric:: Footnotes .. [#fn-action] Obsolete ``:action`` values ``submit``, ``submit_pkg_info``, and ``doc_upload`` are From fd0e6f573bb9a1a986426c31fc2e719516fb1f2c Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 25 Jul 2025 07:06:19 -0400 Subject: [PATCH 28/33] valid-for -> expires-at --- peps/pep-0694.rst | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index b9153fbada5..a9a1d65fed1 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -350,7 +350,7 @@ The successful response includes the following content: }, "mechanisms": ["http-post-bytes"], "session-token": "", - "valid-for": 604800, + "expires-at": "2025-08-01T12:00:00Z", "status": "pending", "files": {}, "notices": [ @@ -376,16 +376,14 @@ the following keys: that can be provided to installers in order to preview the staged release before it's published. If the index does *not* support stage previewing, this key **MUST** be omitted. -``valid-for`` - An integer representing how long, in seconds, until the server itself will expire this session, +``expires-at`` + An ISO8601 formatted timestamp string representing when the server will expire this session, and thus all of its content, including any uploaded files and the URL links related to the - session. This value is roughly relative to the time at which the session was created or - :ref:`extended `. The session **SHOULD** live at least this much longer + session. The session **SHOULD** remain active until at least this time unless the client itself has canceled or published the session. Servers **MAY** choose to - *increase* this time, but should never *decrease* it, except naturally through the passage of - time. + extend this expiration time, but should never move it earlier. Clients can query the :ref:`session status ` - to get time remaining in the session. + to get the current expiration time of the session. ``status`` A string that contains one of ``pending``, ``published``, ``error``, or ``canceled``, @@ -664,7 +662,7 @@ The successful response includes the following: "file-upload-session": "..." }, "status": "pending", - "valid-for": 3600, + "expires-at": "2025-08-01T13:00:00Z", "mechanism": { "identifier": "http-post-bytes", "file_url": "...", @@ -682,6 +680,16 @@ the following keys: A dictionary mapping :ref:`keys to URLs ` related to this session, the details of which are provided below. +``status`` + A string with valid values ``pending``, ``processing``, ``complete``, ``error``, and ``canceled`` + indicating the current state of the File Upload Session. + +``expires-at`` + An ISO8601 formatted timestamp string representing when the server will expire this File Upload Session. + The session **SHOULD** remain active until at least this time + unless the client cancels or completes it. Servers **MAY** choose to + extend this expiration time, but should never move it earlier. + ``mechanism`` A mapping containing the necessary details for the supported mechanism as negotiated by the client and server. @@ -794,7 +802,7 @@ The server will respond to this ``GET`` request with the same :ref:`Publishing Session creation response body ` or :ref:`File Upload Session creation response body `, that they got when they initially created the Publishing Session or File Upload Session, -except with any changes to ``status``, ``valid-for``, or ``files`` reflected. +except with any changes to ``status``, ``expires-at``, or ``files`` reflected. .. _session-extension: @@ -829,11 +837,11 @@ will respond with the same :ref:`Publishing Session creation response body ` or :ref:`File Upload Session creation response body `, that they got when they initially created the Publishing Session or File Upload Session, -except with any changes to ``status``, ``valid-for``, or ``files`` reflected. +except with any changes to ``status``, ``expires-at``, or ``files`` reflected. If the server refuses to extend the session for the requested number of seconds, it still returns a -success response, and the ``valid-for`` key will simply include the number of seconds remaining in -the current session. +success response, and the ``expires-at`` key will simply reflect the current expiration time of +the session. .. _staged-preview: From 472a796739afc39f3fb9078d1cc48c8ee4cd3302 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 25 Jul 2025 07:32:59 -0400 Subject: [PATCH 29/33] clarify who owns what version specifiers --- peps/pep-0694.rst | 43 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 34 insertions(+), 9 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index a9a1d65fed1..30d0fd7306c 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -171,11 +171,19 @@ these steps: Versioning ---------- -This PEP uses the same ``MAJOR.MINOR`` versioning system as used in :pep:`691`, but it is otherwise -independently versioned. The legacy API is considered by this PEP to be version ``1.0``, but this -PEP does not modify the legacy API in any way. +This PEP uses the same ``MAJOR.MINOR`` versioning system as used in :pep:`691`, +but it is otherwise independently versioned. +The legacy API is considered by this PEP to be version ``1.0``, +but this PEP does not modify the legacy API in any way. -The API proposed in this PEP therefor has the version number ``2.0``. +The API proposed in this PEP therefore has the version number ``2.0``. + +Both major and minor version numbers of the Upload API +**MUST** only be changed through the PEP process. +Index operators and implementers **MUST NOT** advertise or implement +new API versions without an approved PEP. +This ensures consistency across all implementations +and prevents fragmentation of the ecosystem. Content Types ------------- @@ -209,14 +217,19 @@ defined in this PEP **MUST** include a ``Content-Type`` header value of: - ``application/vnd.pypi.upload.v2+json``. Similar to :pep:`691`, this PEP also standardizes on using server-driven content negotiation to -allow clients to request different versions or serialization formats, which includes the ``format`` -part of the content type. However, since this PEP expects the existing legacy ``1.0`` upload API -to exist at a different endpoint, and this PEP currently only provides for JSON serialization, this -mechanism is not particularly useful. +allow clients to request different versions or serialization formats, +which includes the ``format`` part of the content type. +However, since this PEP expects the existing legacy ``1.0`` upload API +to exist at a different endpoint, +and this PEP currently only provides for JSON serialization, +this mechanism is not particularly useful. Clients only have a single version and serialization they can request. However clients **SHOULD** be prepared to handle content negotiation gracefully in the case that additional formats or versions are added in the future. +Servers **MUST NOT** advertise support for API versions beyond those defined in approved PEPs. +Any new versions or formats require standardization through a new PEP. + Unless otherwise specified, all HTTP requests and responses in this document are assumed to include the HTTP header: @@ -876,7 +889,9 @@ Each major version of the Upload API **MUST** specify at least one required File New required mechanisms **MUST NOT** be added and existing required mechanisms **MUST NOT** be removed -without an update to the :ref:`major version `. Any server-specific or experimental mechanisms added or removed **MUST NOT** change the major or minor version number of this specification. +without an update to the :ref:`major version `. +Any server-specific or experimental mechanisms added or removed +**MUST NOT** change the major or minor version number of this specification. .. _required-file-upload-mechanisms: @@ -936,15 +951,25 @@ and contain only alphanumeric characters ``[a-z0-9]``. The ``implementation identifier`` **SHOULD** concisely describe the underlying implementation and contain only alphanumeric characters ``[a-z0-9]`` and ``-``. +When server operators need to make breaking changes to their upload mechanisms, +they **SHOULD** create a new mechanism identifier rather than modifying the existing one. +The recommended pattern is to append a version suffix like ``-v1``, ``-v2``, etc. +to the implementation identifier. +This allows clients to explicitly opt into new versions while maintaining +backward compatibility with existing clients. + For example: ====================================== ================ ========================================================================= File Upload Mechanism string Server Operator Mechanism description ====================================== ================ ========================================================================= ``vnd-pypi-s3multipart-presigned`` PyPI S3 multipart upload via pre-signed URL +``vnd-pypi-s3multipart-presigned-v2`` PyPI S3 multipart upload via pre-signed URL version 2 ``vnd-pypi-http-fetch`` PyPI File delivered by instructing server to fetch from a URL via HTTP request ``vnd-acmecorp-http-fetch`` Acme Corp File delivered by instructing server to fetch from a URL via HTTP request ``vnd-acmecorp-postal`` Acme Corp File delivered via postal mail +``vnd-widgetinc-stream-v1`` Widget Inc. Streaming upload protocol version 1 +``vnd-widgetinc-stream-v2`` Widget Inc. Streaming upload protocol version 2 ``vnd-madscience-quantumentanglement`` Mad Science Labs Upload via quantum entanglement ====================================== ================ ========================================================================= From 9010030b809e30bf9375a764498a838379bb636e Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 25 Jul 2025 07:39:48 -0400 Subject: [PATCH 30/33] lint --- peps/pep-0694.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 30d0fd7306c..f063c382627 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -683,7 +683,7 @@ The successful response includes the following: } } -A `Retry-After` response header **MUST** be present +A ``Retry-After`` response header **MUST** be present to indicate to clients when they should next poll for an updated status. Besides the ``meta`` key, which has the same format as the request JSON, the success response has @@ -760,7 +760,7 @@ Servers **MUST** allow clients to poll the File Upload Session status URL to watch for the status to change. If the server responds with a ``202 Accepted``, clients may poll the File Upload Session status URL to watch for the status to change. -Clients **SHOULD** respect the `Retry-After` header value +Clients **SHOULD** respect the ``Retry-After`` header value of the File Upload Session status response. If an error occurs, the appropriate ``4xx`` code should be returned, as described in the From 7286ba53e31bfc7744d833beae592b12d2981f0b Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 25 Jul 2025 08:29:38 -0400 Subject: [PATCH 31/33] say _something_ about authentication for upload 2.0 --- peps/pep-0694.rst | 35 +++++++++++++++++++++++++++++++---- 1 file changed, 31 insertions(+), 4 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index f063c382627..e5e1d6d7d30 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -135,10 +135,14 @@ The upload artifact file itself is sent as a ``application/octet-stream`` part w Authentication -------------- -Upload authentication is also not standardized. On PyPI, authentication is through `API tokens -`__ or `Trusted Publisher (OpenID Connect) -`__. Other indexes may support different authentication -methods. +Upload authentication is also not standardized. + +PyPI uses HTTP Basic Authentication +with `API tokens `__ as the password +and the username ``__token__``. +`Trusted Publishers `__ +authenticate via OpenID Connect and receive short-lived API tokens +that are used in the same way. .. _spec: @@ -247,6 +251,24 @@ the url structure of a domain. For example, the root endpoint could be The choice of the root endpoint is left up to the index operator. + +Authentication for Upload 2.0 API +---------------------------------- + +All endpoints in this specification **MUST** use standard HTTP authentication +mechanisms as defined in :rfc:`7235`. + +Authentication follows the standard HTTP pattern: + +- Servers use the ``WWW-Authenticate`` response header when authentication is required +- Clients provide credentials via the ``Authorization`` request header +- ``401 Unauthorized`` indicates missing or invalid authentication +- ``403 Forbidden`` indicates insufficient permissions + +The specific authentication schemes (e.g., Bearer, Basic, Digest) +are determined by the index operator. + + .. _session-errors: Errors @@ -903,6 +925,11 @@ Required File Upload Mechanisms Upload API version 2.0 compliant servers **MUST** support the ``http-post-bytes`` mechanism. +This mechanism **MUST** use the same authentication scheme as the +:ref:`Root `, +:ref:`Publishing Session `, +and :ref:`File Upload Session ` endpoints. + A client executes this mechanism by submitting a ``POST`` request to the ``file_url`` returned in the ``http-post-bytes`` map of the ``mechanism`` map of the :ref:`File Upload Session creation response body ` like: From 8896963eb2410752c5ceab987bbad7594073f74c Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Fri, 25 Jul 2025 12:56:35 -0400 Subject: [PATCH 32/33] simplify per code review --- peps/pep-0694.rst | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index e5e1d6d7d30..10dc5ec8625 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -925,10 +925,8 @@ Required File Upload Mechanisms Upload API version 2.0 compliant servers **MUST** support the ``http-post-bytes`` mechanism. -This mechanism **MUST** use the same authentication scheme as the -:ref:`Root `, -:ref:`Publishing Session `, -and :ref:`File Upload Session ` endpoints. +This mechanism **MUST** use the same authentication scheme as +the rest of the Upload 2.0 protocol endpoints. A client executes this mechanism by submitting a ``POST`` request to the ``file_url`` returned in the ``http-post-bytes`` map of the ``mechanism`` map of the From eceb7b5a1d2f6a13fec30d5fce38c7a6de8468d2 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Mon, 28 Jul 2025 14:54:27 -0400 Subject: [PATCH 33/33] Restore Open Questions section with notes on deferred extensions Add documentation about webhook notifications and capabilities extensions that were discussed but deferred to avoid complexity and ecosystem fragmentation during initial Upload 2.0 rollout. --- peps/pep-0694.rst | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 10dc5ec8625..1bf63c693cc 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -1030,6 +1030,26 @@ you can simply use the placeholder version number ``"0.0.0"``. The user that created the session will become the owner of the new project. +Open Questions +============== + +Extensions to the Upload 2.0 Protocol +------------------------------------- + +Features such as asynchronous webhook notifications for completion of upload processing +were discussed during review of this PEP. +The concept of a capabilities extension for the upload protocol was discussed, +which would allow implementers to advertise support for optional features +such as asynchronous notifications or webhooks. + +This idea was left open due to the complexity that would arise in designing +such an extension protocol and ensuring that it did not cause excessive +fracturing of the ecosystem as Upload 2.0 is rolled out. + +Future revisions to the upload protocol should explore such extensions +as experience is gained operating Upload 2.0. + + .. rubric:: Footnotes .. [#fn-action] Obsolete ``:action`` values ``submit``, ``submit_pkg_info``, and ``doc_upload`` are