Clarify auth flows (#15)

* Clarify server variable usage for data plane API

* Use overlays to more clearly describe authentication flows

* Apply suggestions from automated review

* Visual indicator of variable

* 'Sign up for emails' is no longer relevant after API GA

* Apply suggestions from code review

Co-authored-by: Michele Cyran <[email protected]>

* Merge commits from review in origin branch

* Unwanted file

* Fix links

* Fix links to other APIs

* Minor edits

* Minor edits

* Accept suggestions from automated review

---------

Co-authored-by: Michele Cyran <[email protected]>

Author: kbatuigas

cloud-controlplane/x-topics/quickstart.md

@@ -1,7 +1,7 @@
-The following steps describe how to authenticate with the Control API and create a new Redpanda cluster. For more information on the Control Plane API, see the Cloud API Overview.
+The following steps describe how to authenticate with the Control Plane API and create a new Redpanda cluster. For more information on the Control Plane API, see the [Cloud API Overview](#topic-cloud-api-overview).
 
 > **Note:** Redpanda Cloud uses a control plane and data plane architecture. 
-To see the available endpoints for managing resources within your clusters, such as topics, users, access control lists (ACLs), and connectors, see the Data Plane API Reference.
+To view the available endpoints for managing resources within your clusters, such as topics, users, access control lists (ACLs), and connectors, see the [Data Plane API Reference](/api/doc/cloud-dataplane).
 
 ## Requirements
 
@@ -14,27 +14,39 @@ To use the Control Plane API:
 
 ### Authenticate to the API from API Explorer
 
-To make API requests in your browser, you must obtain an access token. You can do so by clicking **Get token** on the API endpoint you want to call.
+The API Explorer lets you interact with the API directly from the documentation. You can quickly explore available endpoints and try requests without setting up your own test environment.
 
-If you successfully retrieve an access token, it is valid for one hour. You can use the same token in requests to both Control Plane and Data Plane API endpoints, for as long as the token is valid.
+To make Cloud API requests in your browser, you must obtain an access token. You can do so by clicking **Get token** on the API endpoint you want to call.
 
-## Create a new cluster
+If you successfully retrieve an access token, it is valid for one hour. You can use the same token in requests to both Control Plane and Data Plane API endpoints, for as long as the token is valid.
 
 > **Warning:** API requests from the API Explorer are executed against your actual environment and data, not a sandbox.
 
+## Create a new cluster
+
 ### BYOC or Dedicated
 
-1. In the page header, click **API Explorer**.
-1. On the **Choose an operation** dropdown, select **Create resource group**.
+1. In the subheader, open **API Explorer**.
+
+1. In the **Choose an operation** dropdown, select **Create resource group**.
+
 1. Click **Get token**. You may be prompted to log in to the Redpanda Cloud UI. After you log in, the browser automatically redirects you back to the Create resource group endpoint in the API Explorer.
+
 1. Prepare your Create resource group request.
-  1. Under **Body**, click **+ Add** and provide a name for your resource group. A resource group is a container to organize your Redpanda Cloud resources, such as clusters and networks.
-  1. Click **Send request**. If successful, the response returns a resource group ID. Pass this ID when you make a Create network request. 
-1. On the dropdown, select **Create network**.
+
+    1. Under **Body**, click **+ Add** and provide a name for your resource group. A resource group is a container to organize your Redpanda Cloud resources, such as clusters and networks.
+
+    1. Click **Send request**. If successful, the response returns a resource group ID. Pass this ID when you make a Create network request. 
+
+1. In the dropdown, select **Create network**.
+
 1. Prepare your Create network request.
-  1. Include the ID of the resource group you created in the previous step. 
-  1. Click **Send request**. Note that this endpoint returns a long-running operation. The response returns a network ID in `metadata.network_id`. Pass this ID when you call the Create Cluster endpoint.To check the operation state, make a **Get operation** request with the `operation.id`.  
+
+    1. Include the ID of the resource group you created in the previous step. 
+    1. Click **Send request**. Note that this endpoint returns a long-running operation. The response returns a network ID in `metadata.network_id`. Pass this ID when you call the Create Cluster endpoint. To check the operation state, make a **Get operation** request with the `operation.id`.  
+
 1. When the Create network operation is complete, make a Create cluster request. Use the resource group and network IDs you just created. Note that this endpoint also returns a long-running operation.
+
 1. For BYOC, run `rpk cloud byoc` in the shell, passing the `metadata.cluster_id` from the Create cluster response as a flag:
 
    **AWS:**
@@ -52,27 +64,34 @@ If you successfully retrieve an access token, it is valid for one hour. You can
 
 ### Serverless
 
-1. In the page header, click **API Explorer**.
-1. On the **Choose an operation** dropdown, select **Create resource group**.
+1. In the subheader, open **API Explorer**.
+
+1. In the **Choose an operation** dropdown, select **Create resource group**.
+
 1. Click **Get token**. You may be prompted to log in to the Redpanda Cloud UI. After you log in, the browser automatically redirects you back to the Create resource group endpoint in the API Explorer.
+
 1. Prepare your Create resource group request.
-  1. Under **Body**, click **+ Add** and provide a name for your resource group. A resource group is a container to organize your Redpanda Cloud resources, such as clusters and networks.
-  1. Click **Send request**. If successful, the response returns a resource group ID. Pass this ID later when you make a Create Serverless cluster request. 
-1. On the dropdown, select **Create Serverless cluster**. 
+
+    1. Under **Body**, click **+ Add** and provide a name for your resource group. A resource group is a container to organize your Redpanda Cloud resources, such as clusters and networks.
+    1. Click **Send request**. If successful, the response returns a resource group ID. Pass this ID later when you make a Create Serverless cluster request.
+
+1. In the dropdown, select **Create Serverless cluster**.
+
 1. Prepare your Create Serverless cluster request.
-  1. Make a Get Serverless Regions request to see available regions.
-  1. In the request body, use the resource group ID and desired cloud region.
-  1. Click **Send request**. Note that this endpoint returns a long-running operation. The response returns a Serverless cluster ID in `metadata.cluster_id`. To check the operation state, make a **Get operation** request with the `operation.id`.
+
+    1. Make a Get Serverless Regions request to see available regions.
+    1. In the request body, use the resource group ID and desired cloud region.
+    1. Click **Send request**. Note that this endpoint returns a long-running operation. The response returns a Serverless cluster ID in `metadata.cluster_id`. To check the operation state, make a **Get operation** request with the `operation.id`.
 
 ## Next steps: try the Data Plane APIs
 
 1. Retrieve your cluster's data plane API URL by making a **Get cluster** (BYOC, Dedicated) or **Get Serverless cluster** (Serverless) request in the API Explorer.
-1. Save the value of `dataplane_api.url` from the response body.
+1. From the value of `dataplane_api.url` in the response body, save the subdomain (the part between ``https://` and `.cloud.redpanda.com`).
 1. From the **Redpanda APIs** selector, go to **Cloud Data Plane API**.
 1. Select an operation, for example **Create topic** or **List users**. 
 1. In the URL field, add the data plane API URL. You can now make Data Plane API requests to your target cluster.
 
-See also: [Data Plane API Quickstart](https://docs.redpanda.com/api/doc/cloud-dataplane/topic/topic-quickstart)
+See also: [Data Plane API Quickstart](/cloud-dataplane/topic/topic-quickstart)
 
 ## Suggested reading
 

cloud-dataplane/cloud-dataplane.yaml

@@ -4976,14 +4976,15 @@ security:
   - auth0: []
 servers:
   - description: Data Plane API
-    url: 'https://{dataplane_api_url}.cloud.redpanda.com'
+    url: '{dataplane_api_url}'
     variables:
       dataplane_api_url:
-        default: "dataplane_api.url"
+        default: https://{dataplane_api_url}
         description: |-
-          Find the Data Plane API base URL of a cluster by calling the Get Cluster endpoint of the Control Plane API. The dataplane_api.url field is returned in the response body.<br><br>
-          Example (Dedicated): "https://api-a4cb21.ck09mi9c4vs17hng9gig.fmc.prd.cloud.redpanda.com"<br>
+          Find the Data Plane API base URL of a cluster by calling the Get Cluster endpoint of the Control Plane API. The dataplane_api.url field is returned in the response body.
+
           Example (BYOC): "https://api-a4cb21.ck09mi9c4vs17hng9gig.byoc.prd.cloud.redpanda.com"
+          Example (Dedicated): "https://api-a4cb21.ck09mi9c4vs17hng9gig.fmc.prd.cloud.redpanda.com"
 tags:
   - description: Manage Redpanda access control lists (ACLs). See [Redpanda Cloud Authorization](https://docs.redpanda.com/redpanda-cloud/security/authorization/cloud-authorization/) for more information.
     name: Redpanda ACLs

cloud-dataplane/overlays/update-dataplane-server-config.yaml

@@ -0,0 +1,20 @@
+# Overlay to update server configuration for Bump API Explorer compatibility.
+# This updates the server URL, variable description to better match usage in API Explorer.
+
+overlay: 1.0.0
+info:
+  title: Update Server Configuration
+  version: 1.0.0
+actions:
+  # Update only the server URL template
+  - target: "$.servers[0].url"
+    update: "https://{dataplane_api_subdomain}.cloud.redpanda.com"
+  # Update only the variables section
+  - target: "$.servers[0].variables"
+    update:
+      dataplane_api_subdomain:
+        default: "{dataplane_api_subdomain}"
+        description: |-
+          From the `dataplane_api.url` value in the Control Plane API response, extract the subdomain (the part between `https://` and `.cloud.redpanda.com`). Enter this value in the Data Plane API URL field.
+          
+          Example: If the URL is `https://api-a4cb21.ck09mi9c4vs17hng9gig.fmc.prd.cloud.redpanda.com`, enter `api-a4cb21.ck09mi9c4vs17hng9gig.fmc.prd`.

cloud-dataplane/x-topics/error-and-status-codes.md

@@ -4,6 +4,7 @@ Example request:
 
 ```bash
 curl https://<dataplane_api_url>/topics | jq
+```
 
 Example response:
 

cloud-dataplane/x-topics/quickstart.md

@@ -1,6 +1,6 @@
-The following steps describe how to authenticate with the Data Plane APIs and create a new topic. For more information on the Data Plane APIs, see the Cloud API Overview.
+The following steps describe how to authenticate with the Data Plane APIs and create a new topic. For more information on the Data Plane APIs, see the [Cloud API Overview](#topic-cloud-api-overview).
 
-> **Note:** Redpanda Cloud uses a control plane and data plane architecture. To see the available endpoints for managing your clusters, networks, and resource groups, see the Control Plane API Reference.
+> **Note:** Redpanda Cloud uses a control plane and data plane architecture. To view the available endpoints for managing your clusters, networks, and resource groups, see the [Control Plane API Reference](/api/doc/cloud-controlplane).
 
 ## Requirements
 
@@ -12,24 +12,33 @@ To use the Data Plane APIs:
 
 ### Authenticate to the API from API Explorer
 
-To make API requests in your browser, you must obtain an access token. You can do so by clicking **Get token** on the API endpoint you want to call.
+The API Explorer lets you interact with the API directly from the documentation. You can quickly explore available endpoints and try requests without setting up your own test environment.
+
+To make Cloud API requests in your browser, you must obtain an access token. You can do so by clicking **Get token** on the API endpoint you want to call.
 
 If you successfully retrieve an access token, it is valid for one hour. You can use the same token in requests to both Control Plane and Data Plane API endpoints, for as long as the token is valid.
 
+> **Warning:** API requests from the API Explorer are executed against your actual environment and data, not a sandbox.
+
 ## Create a topic
 
-> **Warning:** API requests from this page are executed against your actual environment and data, not a sandbox.
+1. In the subheader, open **API Explorer**.
+
+1. If you don't already have the data plane API URL for your target cluster, make a Get Cluster (BYOC, Dedicated) or Get Serverless Cluster (Serverless) request with the Control Plane API. The response contains the data plane API URL.
+
+   From the `dataplane_api.url` value in the response, extract only the subdomain (the part between `https://` and `.cloud.redpanda.com`). Enter this value in the Data Plane API URL field.
+
+1. In the **Choose an operation** dropdown, select **Create topic**.
+
+1. If you need a valid access token, click **Get token**. You may be prompted to log in to the Redpanda Cloud UI. After you log in, the browser automatically redirects you back to the Create topic endpoint in the API Explorer.
 
-1. In the page header, click **API Explorer**.
-1. If you don't already have the data plane API URL for your target cluster, make a Get Cluster (BYOC, Dedicated) or Get Serverless Cluster (Serverless) request with the Control Plane API. The response contains the data plane API URL. Copy the value of `dataplane_api.url` from the response body, and enter it in the URL field.
-1. On the **Choose an operation** dropdown, select **Create topic**.
-1. Click **Get token**. You may be prompted to log in to the Redpanda Cloud UI. After you log in, the browser automatically redirects you back to the Create resource group endpoint in the API Explorer.
 1. Enter a name for your topic and click **Send request**.
+
 1. Confirm that your topic is successfully created by making a List topics request.
 
 ## Next steps
 
-The quickest ways to produce to and consume from the topic are to use [`rpk`](https://docs.redpanda.com/redpanda-cloud/manage/rpk/rpk-install/) or the [Redpanda HTTP Proxy](https://docs.redpanda.com/api/doc/http-proxy) (BYOC and Dedicated clusters only).
+The quickest ways to produce to and consume from the topic are to use [`rpk`](https://docs.redpanda.com/redpanda-cloud/manage/rpk/rpk-install/) or the [Redpanda HTTP Proxy](/api/doc/http-proxy) (BYOC and Dedicated clusters only).
 
 For example, to use `rpk` to produce to a topic named `test-topic`, run:
 

shared/overlays/update-securityschemes-for-authentication.yaml

@@ -7,4 +7,11 @@ info:
 actions:
   - target: $.components.securitySchemes.auth0
     update:
-      description: The Cloud APIs use OAuth 2.0 for authentication. You must create a service account in the Cloud UI in order to request an access token. You can use the same access token to authenticate requests to both the Control Plane and Data Plane APIs.
+      description: |-
+        The Cloud APIs use OAuth 2.0 for authentication.
+        
+        **For API Explorer (browser):** Authentication is handled automatically. Click "Get token" on any endpoint to authenticate with your Redpanda Cloud account and include a bearer token for requests made in API Explorer.
+        
+        **For programmatic access (CLI, SDKs, scripts):** You must create a service account in the Redpanda Cloud UI and use the OAuth 2.0 client credentials flow to obtain an access token. See the Authentication topic for detailed instructions.
+        
+        The same access token can be used for both Control Plane and Data Plane API requests for as long as the token is valid (one hour).

shared/x-topics/about-authentication.md

@@ -1,16 +1,30 @@
-The Cloud API uses the Client Credentials Flow as defined in [Auth 2.0 RFC 6749, section 4.4](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4O). In Redpanda Cloud, you must first create a **service account** through which you can authenticate requests to the Cloud API. The service account is associated with your Redpanda Cloud organization. The service account acts as an OAuth 2.0 client that provides its credentials (client ID and client secret) to the API authentication server. The authentication server grants an access token in return. You can then include the access token in each request to the API.
-
-The access token granted to you is associated with a specific Redpanda Cloud organization. If you want to use the API for a different organization, you must acquire a new token through a service account with that organization.
+The Redpanda Cloud API uses OAuth 2.0 for authentication. The authentication method depends on whether you access the API in the browser using the API Explorer or programmatically.
 
 You only need to authenticate once to the Cloud API. That is, after you obtain an access token, you can use the same token in requests to both Control Plane and Data Plane API endpoints, for as long as the token is valid.
 
-## Request an access token
+## Authenticate in API Explorer
+
+When using the API Explorer in your browser, Redpanda Cloud uses the [OAuth 2.0 Implicit Flow](https://datatracker.ietf.org/doc/html/rfc6749#section-4.2):
+
+1. Click **Get token** on any API endpoint.
+2. Log in with your Redpanda Cloud credentials if prompted. After you log in, you are redirected back to the API Explorer.
+3. The token is automatically applied to your API requests.
+
+## Authenticate programmatic requests
+
+For programmatic access (using a CLI, SDKs, and applications), the Cloud API uses the Client Credentials Flow, as defined in [OAuth 2.0 RFC 6749, section 4.4](https://datatracker.ietf.org/doc/html/rfc6749#section-4.4).
+ 
+You must first create a **service account** through which you can authenticate requests to the Cloud API. The service account is associated with your Redpanda Cloud organization. The service account acts as an OAuth 2.0 client that provides its credentials (client ID and client secret) to the API authentication server. The authentication server grants an access token in return. You can then include the access token in each request to the API.
+
+The access token granted to you is associated with a specific Redpanda Cloud organization. If you want to use the API for a different organization, you must acquire a new token through a service account with that organization.
+
+### Request an access token
 
 Users with administrative privileges in a Redpanda Cloud organization can create a service account.
 
 > **Note:** Service accounts have administrative privileges by default. Cloud user roles are not applied for the API.
 
-1. On the [Service account](https://cloud.redpanda.com/organization-iam?tab=service-accounts) tab in the Organization IAM page of the Redpanda Cloud UI, click **Create service account**. Enter a name and description.
+1. In the Redpanda Cloud UI, go to [Organization IAM](https://cloud.redpanda.com/organization-iam), and select the **Service account** tab. Click **Create service account** and enter a name and description.
 
 2. Copy and store the Client ID and Client Secret. These credentials cannot be viewed again after you close the dialog. 
 
@@ -26,7 +40,7 @@ Users with administrative privileges in a Redpanda Cloud organization can create
 
     The request response provides an access token that remains valid for one hour.
 
-## Authenticate API requests
+### Use the access token in API requests
 
 You must pass the access token in the authorization header of each API request: 
 

shared/x-topics/cloud-api-deprecation-policy.md

@@ -48,7 +48,6 @@ Redpanda communicates deprecation notices through the following:
 * Updates to Cloud API documentation
 * [What's New in Redpanda Cloud](https://docs.redpanda.com/redpanda-cloud/get-started/whats-new-cloud)
 * Announcements in the #cloudapi [Community Slack](https://redpandacommunity.slack.com/) channel and in individual customer channels
-* Emails to "Sign up for updates" recipients (sign up on the [Control Plane API](/api/ROOT/cloud-controlplane-api) and [Data Plane APIs](/api/ROOT/cloud-dataplane-api) reference documentation)
 
 ## Versioning policy