Add unmanaged comms in supervisor API

[lionel-]

Progress towards #9094.
Progress towards #7448.

This PR adds support for "unmanaged" comms, i.e. comms that are not mapped to Positron clients. Instead they are fully managed by the kernel supervisor.

• No longer need to call registerClientInstance() implemented in Fix/suppress log noise around comms #2892 to tell the core to ignore the unknown client.

• The comm is fully private to the extension that creates it. It can send and receive messages of any type.

• The API makes it easy to receive and handle messages in order via a new Channel type that works like Rust channels. It also makes it easy to send requests and notifications.

• Like Positron clients, the message structure is based on JSON-RPC messages embedded in the jupyter comm messages. But we're moving towards a stricter version of JSON-RPC with includion of an id field, in order to support notifications from frontend to backend as the field disambiguates with requests. See Support backend-side events/notifications in OpenRPC contracts #7448.

Here is how it works:

The extension is the only side that can create an unmanaged comm. To do so it calls:

	createComm(
		target_name: string,
		params?: Record<string, unknown>,
	): Promise<Comm>;

The promise resolves to an instance of Comm:

export interface Comm {
    /** The comm ID. */
    id: string;

    /**
     * Async-iterable for messages sent from backend.
     *
     * - This receiver channel _must_ be awaited and handled to exhaustion.
     * - When exhausted, you _must_ dispose of the comm.
     *
     * Yields `CommBackendMessage` messages which are a tagged union of
     * notifications and requests. If a request, the `handle` method _must_ be
     * called (see `CommBackendMessage` documentation).
     */
    receiver: ReceiverChannel<CommBackendMessage>;

    /**
     * Send a notification to the backend comm.
     * Throws `CommClosedError` if comm was closed.
     */
    notify: (method: string, params?: Record<string, unknown>) => void;

    /**
     * Make a request to the backend comm.
     *
     * Resolves when backend responds with the result.
     * Throws:
     * - `CommClosedError` if comm was closed
     * - `CommRpcError` for RPC errors.
     */
    request: (method: string, params?: Record<string, unknown>) => Promise<any>;

    /** Clear resources and sends `comm_close` to backend comm (unless the channel
     * was closed by the backend already). */
    dispose: () => Promise<void>;
}

Channels:

• Implemented by the supervisor in Channel.ts, they provide a similar abstraction to Rust channels. This makes it easy to handle messages in order. When the channel closes, this is like a close event instructing the extension code to clean up the comm.

• The Comm wraps a ReceiverChannel, the receiving side. The sending side is owned by the supervisor.

• Messages are a tagged union of Notification | Request. The Request type includes a handle() method that must be called with a closure. Throwing from the closure rejects the request. The following is a minimal implementation of a receiver loop:

  for await (const message of comm.receiver) {
    if (message.kind === "request") {
      message.handle(() => {
        throw new Error(`Unknown request '${message.method}' for DAP comm`);
      });
    }
  }

  There is a bit of boilerplate with this approach but it offers a lot of flexibility. The extension can break the loop at any time or call third-party handlers for messages (which we do for DAP, see below).

Lifecycle:

• If dispose() is called on a frontend Comm, the receiver channel is closed and drained and a comm_close message is sent to the backend.

• If the backend closes the comm, the channel is closed. That serves as an event notifying the extension that the comm is closed and should be disposed.

Sending messages:

• For ergonomics, messages are not sent from the extension with a channel but by calling notify() and request() methods on the comm. Both can throw if channel is closed.

• request() can simply be awaited until the response comes in. In addition to CommClosedError, the caller should handle CommRpcError.

DAP and Server comms:

• The Server comm is a light abstraction over a comm. Any comm can be a server comm if it follows the handshaking protocol. Call createServerComm() instead of createComm(). When this resolves, you'll have a fully started server comm whose backend side is ready to receive connections.

• The DAP comm is a light wrapper around a Comm. It's created with createDapComm() (which internally calls createServerComm()) and returns a DapComm. The point of this wrapper if to gain access to DapComm::handleMessage(). This method can be called from the extension receiver loop to handle all or a subset of our "standard" DAP messages. For instance it can handle start_debug messages to start a debugging session from the backend. The positron-r extension fully delegates to this handle method, but extensions are free to handle more message types, or to override handling of some of the standard DAP messages.

Frontend notifications:

• Ark needed a way determine whether a message (from either a client or unmanaged comm) is a request or a notification. Up to now it assumed all messages are requests. The simplest way to achieve this is to follow the JSON-RPC standard and use presence of an id field as discriminant.

  This required changes in the core. The base comm now generates an id field for requests, and performRpcWithBuffers() make sure the field is present if the caller did not include one.

  Progress toward Support backend-side events/notifications in OpenRPC contracts #7448.

Miscellaneous changes:

• Dap is no longer a Positron Client. In the future, LSP server comms should also become unmanaged comms instead of Positron clients, but I thought we'd start with DAP to lower the stakes in case something goes wrong with the refactor.

• Added a justfile in the supervisor source folder. Call just to regenerate all positron-supervisor.d.ts files in the monorepo, reformatted if needed.

• Renamed the Comm supervisor type to Client. So that unmanaged comms are now Comm.

• Positron-r now creates the "Ark Comm". This can be used in the future for private communication with Ark. For the time being, this is used in unit tests for comm messaging, using the new infrastructure for extension tests developed in Fix OpenEditor event on Windows (and test it) #8816.

Release Notes

New Features

• N/A

Bug Fixes

• N/A

QA Notes

Full suite running at https://github.com/posit-dev/positron/actions/runs/17293101714

[github-actions]

E2E Tests 🚀
This PR will run tests tagged with: @:critical

^{readme  valid tags}

[lionel-]

Comments from pair review with @DavisVaughan

[lionel-]

From review with Davis:

We might be able to make activate() async, then await the supervisor API, then set a global SUPERVISOR_API. Then supervisorApi() no longer needs to be async, which would make it much more ergonomic:

[lionel-]

Would allow us to export error classes and use instanceof to match them, without contagious asyncness.

[jmcphers]

Nit: Better name for this var?

[jmcphers]

Modeling the channel receiver as an async iterable is a cool idea!

[jmcphers]

Hah, thanks for catching this!

[jmcphers]

Nit: clear array of disposables after they are disposed, to clear references and make this idempotent