[API Proposal]: System.ClientModel long-running operation base type `OperationResult`

[annelo-msft]

Background and motivation

The System.ClientModel library provides building blocks for .NET clients that call cloud services. This addition supports client APIs for long-running operations, i.e. service operations where the service returns a response before the requested operation has completed. For such operations, SCM-based clients return a subclient derived from SCM's abstract OperationResult class, which allows users to monitor the operation's status and obtain any resulting value.

The new OperationResult type is similar to the Operation and Operation<T> types in Azure.Core, but has fewer public APIs in ordre to support greater variation of implementation patterns in the third-party cloud service space. It serves as a base type for public operation subclients such as the CreateVectorStoreOperation type in the .NET OpenAI client library. It provides APIs to wait for the operation to complete processing on the service, and a rehydration token that can be to "rehydrate" an operation in progress, e.g. to obtain the value computed by a long-running operation from a different process than the one that started the operation. Client subtypes add public properties such as Value and Status as applicable to the operation implementation.

API Proposal

namespace System.ClientModel.Primitives
{
    public abstract partial class OperationResult : System.ClientModel.ClientResult
    {
        protected OperationResult(System.ClientModel.Primitives.PipelineResponse response) { }
        public abstract bool IsCompleted { get; protected set; }
        public abstract System.ClientModel.ContinuationToken? RehydrationToken { get; protected set; }
        public abstract void WaitForCompletion(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken));
        public abstract System.Threading.Tasks.Task WaitForCompletionAsync(System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken));
    }
}

API Usage

Example usage of derived CreateVectorStoreOperation type in OpenAI client:

VectorStoreClient client = new VectorStoreClient(new OpenAIClientOptions());
CreateVectorStoreOperation operation = await client.CreateVectorStoreAsync(waitUntilCompleted: false);
await operation.WaitForCompletionAsync();
VectorStore vectorStore = operation.Value;

Alternative Designs

No response

Risks

No response

[stephentoub]

Generally looks reasonably, but I'm wondering about the IsCompleted property. What's the use case for that, and does the value change after the result has been constructed? e.g. is that actually going to make networking requests to determine whether the operation has completed? If it doesn't, then I worry it'll be confusing that IsCompleted isn't up-to-date. And if it does, then I worry that there's expensive work happening behind a property, possibly expensive asynchronous work.

[stephentoub]

Also, for the usage example, what happens if instead of:

VectorStoreClient client = new VectorStoreClient(new OpenAIClientOptions());
CreateVectorStoreOperation operation = await client.CreateVectorStoreAsync(waitUntilCompleted: false);
await operation.WaitForCompletionAsync();
VectorStore vectorStore = operation.Value;

I end up writing:

VectorStoreClient client = new VectorStoreClient(new OpenAIClientOptions());
CreateVectorStoreOperation operation = await client.CreateVectorStoreAsync(waitUntilCompleted: false);
VectorStore vectorStore = operation.Value;

which is what I'd naturally be inclined to do. Does Value throw an exception if it's not yet completed? Does it block until the operation completes? I assume waitUntilCompleted defaults to true in general, so by default developers don't need to do the wait explicitly and the "right things" just happen?

[terrajobst]

How does the operation know that it's completed? Is it push based (i.e. does the server tell the client "I'm done") or is it polling based?

[annelo-msft]

I'm wondering about the IsCompleted property. What's the use case for that, and does the value change after the result has been constructed?

Yes, the value of IsCompleted will typically change after the result has been constructed. The property itself doesn't make any service calls (we avoid this from properties because of the difficulty making async calls there), but if a caller passes waitUntilCompleted: false, the intended use of the returned operation subclient is to call the WaitForCompletion method prior to accessing a Value property, if the intention is to access the final output value from the operation. To answer @terrajobst's question, many LROs are polling-based, so WaitForCompletion will poll a service endpoint to obtain the operation status until the operation has completed, and then it will set IsCompleted to true before returning.

IsCompleted is available to check whether the operation has completed, e.g. in a rehydration scenario. Another use case for it is that, in cases where the public subtype adds an UpdateStatus method (or similar), a user can implement their own polling loop (e.g. to use a custom polling interval) by doing:

while (!operation.IsCompleted)
{
    // delay custom interval

    operation.UpdateStatus();    
}

Does Value throw an exception if it's not yet completed? Does it block until the operation completes?

It does not. Value should be a nullable property when it appears on an OperationResult subtype, so if the service operation doesn't provide a response containing a value until the operation has completed, a correct implementation would return null from the property until IsCompleted is true. Some services return responses containing values that represent intermediate progress -- OpenAI does this, for example -- and for these, Value is populated with the intermediate result because it may contain information of value to the end user.

I assume waitUntilCompleted defaults to true in general, so by default developers don't need to do the wait explicitly and the "right things" just happen?

It actually doesn't. We made the design decision in Azure clients to require a user to pass a value for this parameter because it was hard to pick a default that would work across all LRO cases, and we are planning to follow the same design for SCM-based clients. For example, some LROs can take up to 24 hours to complete, and we didn't want user code to block for that long without explicit opt-in to that. Other LROs can return in a few seconds, so we also didn't want to force callers to write extra lines of code to WaitForCompletion in those cases. At the end of the day, we felt it was better to require the end-user to pass a parameter opting-in to their preference, since they should have a better sense of the characteristics of a given operation, and would need to write different code subsequent to the start-operation call depending on the value they passed. (As a side-note, in Azure.Core, we have a separate WaitUntil enum we use for this parameter, but we felt we didn't need an extra type for it in SCM.)

[stephentoub]

Thanks. IsCompleted is still troublesome to me. I'd expect to be able to poll it in a loop like:

while (!IsCompleted)
{
   DoOtherThings();
}

but it sounds like that'll likely hang because I'm not calling am explicit update that doesn't exist on the base where IsCompleted is. Does IsCompleted need to exist on the base? Or should it be a method that also has an async counterpart that can actually poll the service if not completed?

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[karelz]

@stephentoub does it belong to Runtime repo? Or rather into Azure SDK? https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/System.ClientModel/README.md

[bartonjs]

@karelz It's an advisory/discussion item for API Review because it's in the System namespace, and we don't have good process for doing our process on issues in their repository.

We should probably call it area-Meta, or make up an area label for it.