Introduce partial resources for cross RP resource references (#2229)

* add configuration for virtual resources

* add a new layer in inheritance hierarchy VirtualResource. Resource now stands for ConcreteResource and inherits from VirtualResource

* add a new test project about virtual resources

* reverse the inheritance between Resource and VirtualResource

* now we could add virtual resource to resource list

* implement virtual resources

* update test swagger

* update to virtual resource

* fix broken test cases

* add test cases

* update test cases

* rename virtual resource to partial resource

* change the create resource identifier to internal in partial resource

* fix

* some refinement and documentation

* now you can override the name of a partial resource like a normal resource

* modify the description of partial resource accordingly

* fix test cases

* minor fix after merge

* regenerate the new test project

* regenerate

* refactor and regenerate

* add a comment to explain the override

* refactor

* regenerate test projects

* fix a few issues

Author: ArcturusZhang

docs/mgmt/readme.md

@@ -12,6 +12,7 @@
     - [Change Singleton Resources](#change-singleton-resources)
     - [Change the Name of Operations](#change-the-name-of-operations)
     - [List Exception](#list-exception)
+    - [Partial Resources](#partial-resources)
     - [Scope Resources](#scope-resources)
     - [SDK Polishing Configurations](#sdk-polishing-configurations)
     - [Management Debug Options](#management-debug-options)
@@ -625,6 +626,42 @@ list-exception:
 - /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/availabilitySets/{availabilitySetName}
 ```
 
+### Partial resources
+
+Some operations might be operations of a resource from another RP, or some resources might have a parent resource from another RP. 
+
+For instance, in the RP `compute`, we have a resource `VirtualMachine` with the path of `/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachines/{vmName}`, in the meantime, in the RP `network`, which is completely a different SDK, we have an operation with the path of `/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachines/{vmName}/publicIps` which is clearly listing something on the virtual machine. During the generation of network RP, the generator has no knowledge of the resources in another RP, therefore by default this operation will be generated like the following code as an operation on the `ResourceGroupResource` because this is the best parent the generator could find within this RP.
+```csharp
+public static partial class NetworkExtensions
+{
+    public static Pageable<PublicIPAddress> GetPublicIPs(this ResourceGroupResource resourceGroup, string vmName)
+    {
+        /* the implementation */
+    }
+} 
+```
+
+The following configuration
+```yaml
+partial-resources:
+  /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachines/{vmName}: VirtualMachine
+```
+will tell the generator that this path `/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachines/{vmName}` is also a valid resource, and the generator will put it into the list of its known resources, and generates a "partial resource" from it and the operation in above example will show up here:
+```csharp
+public partial class VirtualMachineNetworkResource
+{
+    public virtual Pageable<PublicIPAddress> GetPublicIPs()
+    {
+        /* the implementation */
+    }
+}
+```
+
+The naming pattern of a partial resource is like `[ResourceName][RPName]Resource`, where `ResourceName` is the value you assigned in the configuration, `RPName` is the name of this RP (for instance, `Network`). The name of a partial resource is also configurable using the `request-path-to-resource-name` configuration, please see [change resource name](#change-resource-name) section for more details.
+
+The difference between a normal resource and a partial resource is that the partial resource is not a concrete resource, therefore it does not have a corresponding collection, and you cannot get the list of it from its direct parent (ResourceGroupResource in the above example).
+But like normal resources, partial resources will have an extension method of `Get[PartialResourceName]` on the `ArmClient` to let you get it from its full Id.
+
 ### Scope resources
 
 Some resources are scope resources, which means that these resources can be created under different scopes, like subscriptions, resource groups, management groups, etc. In the swagger, we currently do not have an extension which assigns which type of resources can be the scope of this resource, therefore we add a configuration in our generator `request-path-to-scope-resource-types` for this new information.

src/AutoRest.CSharp/Mgmt/AutoRest/MgmtConfiguration.cs

@@ -120,7 +120,8 @@ public MgmtConfiguration(
             JsonElement? singletonRequiresKeyword = default,
             JsonElement? operationIdMappings = default,
             JsonElement? updateRequiredCopy = default,
-            JsonElement? patchInitializerCustomization = default)
+            JsonElement? patchInitializerCustomization = default,
+            JsonElement? partialResources = default)
         {
             RequestPathToParent = DeserializeDictionary<string, string>(requestPathToParent);
             RequestPathToResourceName = DeserializeDictionary<string, string>(requestPathToResourceName);
@@ -135,6 +136,7 @@ public MgmtConfiguration(
             RenameMapping = DeserializeDictionary<string, string>(renameMapping);
             ParameterRenameMapping = DeserializeDictionary<string, IReadOnlyDictionary<string, string>>(parameterRenameMapping);
             IrregularPluralWords = DeserializeDictionary<string, string>(irregularPluralWords);
+            PartialResources = DeserializeDictionary<string, string>(partialResources);
             try
             {
                 OperationPositions = DeserializeDictionary<string, string[]>(operationPositions);
@@ -209,6 +211,7 @@ private static Dictionary<TKey, TValue> DeserializeDictionary<TKey, TValue>(Json
         public IReadOnlyDictionary<string, string[]> RequestPathToScopeResourceTypes { get; }
         public IReadOnlyDictionary<string, string[]> OperationPositions { get; }
         public IReadOnlyDictionary<string, string[]> MergeOperations { get; }
+        public IReadOnlyDictionary<string, string> PartialResources { get; }
         public IReadOnlyList<string> RawParameterizedScopes { get; }
         private ImmutableHashSet<RequestPath>? _parameterizedScopes;
         internal ImmutableHashSet<RequestPath> ParameterizedScopes
@@ -268,7 +271,8 @@ internal static MgmtConfiguration GetConfiguration(IPluginCommunication autoRest
                 singletonRequiresKeyword: autoRest.GetValue<JsonElement?>("singleton-resource-requires-keyword").GetAwaiter().GetResult(),
                 operationIdMappings: autoRest.GetValue<JsonElement?>("operation-id-mappings").GetAwaiter().GetResult(),
                 updateRequiredCopy: autoRest.GetValue<JsonElement?>("update-required-copy").GetAwaiter().GetResult(),
-                patchInitializerCustomization: autoRest.GetValue<JsonElement?>("patch-initializer-customization").GetAwaiter().GetResult());
+                patchInitializerCustomization: autoRest.GetValue<JsonElement?>("patch-initializer-customization").GetAwaiter().GetResult(),
+                partialResources: autoRest.GetValue<JsonElement?>("partial-resources").GetAwaiter().GetResult());
         }
 
         internal void SaveConfiguration(Utf8JsonWriter writer)
@@ -297,6 +301,7 @@ internal void SaveConfiguration(Utf8JsonWriter writer)
             WriteNonEmptySettings(writer, nameof(ParameterRenameMapping), ParameterRenameMapping);
             WriteNonEmptySettings(writer, nameof(IrregularPluralWords), IrregularPluralWords);
             WriteNonEmptySettings(writer, nameof(OverrideOperationName), OverrideOperationName);
+            WriteNonEmptySettings(writer, nameof(PartialResources), PartialResources);
             MgmtDebug.Write(writer, nameof(MgmtDebug));
             if (IsArmCore)
                 writer.WriteBoolean("ArmCore", IsArmCore);
@@ -340,6 +345,7 @@ internal static MgmtConfiguration LoadConfiguration(JsonElement root)
             root.TryGetProperty(nameof(OverrideOperationName), out var operationIdToName);
             root.TryGetProperty(nameof(MergeOperations), out var mergeOperations);
             root.TryGetProperty(nameof(PromptedEnumValues), out var promptedEnumValuesElement);
+            root.TryGetProperty(nameof(PartialResources), out var virtualResources);
             root.TryGetProperty(nameof(RawParameterizedScopes), out var parameterizedScopesElement);
 
             var operationGroupToOmit = Configuration.DeserializeArray(operationGroupsToOmitElement);
@@ -399,7 +405,8 @@ internal static MgmtConfiguration LoadConfiguration(JsonElement root)
                 singletonRequiresKeyword: singletonRequiresKeyword,
                 operationIdMappings: operationIdMappings,
                 updateRequiredCopy: updateRequiredCopy,
-                patchInitializerCustomization: patchInitializerCustomization);
+                patchInitializerCustomization: patchInitializerCustomization,
+                partialResources: virtualResources);
         }
 
         private static void WriteNonEmptySettings(

src/AutoRest.CSharp/Mgmt/AutoRest/MgmtContext.cs

@@ -2,6 +2,7 @@
 // Licensed under the MIT License.
 
 using System;
+using System.Linq;
 using AutoRest.CSharp.Input;
 using AutoRest.CSharp.Output.Models.Types;
 
@@ -16,6 +17,10 @@ internal static class MgmtContext
 
         public static CodeModel CodeModel => Context.CodeModel;
 
+        public static string DefaultNamespace => Context.DefaultNamespace;
+
+        public static string RPName => DefaultNamespace.Split('.').Last();
+
         public static bool IsInitialized => _context is not null;
 
         public static void Initialize(BuildContext<MgmtOutputLibrary> context)

src/AutoRest.CSharp/Mgmt/AutoRest/MgmtOutputLibrary.cs

@@ -3,7 +3,6 @@
 
 using System;
 using System.Collections.Generic;
-using System.Diagnostics;
 using System.Diagnostics.CodeAnalysis;
 using System.Linq;
 using AutoRest.CSharp.Common.Input;
@@ -23,6 +22,7 @@
 using Azure.ResourceManager.ManagementGroups;
 using Azure.ResourceManager.Resources;
 using Humanizer.Inflections;
+using static AutoRest.CSharp.Mgmt.Decorator.Transformer.PartialResourceResolver;
 
 namespace AutoRest.CSharp.Mgmt.AutoRest
 {
@@ -39,7 +39,7 @@ internal class MgmtOutputLibrary : OutputLibrary
         /// This is a map from raw request path to their corresponding <see cref="OperationSet"/>,
         /// which is a collection of the operations with the same raw request path
         /// </summary>
-        private Dictionary<string, OperationSet> RawRequestPathToOperationSets { get; }
+        internal Dictionary<string, OperationSet> RawRequestPathToOperationSets { get; }
 
         /// <summary>
         /// This is a map from operation to its corresponding operation group
@@ -200,7 +200,7 @@ private IEnumerable<Schema> UpdateBodyParameters()
                         // get the request path and operation set
                         RequestPath requestPath = RequestPath.FromOperation(operation, operationGroup);
                         var operationSet = RawRequestPathToOperationSets[requestPath];
-                        if (operationSet.TryGetResourceDataSchema(out var resourceDataModel))
+                        if (operationSet.TryGetResourceDataSchema(out var resourceDataSchema))
                         {
                             // if this is a resource, we need to make sure its body parameter is required when the verb is put or patch
                             BodyParameterNormalizer.MakeRequired(bodyParam, httpRequest.Method);
@@ -212,14 +212,14 @@ private IEnumerable<Schema> UpdateBodyParameters()
                             BodyParameterNormalizer.UpdateParameterNameOnly(bodyParam, ResourceDataSchemaNameToOperationSets);
                             continue;
                         }
-                        if (resourceDataModel is not null)
+                        if (resourceDataSchema is not null)
                         {
                             //TODO handle expandable request paths. We assume that this is fine since if all of the expanded
                             //types use the same model they should have a common name, but since this case doesn't exist yet
                             //we don't know for sure
                             if (requestPath.IsExpandable)
                                 throw new InvalidOperationException($"Found expandable path in UpdatePatchParameterNames for {operationGroup.Key}.{operation.CSharpName()} : {requestPath}");
-                            var name = GetResourceName(resourceDataModel.Name, operationSet, requestPath);
+                            var name = GetResourceName(resourceDataSchema.Name, operationSet, requestPath);
                             updatedModels.Add(bodyParam.Schema);
                             BodyParameterNormalizer.Update(httpRequest.Method, operation.CSharpName(), bodyParam, name);
                         }
@@ -564,30 +564,45 @@ private Dictionary<RequestPath, ResourceObjectAssociation> EnsureRequestPathToRe
 
             foreach ((var resourceDataSchemaName, var operationSets) in ResourceDataSchemaNameToOperationSets)
             {
-                var resourceOperationsList = FindResourceToChildOperationsMap(operationSets);
-                foreach ((var operationSet, var operations) in resourceOperationsList)
+                foreach (var operationSet in operationSets)
                 {
-                    var isSingleton = operationSet.IsSingletonResource();
+                    var operations = GetChildOperations(operationSet.RequestPath);
                     // get the corresponding resource data
                     var originalResourcePath = operationSet.GetRequestPath();
                     var resourceData = GetResourceData(originalResourcePath);
-                    // we calculate the resource type of the resource
-                    var resourcePaths = originalResourcePath.Expand();
-                    foreach (var resourcePath in resourcePaths)
-                    {
-                        var resourceType = resourcePath.GetResourceType();
-                        var resource = new Resource(operationSet, operations, GetResourceName(resourceDataSchemaName, operationSet, resourcePath), resourceType, resourceData);
-                        var collection = isSingleton ? null : new ResourceCollection(operationSet, operations, resource);
-                        resource.ResourceCollection = collection;
-
-                        requestPathToResources.Add(resourcePath, new ResourceObjectAssociation(resourceType, resourceData, resource, collection));
-                    }
+                    if (resourceData is EmptyResourceData emptyResourceData)
+                        BuildPartialResource(requestPathToResources, resourceDataSchemaName, operationSet, operations, originalResourcePath, emptyResourceData);
+                    else
+                        BuildResource(requestPathToResources, resourceDataSchemaName, operationSet, operations, originalResourcePath, resourceData);
                 }
             }
 
             return requestPathToResources;
         }
 
+        private void BuildResource(Dictionary<RequestPath, ResourceObjectAssociation> result, string resourceDataSchemaName, OperationSet operationSet, IEnumerable<Operation> operations, RequestPath originalResourcePath, ResourceData resourceData)
+        {
+            var isSingleton = operationSet.IsSingletonResource();
+            // we calculate the resource type of the resource
+            var resourcePaths = originalResourcePath.Expand();
+            foreach (var resourcePath in resourcePaths)
+            {
+                var resourceType = resourcePath.GetResourceType();
+                var resource = new Resource(operationSet, operations, GetResourceName(resourceDataSchemaName, operationSet, resourcePath), resourceType, resourceData);
+                var collection = isSingleton ? null : new ResourceCollection(operationSet, operations, resource);
+                resource.ResourceCollection = collection;
+
+                result.Add(resourcePath, new ResourceObjectAssociation(resourceType, resourceData, resource, collection));
+            }
+        }
+
+        private void BuildPartialResource(Dictionary<RequestPath, ResourceObjectAssociation> result, string resourceDataSchemaName, OperationSet operationSet, IEnumerable<Operation> operations, RequestPath originalResourcePath, EmptyResourceData emptyResourceData)
+        {
+            var resourceType = originalResourcePath.GetResourceType();
+            var resource = new PartialResource(operationSet, operations, GetResourceName(resourceDataSchemaName, operationSet, originalResourcePath, isPartial: true), resourceDataSchemaName, resourceType, emptyResourceData);
+            result.Add(originalResourcePath, new ResourceObjectAssociation(originalResourcePath.GetResourceType(), emptyResourceData, resource, null));
+        }
+
         private string? GetDefaultNameFromConfiguration(OperationSet operationSet, ResourceTypeSegment resourceType)
         {
             if (Configuration.MgmtConfiguration.RequestPathToResourceName.TryGetValue(operationSet.RequestPath, out var name))
@@ -598,14 +613,23 @@ private Dictionary<RequestPath, ResourceObjectAssociation> EnsureRequestPathToRe
             return null;
         }
 
-        private string GetResourceName(string candidateName, OperationSet operationSet, RequestPath requestPath)
+        private string GetResourceName(string candidateName, OperationSet operationSet, RequestPath requestPath, bool isPartial = false)
         {
             // read configuration to see if we could get a configuration for this resource
             var resourceType = requestPath.GetResourceType();
             var defaultNameFromConfig = GetDefaultNameFromConfiguration(operationSet, resourceType);
             if (defaultNameFromConfig != null)
                 return defaultNameFromConfig;
 
+            var resourceName = CalculateResourceName(candidateName, operationSet, requestPath, resourceType);
+
+            return isPartial ?
+                $"{resourceName}{MgmtContext.RPName}" :
+                resourceName;
+        }
+
+        private string CalculateResourceName(string candidateName, OperationSet operationSet, RequestPath requestPath, ResourceTypeSegment resourceType)
+        {
             // find all the expanded request paths of resources that are assiociated with the same resource data model
             var resourcesWithSameResourceData = ResourceDataSchemaNameToOperationSets[candidateName]
                 .SelectMany(opSet => opSet.GetRequestPath().Expand()).ToList();
@@ -716,19 +740,12 @@ public int GetHashCode([DisallowNull] IEnumerable<RequestPath> obj)
             }
         }
 
-        private Dictionary<OperationSet, IEnumerable<Operation>> FindResourceToChildOperationsMap(IEnumerable<OperationSet> resourceOperationSets)
-        {
-            return resourceOperationSets.ToDictionary(
-                operationSet => operationSet,
-                operationSet => GetChildOperations(operationSet.RequestPath));
-        }
-
         public IEnumerable<Operation> GetChildOperations(string requestPath)
         {
             if (requestPath == RequestPath.Any)
                 return Enumerable.Empty<Operation>();
 
-            if (EnsureResourceChildOperations().TryGetValue(requestPath, out var operations))
+            if (ChildOperations.TryGetValue(requestPath, out var operations))
                 return operations;
 
             return Enumerable.Empty<Operation>();
@@ -828,6 +845,7 @@ public IEnumerable<Resource> FindResources(ResourceData resourceData)
 
         private TypeProvider BuildResourceData(Schema schema) => schema switch
         {
+            EmptyObjectSchema emptyObjectSchema => new EmptyResourceData(emptyObjectSchema),
             ObjectSchema objectSchema => new ResourceData(objectSchema),
             _ => throw new NotImplementedException()
         };
@@ -857,6 +875,7 @@ private Dictionary<string, HashSet<OperationSet>> DecorateOperationSets()
                     result.Add(operationSet);
                 }
             }
+
             return resourceDataSchemaNameToOperationSets;
         }
 
@@ -885,6 +904,13 @@ private Dictionary<string, OperationSet> CategorizeOperationGroups()
                     }
                 }
             }
+
+            // add operation set for the partial resources here
+            foreach (var path in Configuration.MgmtConfiguration.PartialResources.Keys)
+            {
+                rawRequestPathToOperationSets.Add(path, new OperationSet(path));
+            }
+
             return rawRequestPathToOperationSets;
         }