User Story 37654: Create Abstractions package

- Added empty Extensions package with some sample class and docs to demonstrate packaging.
- Created CI stage to build, test, pack, and publish the Extensions NuGet package.
- Updated downstream CI stages/jobs to use the Extensions package.
- Updated build.proj Clean target to not delete packages/ dir.
- Updated BUILDGUIDE with instructions for the Extensions package.
- Cleaned up stale BUIDGUIDE sections.
- Added temporary GitHub Discussion content so the team can review before posting it as a real Discussion.
- Disable .pdb file inclusion in the application package.
- Renamed Extensions package to Abstractions.
- Updated README related to extensions design.

Author: paulmedynski

.editorconfig

@@ -163,7 +163,7 @@ indent_size = 2
 indent_size = 2
 
 # Xml files
-[*.{xml,stylecop,resx,ruleset}]
+[*.{xml,stylecop,resx,ruleset,slnx}]
 indent_size = 2
 
 # Xml config files

BUILDGUIDE.md

@@ -16,40 +16,47 @@ Once the environment is setup properly, execute the desired set of commands belo
 
 ### Targets
 
+The following build targets are defined in `build.proj`:
+
 |Target|Description|
 |-|-|
 |`BuildAllConfigurations`|Default target. Builds the .NET Framework and .NET drivers for all target frameworks and operating systems.|
+|`BuildAbstractionsPackage`|Restore, build, and pack the Abstractions package, publishing the resulting NuGet into `packages/`.|
 |`BuildNetCore`|Builds the .NET driver for all target frameworks.|
 |`BuildNetCoreAllOS`|Builds the .NET driver for all target frameworks and operating systems.|
 |`BuildNetFx`|Builds the .NET Framework driver for all target frameworks.|
 |`BuildTestsNetCore`|Builds tests for the .NET driver.|
 |`BuildTestsNetFx`|Builds tests for the .NET Framework driver.|
-|`Clean`|Cleans generated files.|
-|`Restore`|Restores Nuget packages.|
+|`Clean`|Cleans generated files, except for NuGet packages published to `packages/`.|
+|`CleanAll`|Cleans all generated files.|
+|`Restore`|Restores NuGet packages.|
 |`RunTests`|Runs the unit, functional, and manual tests for the .NET Framework and .NET drivers|
 |`RunUnitTests`|Runs just the unit tests for the .NET Framework and .NET drivers|
 |`RunFunctionalTests`|Runs just the functional tests for the .NET Framework and .NET drivers|
 |`RunManualTests`|Runs just the manual tests for the .NET Framework and .NET drivers|
 |`BuildAkv`|Builds the Azure Key Vault Provider package for all supported platforms.|
 
-
 ### Parameters
+
+The following parameters may be defined as MSBuild properties to configure the
+build:
+
 |Name|Supported Values|Default|Description|
 |-|-|-|-|
 |`Configuration`|`Debug`, `Release`|`Debug`|Sets the release configuration.|
-|`BuildNetFx`|`true`, `false`|`true` (Windows), `false` (other)|If false, skips building the .NET Framework driver on Windows.|
 |`OSGroup`|`Unix`, `Windows_NT`, `AnyOS`|typically defaults to the client system's OS, unless using `BuildAllConfigurations` or an `AnyOS` specific target|The operating system to target.|
 |`Platform`|`AnyCPU`, `x86`, `x64`, `ARM`, `ARM64`|`AnyCPU`|May only be set when using package reference type or running tests.|
 |`TestSet`|`1`, `2`, `3`, `AE`|all|Build or run a subset of the manual tests. Omit (default) to target all tests.|
 |`DotnetPath`|Absolute file path to an installed `dotnet` version.|The system default specified by the path variable|Set to run tests using a specific dotnet version (e.g. C:\net6-win-x86\)|
 |`TF`|`net8.0`, `net462`, `net47`, `net471`, `net472`, `net48`, `net481`|`net8.0` in netcore, `net462` in netfx|Sets the target framework when building or running tests. Not applicable when building the drivers.|
 |`ResultsDirectory`|An absolute file path|./TestResults relative to current directory|Specifies where to write test results.|
 
-
 ## Example Workflows using MSBuild (Recommended)
+
 Using the default configuration and running all tests:
 
 ```bash
+msbuild -t:BuildAbstractionsPackage
 msbuild
 msbuild -t:BuildTestsNetFx -p:TF=net462
 msbuild -t:BuildTestsNetCore
@@ -59,28 +66,31 @@ msbuild -t:RunTests
 Using the Release configuration:
 
 ```bash
-msbuild -p:configuration=Release
-msbuild -t:BuildTestsNetFx -p:TF=net462 -p:configuration=Release
-msbuild -t:BuildTestsNetCore -p:configuration=Release
-msbuild -t:RunTests -p:configuration=Release
+msbuild -t:BuildAbstractionsPackage -p:Configuration=Release
+msbuild -p:Configuration=Release
+msbuild -t:BuildTestsNetFx -p:TF=net462 -p:Configuration=Release
+msbuild -t:BuildTestsNetCore -p:Configuration=Release
+msbuild -t:RunTests -p:Configuration=Release
 ```
 
 Running only the unit tests:
 
 ```bash
+msbuild -t:BuildAbstractionsPackage
 msbuild
 msbuild -t:BuildTestsNetFx -p:TF=net462
 msbuild -t:BuildTestsNetCore
 msbuild -t:RunUnitTests
 ```
 
-Using a specific dotnet version/architecture:
+Using a specific .NET runtime to run tests:
 
 ```bash
-msbuild -p:configuration=Release
-msbuild -t:BuildTestsNetFx -p:TF=net462 -p:configuration=Release
-msbuild -t:BuildTestsNetCore -p:configuration=Release
-msbuild -t:RunTests -p:configuration=Release -p:DotnetPath=C:\net8-win-x86\
+msbuild -t:BuildAbstractionsPackage
+msbuild
+msbuild -t:BuildTestsNetFx -p:TF=net462
+msbuild -t:BuildTestsNetCore
+msbuild -t:RunTests -p:DotnetPath=C:\net8-win-x86\
 ```
 
 ### Running Manual Tests
@@ -119,15 +129,13 @@ Manual Tests require the below setup to run:
   |IsManagedInstance | (Optional) When set to `true` **TVP** related tests will use on non-Azure bs files to compare test results. this is needed when testing against Managed Instances or TVP Tests will fail on Test set 3. The default value is `false`. |
   |PowerShellPath | The full path to PowerShell.exe. This is not required if the path is present in the PATH environment variable. | `D:\\escaped\\absolute\\path\\to\\PowerShell.exe` |
 
-
 ## Example workflows using the Dotnet SDK
 
-#### Run Functional Tests
+### Run Functional Tests
 
 - Windows (`netfx x86`):
 
 ```bash
-msbuild 
 dotnet test "src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.Tests.csproj" -p:Platform="x86" -p:Configuration="Release" -p:TestTargetOS="Windowsnetfx" --no-build -v n --filter "category!=nonnetfxtests&category!=failing&category!=nonwindowstests"
 ```
 
@@ -152,7 +160,8 @@ dotnet test "src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.S
 ```bash
 dotnet test "src/Microsoft.Data.SqlClient/tests/FunctionalTests/Microsoft.Data.SqlClient.Tests.csproj" -p:Platform="AnyCPU" -p:Configuration="Release" -p:TestTargetOS="Unixnetcoreapp" --no-build -v n --filter "category!=nonnetcoreapptests&category!=failing&category!=nonlinuxtests&category!=nonuaptests"
 ```
-#### Run Manual Tests
+
+### Run Manual Tests
 
 - Windows (`netfx x86`):
 
@@ -194,35 +203,40 @@ dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlCl
 
 Tests can be built and run with custom "Reference Type" property that enables different styles of testing:
 
-- "Project" => Build and run tests with Microsoft.Data.SqlClient as Project Reference
-- "Package" => Build and run tests with Microsoft.Data.SqlClient as Package Reference with configured "TestMicrosoftDataSqlClientVersion" in "Versions.props" file.
+- "Project" => Build and run tests with Microsoft.Data.SqlClient as a Project Reference
+- "Package" => Build and run tests with Microsoft.Data.SqlClient as a Package Reference with configured "TestMicrosoftDataSqlClientVersion" in "Versions.props" file.
 
 > ************** IMPORTANT NOTE BEFORE PROCEEDING WITH "PACKAGE" REFERENCE TYPE ***************
 > CREATE A NUGET PACKAGE WITH BELOW COMMAND AND ADD TO LOCAL FOLDER + UPDATE NUGET CONFIG FILE TO READ FROM THAT LOCATION
 >
 > ```bash
->  msbuild -p:configuration=Release
+> msbuild -t:BuildAbstractionsPackage -p:Configuration=Release
+> msbuild -p:Configuration=Release
 > ```
 
 A non-AnyCPU platform reference can only be used with package reference type. Otherwise, the specified platform will be replaced with AnyCPU in the build process.
 
 ### Building Tests with Reference Type
 
-For .NET, all 4 reference types are supported:
+For .NET:
 
 ```bash
+# Project is the default reference type.  The below commands are equivalent:
+msbuild -t:BuildTestsNetCore
 msbuild -t:BuildTestsNetCore -p:ReferenceType=Project
-# Default setting uses Project Reference.
 
+# Package reference type:
 msbuild -t:BuildTestsNetCore -p:ReferenceType=Package
 ```
 
-For .NET Framework, below reference types are supported:
+For .NET Framework:
 
 ```bash
+# Project is the default reference type.  The below commands are equivalent:
+msbuild -t:BuildTestsNetFx -p:TF=net462
 msbuild -t:BuildTestsNetFx -p:TF=net462 -p:ReferenceType=Project
-# Default setting uses Project Reference.
 
+# Package reference type:
 msbuild -t:BuildTestsNetFx -p:TF=net462 -p:ReferenceType=Package
 ```
 
@@ -241,26 +255,25 @@ Tests can be built and run with custom Target Frameworks. See the below examples
 ### Building Tests with custom target framework
 
 ```bash
-msbuild -t:BuildTestsNetFx -p:TF=net462
 # Build the tests for custom .NET Framework target
+msbuild -t:BuildTestsNetFx -p:TF=net462
 ```
 
 ```bash
-msbuild -t:BuildTestsNetCore -p:TF=net8.0
 # Build the tests for custom .NET target
+msbuild -t:BuildTestsNetCore -p:TF=net8.0
 ```
 
 ### Running Tests with custom target framework (traditional)
 
 ```bash
+# Run tests with custom .NET Framework target
 dotnet test -p:TargetNetFxVersion=net462 ...
-# Use above property to run Functional Tests with custom .NET Framework target
 
+# Run tests with custom .NET target
 dotnet test -p:TargetNetCoreVersion=net8.0 ...
-# Use above property to run Functional Tests with custom .NET target
 ```
 
-
 ## Using Managed SNI on Windows
 
 Managed SNI can be enabled on Windows by enabling the below AppContext switch:
@@ -285,20 +298,6 @@ When connecting to a server, if a protocol lower than TLS 1.2 is negotiated, a s
 
 `Switch.Microsoft.Data.SqlClient.SuppressInsecureTLSWarning`
 
-### Troubleshooting Docker issues
-
-There may be times where connection cannot be made to SQL Server, we found below ideas helpful:
-
-- Clear Docker images to create clean image from time-to-time, and clear docker cache if needed by running `docker system prune` in Command Prompt.
-
-- If you face `Microsoft.Data.SqlClient.SNI.dll not found` errors when debugging, try updating the below properties in the netcore\Microsoft.Data.SqlClient.csproj file and try again:
-
-  ```xml
-    <OSGroup>Unix</OSGroup>
-    <TargetsWindows>false</TargetsWindows>
-    <TargetsUnix>true</TargetsUnix>
-  ```
-
 ## Collecting Code Coverage
 
 ### Using VSTest

NuGet.config

@@ -2,10 +2,13 @@
 <configuration>
   <packageSources>
     <clear />
-    <add key="sqlclient" value="https://sqlclientdrivers.pkgs.visualstudio.com/public/_packaging/sqlclient/nuget/v3/index.json" />
+    <!-- We use a curated, vetted ADO feed for our dependencies. -->
+    <add key="curated" value="https://sqlclientdrivers.pkgs.visualstudio.com/public/_packaging/sqlclient/nuget/v3/index.json" />
   </packageSources>
   <auditSources>
     <clear />
+
+    <!-- There is no curated, vetted ADO feed for auditing, so we use the official source. -->
     <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
   </auditSources>
 </configuration>

build.proj

@@ -51,6 +51,7 @@
 
   <!-- Populate all managed projects -->
   <ItemGroup>
+    <AbstractionsPackage Include="src/Microsoft.Data.SqlClient.Extensions/Abstractions/Abstractions.slnx" />
     <SqlServerLib Include="**/Microsoft.SqlServer.Server.csproj" />
     <NetFxDriver Include="**/netfx/**/Microsoft.Data.SqlClient*.csproj" Condition="'$(IsEnabledWindows)' == 'true'" />
     <NetCoreDriver Include="**/netcore/**/Microsoft.Data.SqlClient*.csproj" />
@@ -91,6 +92,32 @@
   <Target Name="BuildTestsNetCore" DependsOnTargets="RestoreTestsNetCore;BuildAKVNetCore;BuildUnitTestsNetCore;BuildFunctionalTestsNetCore;BuildManualTestsNetCore"/>
   <Target Name="BuildTestsNetFx" DependsOnTargets="RestoreTestsNetFx;BuildAKVNetFx;BuildUnitTestsNetFx;BuildFunctionalTestsNetFx;BuildManualTestsNetFx" Condition="$(IsEnabledWindows) == 'true'"/>
 
+  <!--
+    The Abstractions package must be built and packed into the packages/
+    directory before the MDS projects can be restored.
+
+    We cannot make the driver targets depend on this one because of the way
+    MSBuild restores projects in "solution mode".  It will attempt to restore
+    all targets in the graph before building any of them, which fails because
+    MDS depends on the Abstractions package being built and packed into packages/
+    first.
+  -->
+  <Target Name="BuildAbstractionsPackage">
+    <PropertyGroup>
+      <!--
+        Omit the AbstractionsPackageVersion property entirely if it is empty.
+        Otherwise, the command-line property will override the default value,
+        even if empty.
+      -->
+      <BuildProperties Condition="'$(AbstractionsPackageVersion)' != ''">AbstractionsPackageVersion=$(AbstractionsPackageVersion)</BuildProperties>
+    </PropertyGroup>
+
+    <MSBuild
+      Projects="@(AbstractionsPackage)"
+      Targets="Restore;Build;Pack"
+      Properties="$(BuildProperties)" />
+  </Target>
+
   <Target Name="RestoreSqlServerLib">
     <MSBuild Projects="@(SqlServerLib)" Targets="restore" />
   </Target>
@@ -346,13 +373,17 @@
     <Exec ConsoleToMsBuild="true" Command="$(TestCommand)" />
   </Target>
 
-  <!-- Clean -->
+  <!-- Clean all intermediate outputs. -->
   <Target Name="Clean">
-    <RemoveDir Directories='$([System.IO.Directory]::GetDirectories(".","artifacts", SearchOption.AllDirectories))' />
-    <RemoveDir Directories='$([System.IO.Directory]::GetDirectories(".","bin", SearchOption.AllDirectories))' />
-    <RemoveDir Directories='$([System.IO.Directory]::GetDirectories(".","obj", SearchOption.AllDirectories))' />
-    <RemoveDir Directories='$([System.IO.Directory]::GetDirectories(".","packages", SearchOption.AllDirectories))' />
-    <RemoveDir Directories='$([System.IO.Directory]::GetDirectories(".",".nuget", SearchOption.AllDirectories))' />
+    <RemoveDir Directories='$([System.IO.Directory]::GetDirectories(".", "artifacts", SearchOption.AllDirectories))' />
+    <RemoveDir Directories='$([System.IO.Directory]::GetDirectories(".", "bin", SearchOption.AllDirectories))' />
+    <RemoveDir Directories='$([System.IO.Directory]::GetDirectories(".", "obj", SearchOption.AllDirectories))' />
+    <RemoveDir Directories='$([System.IO.Directory]::GetDirectories(".", ".nuget", SearchOption.AllDirectories))' />
+  </Target>
+
+  <!-- Clean all outputs, including NuGet packages. -->
+  <Target Name="CleanAll" DependsOnTargets="Clean">
+    <RemoveDir Directories='$([System.IO.Directory]::GetDirectories(".", "packages", SearchOption.AllDirectories))' />
   </Target>
 
   <!-- AKV Targets ========================================================= -->

eng/pipelines/common/templates/jobs/ci-build-nugets-job.yml

@@ -12,9 +12,13 @@ parameters:
     type: string
     default: ADO-MMS22-SQL19
 
-  - name: artifactName
+  - name: abstractionsArtifactName
     type: string
-    default: Artifacts
+    default: Abstractions.Artifact
+
+  - name: mdsArtifactName
+    type: string
+    default: MDS.Artifact
 
   - name: platform
     type: string
@@ -28,6 +32,11 @@ parameters:
     type: stepList
     default: []
 
+  - name: abstractionsPackageVersion
+    displayName: Abstractions Package Version Override
+    type: string
+    default: ''
+
 jobs:
 - job: build_nugets
 
@@ -44,12 +53,22 @@ jobs:
   - ${{ if ne(parameters.prebuildSteps, '') }}:
     - ${{ parameters.prebuildSteps }} # extra steps to run before the build like downloading sni and the required configuration
 
+  # Download the Abstractions package artifacts and put them in the packages/
+  # directory in the repo root.  This is where the MDS NuGet.config file will
+  # look for local packages.
+  - task: DownloadPipelineArtifact@2
+    displayName: Download Abstractions Package Artifact
+    inputs:
+      artifactName: ${{ parameters.abstractionsArtifactName }}
+      targetPath: $(Build.SourcesDirectory)/packages
+
   - template: ../steps/ci-project-build-step.yml@self
     parameters:
       platform: ${{ parameters.platform }}
       configuration: ${{ parameters.configuration }}
       operatingSystem: Windows
       build: all
+      abstractionsPackageVersion: ${{parameters.abstractionsPackageVersion}}
 
   - template: ../steps/generate-nuget-package-step.yml@self
     parameters:
@@ -70,8 +89,8 @@ jobs:
       installNuget: false
       displayName: 'Generate NuGet package AKV Provider'
 
-  - task: PublishBuildArtifacts@1
-    displayName: 'Publish Artifact: Artifacts'
+  - task: PublishPipelineArtifact@1
+    displayName: 'Publish Pipeline Artifact'
     inputs:
-      PathtoPublish: $(packagePath)
-      ArtifactName: ${{ parameters.artifactName }}
+      targetPath: $(packagePath)
+      artifactName: ${{ parameters.mdsArtifactName }}