DPG best-practice and automation template (#28016)

* Add request condition argument check

* resolve build failure

* remove unused import

* remove unused Azure.Core dependency

* move eng template to sdk

* update sample template to DPG + growup

* update test cases

* remove the templates from eng directory

* update Azure.ServiceTemplate.Template

* merge Azure.ServiceTemplate.Template with Azure.Template

* update api

* resolve rebase conflict

* complete Azure.Template to match DPG guideline

* refine template samples

* update template content

* remove project file in template content

* refine snippet in Readme

* resolve inner scope conflict issule

* refine test

* remove duplicate snippet

* update template sample

* update script to use the Azure.Template

* add function to update ci.yml

* remove log in script

* resolve comments

* use actual type instead of var

* resolve comments

* refine readme files, add error checking in outer script

* add Azure.Template as the content of template

* move .content folder out of Azure.Template

doc/DataPlaneCodeGeneration/AzureSDKCodeGeneration_DataPlane_Quickstart.md

@@ -61,7 +61,7 @@ sdk\<service name>\<package name>\CHANGELOG.md
 - `<service name>` - Should be the short name for the azure service. e.g. deviceupdate
 - `<package name>` -  Should be the name of the shipping package, or an abbreviation that distinguishes the given shipping artifact for the given service. It will be `Azure.<group>.<service>`, e.g. Azure.IoT.DeviceUpdate
 
-We will use dotnet project template [Azure.ServiceTemplate.Template](https://github.com/Azure/azure-sdk-for-net/blob/3ac301ac6435c818ad7a9946ab1c4023cee236ff/eng/templates) to automatically create the project.
+We will use dotnet project template [Azure.Template](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/template/Azure.Template) to automatically create the project.
 
 You can run `eng\scripts\automation\Invoke-DataPlaneGenerateSDKPackage.ps1` to generate the starting SDK client library package directly as following:
 
@@ -80,12 +80,13 @@ pwsh /home/azure-sdk-for-net/eng/scripts/automation/Invoke-DataPlaneGenerateSDKP
 - For `- namespace`, please use one of the pre-approved namespace groups on the [.NET Azure SDK Guidelines Approved Namespaces list](https://azure.github.io/azure-sdk/dotnet_introduction.html#dotnet-namespaces-approved-list). This value will also provide the name for the shipped package, and should be of the form `Azure.<group>.<service>`.
 - `-inputfiles` takes the address of the Open API spec files,  separated by semicolon if there is more than one file.  The Open API spec file can be local file, e.g. ./swagger/compute.json, or the web address of the file in the `azure-rest-api-specs` repo.  When pointing to a file in the `azure-rest-api-specs` repo, make sure to include the commit id in the URI, i.e. `https://github.com/Azure/azure-rest-api-specs/blob/73a0fa453a93bdbe8885f87b9e4e9fef4f0452d0/specification/webpubsub/data-plane/WebPubSub/stable/2021-10-01/webpubsub.json`.  This ensures that you can choose the time to upgrade to new swagger file versions.
 - `-readme` takes the address of the readme configuration file. The configuration can be local file, e.g. ./swagger/readme.md or the web address of the file in the `azure-rest-api-specs` repo, i.e. `https://github.com/Azure/azure-rest-api-specs/blob/23dc68e5b20a0e49dd3443a4ab177d9f2fcc4c2b/specification/deviceupdate/data-plane/readme.md`
+- You need to provide one of `-inputfiles` and `-readme` parameters. If you provide both, `-inputfiles` will be ignored.
 - `-securityScope` designates the authentication scope to use if your library supports **Token Credential** authentication.
 - `-securityHeaderName` designates the key to use if your library supports **Azure Key Credential** authentication.
 
 When you run the `eng\scripts\automation\Invoke-DataPlaneGenerateSDKPackage.ps1` script, it will:
 
-- Create a project folder, install template files from `eng/templates/Azure.ServiceTemplate.Template`, and create `.csproj` and `.sln` files for your new library.
+- Create a project folder, install template files from `sdk/template/Azure.Template`, and create `.csproj` and `.sln` files for your new library.
 
     These files are created following the guidance for the [Azure SDK Repo Structure](https://github.com/Azure/azure-sdk/blob/master/docs/policies/repostructure.md).
 
@@ -111,8 +112,7 @@ Here is the step by step process to add tests:requirements
 
 - Add other client parameters in `<client-name>ClientTestEnvironment.cs`
 - Update `<client-name>ClientTest.cs`.
-  - Comment-out the 'CreateClient' method, and update the new `<service>Client` statement.
-  - remove all the template project tests, and write the tests according to the commented Test method template. Please refer to [Using the TestFramework](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core.TestFramework/README.md) to add tests.
+  - Please refer to [Using the TestFramework](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core.TestFramework/README.md) to add tests.
 
 **Note**:
 
@@ -131,13 +131,13 @@ You will update all the `Sample<sample_number>_<scenario>.md` and README.md file
 
 ### Snippets
 
-Snippets are the great way to reuse the sample code. Snippets allow us to verify that the code in our samples and READMEs is always up to date, and passes unit tests. We have added the snippet [here](https://github.com/Azure/azure-sdk-for-net/blob/3ac301ac6435c818ad7a9946ab1c4023cee236ff/eng/templates/Azure.ServiceTemplate.Template/tests/Samples/Sample1_CreateResource.cs#L32) in a sample and used it in the [README](https://github.com/Azure/azure-sdk-for-net/blob/3ac301ac6435c818ad7a9946ab1c4023cee236ff/eng/templates/Azure.ServiceTemplate.Template/README.md#create-resource). Please refer to [Updating Sample Snippets](https://github.com/Azure/azure-sdk-for-net/blob/main/CONTRIBUTING.md#updating-sample-snippets) to add snippets in your samples.
+Snippets are the great way to reuse the sample code. Snippets allow us to verify that the code in our samples and READMEs is always up to date, and passes unit tests. We have added the snippet [here](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/template/Azure.Template/tests/Samples/Sample1_HelloWorld.cs#L21) in a sample and used it in the [README](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/template/Azure.Template/README.md#get-secret). Please refer to [Updating Sample Snippets](https://github.com/Azure/azure-sdk-for-net/blob/main/CONTRIBUTING.md#updating-sample-snippets) to add snippets in your samples.
 
 ### README
 
 README.md file instructions are listed in `Azure.<group>.<service>/README.md` file. Please add/update the README.md file as per your library.
 
-**Learn more:** to understand more about README, see the [README.md](https://github.com/Azure/azure-sdk-for-net/blob/3ac301ac6435c818ad7a9946ab1c4023cee236ff/eng/templates/Azure.ServiceTemplate.Template/README.md). Based on that [here](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/keyvault/Azure.Security.KeyVault.Keys/README.md) is an example.
+**Learn more:** to understand more about README, see the [README.md](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/template/Azure.Template/README.md). Based on that [here](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/keyvault/Azure.Security.KeyVault.Keys/README.md) is an example.
 
 ### Changelog
 
@@ -151,7 +151,7 @@ You can add convienice APIs by adding a customization layer on top of the genera
 
 If other modifications are needed to the generated API, you can consider making them directly to the Open API specification, which will have the benefit of making the changes to the library in all languages you generate the library in.  As a last resort, you can add modifications with swagger transforms in the `autorest.md` file.  [AnomalyDetector autorest.md](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/anomalydetector/Azure.AI.AnomalyDetector/src/autorest.md) shows and example of how this can be accomplished.
 
-Once you've made changes to the public API, you will need to run the `eng\scripts\Export-API.ps1` script to update the public API listing. This will generate a file in the library's directory similar to the example found in [eng\templates\Azure.ServiceTemplate.Template\api\Azure.ServiceTemplate.Template.netstandard2.0.cs](https://github.com/Azure/azure-sdk-for-net/blob/bb0fbccfc33dd27d1ec6f0870022824d47181e61/sdk/template-dpg/Azure.ServiceTemplate.Template/api/Azure.ServiceTemplate.Template.netstandard2.0.cs).
+Once you've made changes to the public API, you will need to run the `eng\scripts\Export-API.ps1` script to update the public API listing. This will generate a file in the library's directory similar to the example found in [sdk\template\Azure.Template\api\Azure.Template.netstandard2.0.cs](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/template/Azure.Template/api/Azure.Template.netstandard2.0.cs).
 
 e.g. Running the script for a project in `sdk\deviceupdate` would look like this:
 

eng/scripts/automation/GenerateAndBuildLib.ps1

@@ -1,5 +1,5 @@
 #Requires -Version 7.0
-
+$CI_YAML_FILE = "ci.yml"
 function Get-SwaggerInfo()
 {
     param(
@@ -85,6 +85,30 @@ function Update-AutorestConfigFile() {
         exit 1
     }
 }
+
+function Update-CIYmlFile() {
+    param (
+        [string]$ciFilePath,
+        [string]$artifact
+    )
+    if (Test-Path -Path $ciFilePath) {
+        $packageRex = "name *: $artifact"
+        if ((Get-Content $ciFilePath | Select-String -Pattern $packageRex).Matches.Success) {
+            Write-Host "CI already enabled."
+        } else {
+            $safeName = $artifact.Replace('.', '')
+            $artifactsBlockRex = "Artifacts *:"
+            $startNum = (Get-Content $ciFilePath | Select-String -Pattern $artifactsBlockRex).LineNumber[0]
+            $fileContent = Get-Content -Path $ciFilePath
+            $fileContent[$startNum - 1] += ([Environment]::NewLine + "    - " + "name: $artifact" + [Environment]::NewLine + "      safeName: $safeName")
+            $fileContent | Set-Content $ciFilePath
+        }
+    } else {
+        Write-Error "ci.yml doesn't exist."
+        exit 1
+    }
+}
+
 function New-DataPlanePackageFolder() {
   param(
       [string]$service,
@@ -103,41 +127,50 @@ function New-DataPlanePackageFolder() {
   $inputfile = ""
   $fileArray = $inputfiles.Split(";")
   if (($inputfiles -ne "") -And ($fileArray.Length -gt 0)) {
-    $inputfile = "- " + $fileArray[0];
-    for ($i = 1; $i -lt $fileArray.Count ; $i++) {
+    for ($i = 0; $i -lt $fileArray.Count ; $i++) {
         $inputfile = $inputfile + [Environment]::NewLine + "- " + $fileArray[$i]
     }
   }
 
+  $serviceFolder = (Join-Path $sdkPath "sdk" $service)
+  if (!(Test-Path -Path $serviceFolder)) {
+    Write-Host "service folder does not exist! create the folder $serviceFolder"
+    New-Item -Path $serviceFolder -ItemType Directory
+  }
   $projectFolder=(Join-Path $sdkPath "sdk" $service $namespace)
+  $ciymlFilePath =(Join-Path $sdkPath "sdk" $service $CI_YAML_FILE)
   $apifolder = (Join-Path $projectFolder "api")
   Write-Host "projectFolder:$projectFolder, apifolder:$apifolder"
   if ((Test-Path -Path $projectFolder) -And (Test-Path -Path $apifolder)) {
     Write-Host "Path exists!"
     # update the input-file url if needed.
     $file = (Join-Path $projectFolder "src" $AUTOREST_CONFIG_FILE)
     Update-AutorestConfigFile -autorestFilePath $file -inputfile $inputfile -readme $readme
+    Update-CIYmlFile -ciFilePath $ciymlFilePath -artifact $namespace
   } else {
     Write-Host "Path doesn't exist. create template."
     if ($inputfile -eq "" -And $readme -eq "") {
         Write-Error "Error: input file should not be empty."
         exit 1
     }
-    dotnet new -i $sdkPath/eng/templates/Azure.ServiceTemplate.Template
+    dotnet new -i $sdkPath/sdk/template
     Write-Host "Create project folder $projectFolder"
-    New-Item -Path $projectFolder -ItemType Directory
-    Push-Location $projectFolder
+    if (Test-Path -Path $projectFolder) {
+        Remove-Item -Path $projectFolder -ItemType Directory
+    }
+
+    Push-Location $serviceFolder
     $namespaceArray = $namespace.Split(".")
     if ( $namespaceArray.Count -lt 3) {
         Write-Error "Error: invalid namespace name."
         exit 1
     }
 
-    $libraryName = $namespaceArray[-1]
+    $clientName = $namespaceArray[-1]
     $groupName = $namespaceArray[1]
-    $dotnetNewCmd = "dotnet new dataplane --libraryName $libraryName --groupName $groupName --includeCI true --force"
+    $dotnetNewCmd = "dotnet new azsdkdpg --name $namespace --clientName $clientName --groupName $groupName --serviceDirectory $service --force"
     if ($inputfile -ne "") {
-        $dotnetNewCmd = $dotnetNewCmd + " --swagger $inputfile"
+        $dotnetNewCmd = $dotnetNewCmd + " --swagger '$inputfile'"
     }
     if ($securityScope -ne "") {
         $dotnetNewCmd = $dotnetNewCmd + " --securityScopes $securityScope";
@@ -147,13 +180,21 @@ function New-DataPlanePackageFolder() {
         $dotnetNewCmd = $dotnetNewCmd + " --securityHeaderName $securityHeaderName";
     }
 
-    # dotnet new dataplane --libraryName $libraryName --swagger $inputfile --securityScopes $securityScope --securityHeaderName $securityHeaderName --includeCI true --force
+    if (Test-Path -Path $ciymlFilePath) {
+        Write-Host "ci.yml already exists. update it to include the new serviceDirectory."
+        Update-CIYmlFile -ciFilePath $ciymlFilePath -artifact $namespace
+
+        $dotnetNewCmd = $dotnetNewCmd + " --includeCI false"
+    }
+    # dotnet new azsdkdpg --name $namespace --clientName $clientName --groupName $groupName --serviceDirectory $service --swagger $inputfile --securityScopes $securityScope --securityHeaderName $securityHeaderName --includeCI true --force
     Write-Host "Invote dotnet new command: $dotnetNewCmd"
     Invoke-Expression $dotnetNewCmd
 
     $file = (Join-Path $projectFolder "src" $AUTOREST_CONFIG_FILE)
-    Update-AutorestConfigFile -autorestFilePath $file -inputfile $inputfile -readme $readme
+    Update-AutorestConfigFile -autorestFilePath $file -readme $readme
+    Pop-Location
     # dotnet sln
+    Push-Location $projectFolder
     dotnet sln remove src\$namespace.csproj
     dotnet sln add src\$namespace.csproj
     dotnet sln remove tests\$namespace.Tests.csproj
@@ -244,6 +285,26 @@ function Invoke-Generate() {
     $sdkfolder = $sdkfolder -replace "\\", "/"
     Push-Location $sdkfolder/src
     dotnet build /t:GenerateCode
+    if ( !$? ) {
+        Write-Error "Failed to generate sdk."
+        Pop-Location
+        exit 1
+    }
+    Pop-Location
+}
+
+function Invoke-Build() {
+    param(
+        [string]$sdkfolder= ""
+    )
+    $sdkfolder = $sdkfolder -replace "\\", "/"
+    Push-Location $sdkfolder
+    dotnet build
+    if ( !$? ) {
+        Write-Error "Failed to build sdk. exit code: $?"
+        Pop-Location
+        exit 1
+    }
     Pop-Location
 }
 
@@ -254,6 +315,11 @@ function Invoke-Pack() {
     $sdkfolder = $sdkfolder -replace "\\", "/"
     Push-Location $sdkfolder
     dotnet pack
+    if ( !$? ) {
+        Write-Error "Failed to build sdk package. exit code: $?"
+        Pop-Location
+        exit 1
+    }
     Pop-Location
 }
 function Get-ResourceProviderFromReadme($readmeFile) {

eng/scripts/automation/Invoke-DataPlaneGenerateSDKPackage.ps1

@@ -22,9 +22,16 @@ $outputJson = Get-Content $outputJsonFile | Out-String | ConvertFrom-Json
 $projectFolder = $outputJson.projectFolder
 Write-Host "projectFolder:$projectFolder"
 Remove-Item $outputJsonFile
+# Generate Code
 Invoke-Generate -sdkfolder $projectFolder
-if ( $? -ne $True) {
-  Write-Error "Failed to create generate sdk."
+if ( !$? ) {
+  Write-Error "Failed to generate sdk."
+  exit 1
+}
+# Build
+Invoke-Build -sdkfolder $projectFolder
+if ( !$? ) {
+  Write-Error "Failed to build sdk. exit code: $?"
   exit 1
 }
 # Generate APIs

eng/scripts/automation/Invoke-GenerateAndBuild.ps1

@@ -30,7 +30,7 @@ if ( $serviceType -eq "resource-manager" ) {
   Write-Host "Data-plane SDK Generation is not implemented currently."
   exit 1
 }
-if ( $? -ne $True) {
+if ( !$? ) {
   Write-Error "Failed to create sdk project folder. exit code: $?"
   exit 1
 }
@@ -39,12 +39,11 @@ $projectFolder = $newpackageoutputJson.projectFolder
 $path = $newpackageoutputJson.path
 Write-Host "projectFolder:$projectFolder"
 Remove-Item $newpackageoutput
-
+# Generate Code
 Invoke-Generate -sdkfolder $projectFolder
-if ( $? -ne $True) {
-  Write-Error "Failed to generate sdk. exit code: $?"
-  exit 1
-}
+# Build
+Invoke-Build -sdkfolder $projectFolder
+
 $outputJson = [PSCustomObject]@{
     packages = @([pscustomobject]@{packageName="$packageName"; result='succeeded'; path=@("$path");packageFolder="$path"})
 }