[CI Example Analyzer] Show line for ExampleIssues

documentation/Debugging-StaticAnalysis-Errors.md

@@ -9,6 +9,7 @@ Our StaticAnalysis tools help us ensure our modules follow PowerShell guidelines
     - [Signature Issues](#signature-issues)
     - [Help Issues](#help-issues)
     - [Example Issues](#example-issues)
+- [Troubleshotting Example Issues](#troubleshotting-example-issues)
 
 ## How to know if you have a StaticAnalysis Error
 If your build is failing, click on the Jenkins job inside the PR (marked as "Default" within checks).  Then check the Console Output within the Jenkins job.  If you have this error, then you have failed StaticAnalysis:
@@ -63,4 +64,39 @@ Example issues occur when your changed markdown files in the `help` folder (_e.g
 - Copy each of the errors you would like to suppress directly from the ExampleIssues.csv file output in the CI pipeline artifacts
 - Push the changes to the .csv file and ensure the errors no longer show up in the `ExampleIssues.csv` file output from the CI pipeline artifacts.
 
-If you have unexpected errors, please check whether you have splitted outputs from codes. If outputs cannot be separated from codes, then please add the tag `<!-- Skip: Output cannot be splitted from code -->` to the next line of the example title and in front of the code block.
+## Troubleshotting Example Issues
+### Scenario 1: Unexpected errors caused by the mixture of outputs and codes 
+PowerShell code and output are required to be in sepreated code blocks (```). If you have put outputs in the code block, then the outputs will be recognized as invalid PowerShell syntax. Please make sure you have splitted outputs from codes. The following shows the correct scene. Note that if the example has no output, you don't need to add an output block.
+### Example: Codes and outputs are split correctly
+````
+```powershell
+Get-AzConfig -EnableDataCollection
+```
+
+```output
+Key                           Value Applies To Scope       Help Message
+---                           ----- ---------- -----       ------------
+EnableDataCollection          False Az         CurrentUser When enabled, Azure PowerShell cmdlets send telemetry data to Microsoft to improve the custom…
+```
+````
+If outputs cannot be separated from codes, then please add the tag `<!-- Skip: Output cannot be splitted from code -->` to the next line of the example title and in front of the code block. The following is an example. 
+### Example: Add skip tag to the example whose outputs cannot be separated from codes
+````
+<!-- Skip: Output cannot be splitted from code -->
+```powershell
+$Context = Get-AzBatchAccountKey -AccountName myaccount
+$Context.PrimaryAccountKey
+ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMN==
+$Context.SecondaryAccountKey
+ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMN==
+```
+````
+
+### Scenario 2: Unexpected errors caused by unpaired quotes or brackets
+Please check whether you have matched the correct number of **quotes** and **brackets**. The common error messages in this scenario are as follows.
+- MissingEndParenthesisInExpression: Missing closing ')' in expression.
+- MissingEndCurlyBrace: Missing closing '}' in statement block or type definition.
+- MissingArrayIndexExpression: Array index expression is missing or not valid.
+- UnexpectedToken: Unexpected token xxx. (Check whether you have missed or added extra quote)
+
+In this scenario, many other unreasonable errors will occur. Leave them alone. Just make sure you have correct the number of **quotes** and **brackets** and rerun the CI verification. 

tools/StaticAnalysis/ExampleAnalyzer/AnalyzeRules/CommandName.psm1

@@ -113,7 +113,7 @@ function Measure-CommandName {
                 }
                 $ModuleCmdletExNum = $($CommandParameterPair[$i].ModuleCmdletExNum)
                 $Result = [Microsoft.Windows.PowerShell.ScriptAnalyzer.Generic.DiagnosticRecord]@{
-                    Message = "$ModuleCmdletExNum-#@#$Message#@#$Remediation";
+                    Message = "$Message#@#$Remediation";
                     Extent = $Asts[$i].Extent;
                     RuleName = $RuleName;
                     Severity = $Severity

tools/StaticAnalysis/ExampleAnalyzer/AnalyzeRules/ParameterNameAndValue.psm1

@@ -699,43 +699,43 @@ function Measure-ParameterNameAndValue {
             [System.Management.Automation.Language.Ast[]]$Asts = $ScriptBlockAst.FindAll($Predicate, $false)
             for ($i = 0; $i -lt $Asts.Count; $i++) {
                 if ($global:CommandParameterPair[$i].ParameterName -eq "" -and $global:CommandParameterPair[$i].ExpressionToParameter -eq "") {
-                    $Message = "$($CommandParameterPair[$i].ModuleCmdletExNum)-#@#$($CommandParameterPair[$i].CommandName) has a parameter not in the same ParameterSet as others."
+                    $Message = "$($CommandParameterPair[$i].CommandName) has a parameter not in the same ParameterSet as others."
                     $RuleName = [RuleNames]::Unknown_Parameter_Set
                     $Severity = "Error"
                     $RuleSuppressionID = "5010"
                     $Remediation = "Make sure the parameters are from the same parameter set."
                 }
                 elseif ($global:CommandParameterPair[$i].ExpressionToParameter -eq "") {
-                    $Message = "$($CommandParameterPair[$i].ModuleCmdletExNum)-#@#$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) is not a valid parameter name."
+                    $Message = "$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) is not a valid parameter name."
                     $RuleName = [RuleNames]::Invalid_Parameter_Name
                     $Severity = "Error"
                     $RuleSuppressionID = "5011"
                     $Remediation = "Check validity of the parameter -$($CommandParameterPair[$i].ParameterName)."
                 }
                 elseif ($global:CommandParameterPair[$i].ExpressionToParameter -eq "<duplicate>") {
-                    $Message = "$($CommandParameterPair[$i].ModuleCmdletExNum)-#@#$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) appeared more than once."
+                    $Message = "$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) appeared more than once."
                     $RuleName = [RuleNames]::Duplicate_Parameter_Name
                     $Severity = "Error"
                     $RuleSuppressionID = "5012"
                     $Remediation = "Remove redundant parameter -$($CommandParameterPair[$i].ParameterName)."
                 }
                 elseif ($null -eq $global:CommandParameterPair[$i].ExpressionToParameter) {
-                    $Message = "$($CommandParameterPair[$i].ModuleCmdletExNum)-#@#$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) must be assigned with a value."
+                    $Message = "$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) must be assigned with a value."
                     $RuleName = [RuleNames]::Unassigned_Parameter
                     $Severity = "Error"
                     $RuleSuppressionID = "5013"
                     $Remediation = "Assign value for the parameter -$($CommandParameterPair[$i].ParameterName)."
                 }
                 elseif ($global:CommandParameterPair[$i].ExpressionToParameter.EndsWith(" is a null-valued parameter value.")) {
-                    $Message = "$($CommandParameterPair[$i].ModuleCmdletExNum)-#@#$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) $($CommandParameterPair[$i].ExpressionToParameter)"
+                    $Message = "$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) $($CommandParameterPair[$i].ExpressionToParameter)"
                     $RuleName = [RuleNames]::Unassigned_Variable
                     $Severity = "Warning"
                     $RuleSuppressionID = "5110"
                     $variable = $CommandParameterPair[$i].ExpressionToParameter -replace " is a null-valued parameter value."
                     $Remediation = "Assign value for $variable."
                 }
                 elseif ($global:CommandParameterPair[$i].ParameterName -eq "<unknown>") {
-                    $Message = "$($CommandParameterPair[$i].ModuleCmdletExNum)-#@#$($CommandParameterPair[$i].CommandName) $($CommandParameterPair[$i].ExpressionToParameter) is not explicitly assigned to a parameter."
+                    $Message = "$($CommandParameterPair[$i].CommandName) $($CommandParameterPair[$i].ExpressionToParameter) is not explicitly assigned to a parameter."
                     $RuleName = [RuleNames]::Unbinded_Expression
                     $Severity = "Error"
                     $RuleSuppressionID = "5014"
@@ -744,7 +744,7 @@ function Measure-ParameterNameAndValue {
                 else {
                     $ExpressionToParameter = ($CommandParameterPair[$i].ExpressionToParameter -split "-#-")[0]
                     $ExpectedType = ($CommandParameterPair[$i].ExpressionToParameter -split "-#-")[1]
-                    $Message = "$($CommandParameterPair[$i].ModuleCmdletExNum)-#@#$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) $ExpressionToParameter is not an expected parameter value type."
+                    $Message = "$($CommandParameterPair[$i].CommandName) -$($CommandParameterPair[$i].ParameterName) $ExpressionToParameter is not an expected parameter value type."
                     $RuleName = [RuleNames]::Mismatched_Parameter_Value_Type
                     $Severity = "Warning"
                     $RuleSuppressionID = "5111"

tools/StaticAnalysis/ExampleAnalyzer/ExampleIssue.cs

@@ -24,6 +24,7 @@ public class ExampleIssue : IReportRecord
         public string Module { get; set; }
         public string Cmdlet { get; set; }
         public int Example { get; set; }
+        public string Line { get; set; }
         public string RuleName { get; set; }
         public string Extent { get; set; }
         public int ProblemId { get; set; }
@@ -32,13 +33,13 @@ public class ExampleIssue : IReportRecord
         public string Remediation { get; set; }
         public string PrintHeaders()
         {
-            return "\"Module\",\"Cmdlet\",\"Example\",\"RuleName\",\"ProblemId\",\"Severity\",\"Description\",\"Extent\",\"Remediation\"";
+            return "\"Module\",\"Cmdlet\",\"Example\",\"Line\",\"RuleName\",\"ProblemId\",\"Severity\",\"Description\",\"Extent\",\"Remediation\"";
         }
 
         public string FormatRecord()
         {
-            return string.Format("\"{0}\",\"{1}\",\"{2}\",\"{3}\",\"{4}\",\"{5}\",\"{6}\",\"{7}\",\"{8}\"",
-                Module, Cmdlet, Example, RuleName, ProblemId, Severity, Description, Extent, Remediation);
+            return string.Format("\"{0}\",\"{1}\",\"{2}\",\"{3}\",\"{4}\",\"{5}\",\"{6}\",\"{7}\",\"{8}\",\"{9}\"",
+                Module, Cmdlet, Example, Line, RuleName, ProblemId, Severity, Description, Extent, Remediation);
         }
 
         // The code that excludes exceptions is in tools/StaticAnalysis/ExampleAnalyzer/utils.ps1 Get-NonExceptionRecord.
@@ -51,28 +52,30 @@ public bool Match(IReportRecord other)
                 result = (record.Module == Module)&&
                 (record.Cmdlet == Cmdlet)&&
                 (record.Example == Example)&&
+                (record.Line == Line)&&
                 (record.Description == Description);
             }
             return result;
         }
 
         public IReportRecord Parse(string line)
         {
-            var matcher = "\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\"";
+            var matcher = "\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\",\"([^\"]*)\"";
             var match = Regex.Match(line, matcher);
-            if (!match.Success || match.Groups.Count < 10)
+            if (!match.Success || match.Groups.Count < 11)
             {
                 throw new InvalidOperationException(string.Format("Could not parse '{0}' as ExampleIssue record", line));
             }
             Module = match.Groups[1].Value;
             Cmdlet = match.Groups[2].Value;
             Example = int.Parse(match.Groups[3].Value);
-            RuleName = match.Groups[4].Value;
-            ProblemId = int.Parse(match.Groups[5].Value);
-            Severity = int.Parse(match.Groups[6].Value);
-            Description = match.Groups[7].Value;
-            Extent = match.Groups[8].Value;
-            Remediation = match.Groups[9].Value;
+            Line = match.Groups[4].Value;
+            RuleName = match.Groups[5].Value;
+            ProblemId = int.Parse(match.Groups[6].Value);
+            Severity = int.Parse(match.Groups[7].Value);
+            Description = match.Groups[8].Value;
+            Extent = match.Groups[9].Value;
+            Remediation = match.Groups[10].Value;
             return this;
         }
     }

tools/StaticAnalysis/ExampleAnalyzer/Measure-MarkdownOrScript.ps1

@@ -17,6 +17,7 @@ param (
     [string[]]$ScriptPaths,
     [Parameter(HelpMessage = "PSScriptAnalyzer custom rules paths. Empty for current path. Supports wildcard.")]
     [string[]]$RulePaths,
+    [string]$CodeMapPath,
     [switch]$Recurse,
     [switch]$IncludeDefaultRules,
     [string]$OutputFolder = "$PSScriptRoot\..\..\..\artifacts\StaticAnalysisResults\ExampleAnalysis",
@@ -28,16 +29,18 @@ param (
 . $PSScriptRoot\utils.ps1
 
 $analysisResultsTable = @()
-
-# Clean caches, remove files in "output" folder
-if ($OutputScriptsInFile.IsPresent) {
-    Remove-Item $OutputFolder\TempScript.ps1 -ErrorAction SilentlyContinue
-    Remove-Item $PSScriptRoot\..\..\..\artifacts\StaticAnalysisResults\ExampleIssues.csv -ErrorAction SilentlyContinue
-    Remove-Item $OutputFolder -ErrorAction SilentlyContinue
-}
+$codeMap = @()
+$totalLine = 1
 
 # Find examples in "help\*.md", output ".ps1"
 if ($PSCmdlet.ParameterSetName -eq "Markdown") {
+    # Clean caches, remove files in "output" folder
+    if ($OutputScriptsInFile.IsPresent) {
+        Remove-Item $OutputFolder\TempScript.ps1 -ErrorAction SilentlyContinue
+        Remove-Item $OutputFolder\TempCodeMap.csv -ErrorAction SilentlyContinue
+        Remove-Item $PSScriptRoot\..\..\..\artifacts\StaticAnalysisResults\ExampleIssues.csv -ErrorAction SilentlyContinue
+        Remove-Item $OutputFolder -ErrorAction SilentlyContinue
+    }
     $null = New-Item -ItemType Directory -Path $OutputFolder -ErrorAction SilentlyContinue
     $null = New-Item -ItemType File  $OutputFolder\TempScript.ps1
     # When the input $MarkdownPaths is the path of txt file
@@ -64,20 +67,27 @@ if ($PSCmdlet.ParameterSetName -eq "Markdown") {
             $cmdlet = $_.BaseName
             $result = Measure-SectionMissingAndOutputScript $module $cmdlet $_.FullName `
                 -OutputScriptsInFile:$OutputScriptsInFile.IsPresent `
-                -OutputFolder $OutputFolder
+                -OutputFolder $OutputFolder `
+                -TotalLine $totalLine
             $analysisResultsTable += $result.Errors
+            $codeMap += $result.CodeMap
+            $totalLine = $result.TotalLine
         }
     }
+    $codeMap| Export-Csv "$OutputFolder\TempCodeMap.csv" -NoTypeInformation
     if ($AnalyzeScriptsInFile.IsPresent) {
         $ScriptPaths = "$OutputFolder\TempScript.ps1"
+        $CodeMapPath = "$OutputFolder\TempCodeMap.csv"
     }
 }
 
 # Analyze scripts
 if ($PSCmdlet.ParameterSetName -eq "Script" -or $AnalyzeScriptsInFile.IsPresent) {
+    # Read code map from file
+    $codeMap = Import-Csv $CodeMapPath
     # Read and analyze ".ps1" in \ScriptsByExample
     Write-Output "Analyzing file ..."
-    $analysisResultsTable += Get-ScriptAnalyzerResult (Get-Item -Path $ScriptPaths) $RulePaths -IncludeDefaultRules:$IncludeDefaultRules.IsPresent -ErrorAction Continue
+    $analysisResultsTable += Get-ScriptAnalyzerResult (Get-Item -Path $ScriptPaths) $RulePaths -IncludeDefaultRules:$IncludeDefaultRules.IsPresent $codeMap -ErrorAction Continue
 
     # Summarize analysis results, output in Result.csv
     if($analysisResultsTable){