Resource Provisioner: add retries to network calls

[danieljurek]

We've seen an error on Connect-AzAccount in a pipeline execution where other jobs have succeeded on this call. The cause might be a transient network issue. To that end we could wrap network calls in retries with some detection of transient errors (e.g. network timeout, 5xx status code, etc.) and retry on errors that look transient.

Logs of this behavior: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=234118&view=logs&jobId=42c9484e-bad6-5a7b-ceb1-abf0d427bbf2&j=42c9484e-bad6-5a7b-ceb1-abf0d427bbf2&t=05996a6a-8093-5d15-a4af-44930c1872b9

[danieljurek]

Might also make sense to use the -Debug flag on calls to Az

[heaths]

Testing -Debug, this can only be done in non-interactive PowerShell sessions otherwise it, as it should, prompts with each line. You can turn most of that off with -Confirm:$false but there are sub-scripts that still prompt, meaning you would probably have to set Set-PSDebug with the right parameters. I.e. it's not straight forward to do this. Write-Debug is meant solely for debugging. They should've used Write-Verbose to output more information. I think it's risky to try to do anything other than retrying at this point.

/cc @mikeharder

[mikeharder]

@heaths: Are you saying there is no good way to get more detailed information from CLI failures?

[heaths]

Not using -Debug - not without taking into account all the gotchas that entails to work correctly in both CIs and interactive. Good information like that should've used Write-Verbose. That said, I can still use trap or catch (depending on whether the exception is thrown out from the cmdlet or just an error record written) and dump any information that might be useful.

[heaths]

If I set $DebugPreference to 'Continue' and run in non-interactive mode, I can seemingly get this to work but do we gate that all on our $CI switch parameter? That information wouldn't benefit customers, since an interactive session is what we'd expect them to run. It's this various permutations to support that worry me. Catching that exception and dumping information would work in either case.