TryGetDownloadCountForPackageRegistration can throw a modified enumerator exception

[joelverhagen]

This results in an HTTP 500 for the user. This has happened at least 6 times in the last 7 days on PROD.

AI query

requests
| where timestamp > ago(90d)
| where name contains "PushPackageApi"  or name contains "verify" 
| where name startswith "PUT" or name startswith "POST"
| summarize sum(itemCount) by bin(timestamp, 1d), tostring(resultCode)

requests
| where timestamp > ago(7d)
| where name contains "PushPackageApi"  or name contains "verify" 
| where name startswith "PUT" or name startswith "POST"
| where resultCode == 500
| join kind=inner (
   exceptions 
) on $left.operation_Id == $right.operation_Id
| where innermostMessage == "Collection was modified; enumeration operation may not execute."
| project problemId

This is where the exception is being thrown from:

NuGetGallery/src/NuGetGallery/Services/CloudDownloadCountService.cs

Line 53 in 4607a1d

downloadCount = _downloadCounts[id].Sum(kvp => kvp.Value);

Probably need some locking.

[joelverhagen]

FYI this has happened 76 times in the last 90 days.

exceptions
| where timestamp > ago(90d)
| where innermostMessage == "Collection was modified; enumeration operation may not execute."

[joelverhagen]

This is related to the push flow because every time we materialize a PackageRegistration record via EF, we call into CloudDownloadCountService.

[joelverhagen]

I've merged the fix to dev branch. It is not validated anywhere except my devbox.