Investigate missing symbol publishing for all repos and branches using V3

[riarenas]

• This issue is blocking

This is blocking msbuild insertion into VS.

• This issue is causing unreasonable pain

There seems to have been a regression in symbol publishing that might be related to a661399

The logs for the latest maestro publishing build no longer shows any indication that symbols are actually being pushed to the symbol servers:

https://dev.azure.com/devdiv/DevDiv/_build/results?buildId=5309117&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a.

[riarenas]

@epananth FYI

[benvillalobos]

@dotnet/kitten FYI

[hoyosjs]

This is actually affecting all repos - runtime is also having this issue.

[hoyosjs]

This is also affecting release branches (apparently publish using darc always points to the yml of the main branch).

[hoyosjs]

https://dev.azure.com/dnceng/internal/_build/results?buildId=1412669

A revert of the specified commit didn't help. Something else seems to be causing this issue.

[ViktorHofer]

Official builds targeting both 5.0 and 6.0 are failing for dotnet/runtime presumably because of this. Ie https://dev.azure.com/dnceng/internal/_build/results?buildId=1412894&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a

[hoyosjs]

This one is different - and new...

[ViktorHofer]

Could this be the reason for the missing symbols here: dotnet/runtime#57259 (comment)?

[michellemcdaniel]

@alexperovich Is it possible something about the new publishing work you're doing affected symbols publishing?

[michellemcdaniel]

My best guess on this, from just scanning the logs, is that the serversToPublish list (ie the list of servers to publish to) is empty.

[riarenas]

Yep, that's a good assessment. I'll see if I can find where the targetFeedConfigs disconnect is happening.

[alexperovich]

With my publishing changes we made a specific decision to publish symbols to azure storage instead of the nuget feeds they used to be in. I didn't touch any of the code that uploads to msdl or symweb, but potentially it got disconnected somewhere. I will check.

[riarenas]

This call seems to be returning empty:

arcade/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestBase.cs

Lines 670 to 671 in 8d89c24

	Dictionary<string, string> serversToPublish =
	GetTargetSymbolServers(feedConfigsForSymbols, msdlToken, symWebToken);

so there's probably some disconnect around the feedConfigsForSymbols

[alexperovich]

Yep, it looks like thats the problem. "SymbolTargetType" is unset for the new feed configs. I will push a fix.

[alexperovich]

#8024

[riarenas]

Thanks Alex. I'll make sure to get the fix into the publishing pipeline as soon as it's available.

[riarenas]

#8027 should fix this once merged. The validation build at https://dev.azure.com/dnceng/internal/_build/results?buildId=1416538&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a published symbols again as it used to.

[riarenas]

Change is merged and newer runs should publish symbols again. I'll look at some runs for the publishing pipeline tomorrow and close this if things look good.

[riarenas]

Newer runs are attempting to publish symbols. There's a new problem I'm seeing during symbol publishing so I opened #8033 for that.