No Way to Control Order of Sources with nuget sources

[YipYipX4]

I have a network path where I have cached numerous *.nupkg files (from the public gallery) for other developers in my department with slow Internet but fast LAN access. I'd like to add this source at the top of the list so that during package restore, packages will be taken from the cache first (if available).

The command-line tool does not seem to offer any way to accomplish this. Sources are always added at the bottom of the list. I thought about removing the main "nuget.org" source and adding it back, but this seems to be a special entry that is listed twice, once with protocolVersion="3", and again, there doesn't seem to be a way to add that from the command line.

I would be satisfied with a "nuget sources move" command with a -Location parameter with options of "top" or "bottom" (also with a -Name parameter), or the "nuget sources add" command could have the -Location parameter (with default value of "bottom"). I guess the location could also be a 0-based index if you want better control, but the ability to insert at the top is the basic requirement.

A unit test for this could:

1. Start with the default config.
2. Run a command or sequence of commands that puts a new source at the top without interfering with the existing "nuget.org" source. For example, nuget.exe sources add -Name "Public NuGet Cache" -Source "\server\path\to\nuget\cache" -Location top
3. Compare nuget.config to a sample config file in the desired state.

Thanks!

[yishaigalatzer]

This is by design, sources are unordered and they are searched by order of the quickest response rather than by order of how they show up in file.

[YipYipX4]

Does that also apply to NuGet 2.x and older versions of Visual Studio (2010 - 2013)?

Why does Visual Studio show up and down arrows, allowing users to reorder the list of sources?

(I currently have a Word document to help developers in my group setup their systems for development, and I already have instructions to move a source to the top using the GUI, and I'm converting a lot of those system setup steps into a PowerShell script.)

[wkit23]

I also faced a similar issue with the source sequence. I have a local repository and an internal feed to host my own packages. When building, projects will be packaged and dump to the local repository so that sub sequence reference will refer to the local repository. When I finished build, the packages from the local repository will be uploaded to the internal feed.

Now, my problem is, I set the local repository to be before the internal feed, thinking that it should always try to get from local repository before moving to internal feed. However, it doesn't seems to be so as I always encounter problem on either cannot find new packages, or it's using the older packages version from the internal feed which was build previously.

Even if you says that it'll choose the fastest response, I still don't think it makes sense. The local repository should be faster than internal feed server, but the actual runs says otherwise.

[yishaigalatzer]

Are you versioning the packages on your builds or keeping the same version? If it's the latter that could be the cause of the issue

[wkit23]

Yes, you're right. What we're doing is something as follow:

• We develop using version 1.0

• In build, we change the version to the actual version.

• Build, package, and then for the projects that requires those packages, we use Nuget Update to update them to the actual version.

But I actually wonder, why it didn't get it from local folder but instead take it from the internal feed? Any way for us to trace how nuget choose from which source?

[EamonNerbonne]

This would be quite handy also to evaluate temporary fixes to packages. I.e. if a package likely contains (or triggers) a bug, then it's useful to be able to make a change to that package, and see if that change really fixes downstream dependencies you care about, and then repeat that process a few times until you settle on the most appropriate fix (which you then can publish as usual).

[zachweg]

@yishaigalatzer
I think this is a good design but looking for more clarity on this issue...
Is package source search order by quickest response specific to nuget.exe client? Otherwise why does Visual Studio NuGet Package Manager Settings allow you to order the available package sources? There is also contradictory documentation on docs.microsoft.com (see step 4):
https://docs.microsoft.com/en-us/nuget/consume-packages/package-restore#build-time-restore-in-visual-studio

[BradleyUffner]

I would also like an answer to the questions @zachweg raised. Why is the documentation conflicting? What is the actual behavior? If order doesn't matter, why does Visual Studio allow it to be changed? If order does matter, how do I control the order when multiple sources are added from different nuget.config files?

[AceHack]

Please allow for order, I rely on order.

[YipYipX4]

I agree that we should have control over the order. I'm also not convinced that it even goes by fastest response as that hasn't been my experience, but in any case, as others have pointed out, some people want to be able to specify a preferred source. There could be an option to enable or disable the "poll for fastest source" feature, and it could be enabled by default but something we can control.

[walter-w]

It doesn't use the fastest. I have a local repo setup on my disk which it doesn't attempt to use

[AceHack]

I need the ability to order/prioritize sources, please add.

[RobertArmic]

I have a VERY valid use case for needing this, Azure DevOps private feed has a feature called "upstream sources" that is almost useless if we cannot guarantee order.

We have our private feed with upstream to nuget.org, but because searches are not upstreamed we also have to add nuget.org directly to nuget.config, so that developers can find new packages from Nuget Manager in Visual Studio. Since we can not force order, upstream sources sometimes work and sometimes not, we would like to force all already-downloaded packages to go to our private feed and only first-timers going to nuget.org. Without forced order we have this broken functionality when upstream source is sometimes used and sometimes not, also having statistic on number of download requests from private feed useless.

[mlhpdx]

Likewise, I would much rather have the order be deterministic rather than the whim of internet latency (or otherwise randomly). This will be particularly important as packages migrate to Github from Nuget.org, where it will be necessary to check the "normative" source first and the "legacy source" as fallback.

[rob3c]

It's clear that internet latency has nothing to do with it, or my local drive would win every time (which it doesn't).

It also seems clear that nuget.org doesn't mind all of the requests for private packages and corresponding IP addresses leaking into their server logs, or they'd likely have had motivation to get this breaking change fixed long ago.

[rrelyea]

Somebody mentioned that they couldn't find the comment I referred to in #7398 - "@AceHack - see my comments about different techniques that one can control order in my comment from today on #3676"

When looking for a package, a NuGet search in the following order:

1. in the global package folder
2. in fallback folders configured
3. in file: feeds configured in nuget.config files
4. in http: feeds configured in nuget.config files

Would love to understand people's motivation for desiring ordering.

We strongly believe you should not have different content in the same identity package (Foo.Bar.1.2.3.nupkg found on any feed should be identical)

[mlhpdx]

Somebody mentioned that they couldn't find the comment I referred to in #7398 - "@AceHack - see my comments about different techniques that one can control order in my comment from today
We strongly believe you should not have different content in the same identity package (Foo.Bar.1.2.3.nupkg found on any feed should be identical)

And we strongly agree with that, but it isn't the problem. I have existing packages on Nuget.org but for current and future development I want to publish to GitHub package hosting since it supports event hooks and other niceties. So, all my consumer will need both feeds since new versions of my packages will be on Github but old version and other necessary packages will be on Nuget.org. With that setup, particularly when preview packages are involved, things get random -- they neither consistently work or fail. That's just nuts.

[greektreat]

I find this strange I have been using VS for years, I haven't had to make NuGet Packages for a while but when I did I was able to order the Package sources in order of importance. I believe I used VS 2012 at the time. I am trying to do that now and noticed this issue

[varunmankal]

Is there any work around to control the order of nuget package sources ?

[samcragg]

@rrelyea I see where you're coming from; ordinarily a package should be the same on each feed so, by that logic, the order is unimportant. However, here's a concrete example of what I'm trying to achieve to see why I think it's important that the package sources should be searched in the order they are specified (or some other way of controlling the search order).

As part of my build I create a (relatively complicated) package and then have an integration test to use that package on a test project. Since I want it to use the version that I just created, I specify the RestoreSources to include the directory where the package I just built is (it's the only thing in that folder). However, because it's a dotnet core project, it also needs Microsoft.AspNetCore.App.Ref, so I must use the nuget.org feed too. However, the nuget feed always gets searched for before my local directory, meaning the integration test isn't actually testing the package I just built but the one on nuget.org.

I don't think this is a unique requirement to be able to test your locally generated package, CoreRT does the same thing so any help to achieve this would be appreciated.

Can we at least get this issue open again please? Thanks.

[rob3c]

Order is also important when you’d like to mix private and public feeds and not leak private package requests to public servers. Of course, entities running public feeds don’t necessarily mind sucking up all of that private request data, and may not be motivated to restore deterministic feed order.

[smchristensen]

My use case is identical to Longsarg (from Dec 14, 2019 above). I admit this is not necessarily NuGet's problem to solve, but Azure Artifacts has the ability to have upstream sources in an Azure Artifact feed. It however does not index the upstream feed therefore I am unable to search for nuget.org packages from the Visual Studio package manager UI when I only have the Azure Artifacts feed defined in the package sources. I am only able to find a nuget.org package when I also have the public nuget feed configured in the VS package sources. If I am able to find the package, I could theoretically pull it from the Azure Artifact feed because I have upstream sources configured in the Azure Artifact feed. And if I can pull them through the Azure Artifact feed, the pacakge will be saved locally in the Azure Artifact feed which is what I'm looking for.

I could get around the Azure Artifact limitation if I could configure both my Azure Artifact feed and the public nuget.org feed in the Visual Studio package manager sources and have the Azure Artifact feed be the primary download feed. Essentially, I'd use the nuget.org feed for searching and the Azure Artifact feed for pulling and storing the packages.

I don't know if that makes it any clearer. And even though this isn't exactly a NuGet issue, I think it would be a useful feature to have. Thanks!

[Belorus]

We have local artifactory in the office which is a preferred source, but we can't remove nuget official repo since new packages can only be found there.

[augustoproiete]

@Belorus I'm hoping this story will improve with #10214 / #6867 when you can restrict the feed that should be searched for each package. That way you can whitelist the packages you know to always go on your company's feed.

In the meantime, as a workaround, you might want to consider using Nexus repository as your main feed, and put it in front of your office artifactory feed and nuget.org.

Nexus has a concept of proxy repository where it can get a package from nuget.org automatically if it doesn't exist in its cache, and it also has the concept of repository groups, which can aggregate multiple nuget feeds and make them look like a single feed. So that way you can point Visual Studio to a single feed (your company's Nexus instance) and let it do its magic for you.

[nkolev92]

@samcragg

I think you are looking for #6579.
Your problem wouldn't be addressed easily by source priority because the global packages folder is not a cache, so never refreshed. Meaning if you make edits for the package, you'd need to delete the global packages folder. At that point, you're better off using a in development package version, that won't clash with the publicly available ones.

[samcragg]

Thanks for the link to that issue @nkolev92 - looks like I'm not the only one who wants to test their packages locally before publishing them 🙂

I'm not actually concerned about the global cache, as for my integration tests I specify the folder to restore the packages to, so I ensure the folder is nuked before running any of the integration tests:

dotnet restore -v:q --no-cache --packages "path/to/temporary/test/directory"

I've manually checked and can confirm that the package isn't in my local NuGet cache when running the above.

The problem I have is I specify the RestoreSources in the csproj file but the order is ignored, meaning it always searches NuGet before my local directory. To be honest, I'd be happy if there was an option to search them all and use the latest version it finds, but that doesn't seem to be the behaviour (I still don't understand why a search on NuGet is prefered over a local folder - from a performance perspective this makes no sense).

Hope that clarifys a little why I think the order is so important - it's the one thing blocking me at the moment.

[nkolev92]

NuGet checks local feeds prior to checking online feeds. Pretty confident about that :) If you have a minimal repro, we'd love to get another issue about that.

See: https://docs.microsoft.com/en-us/nuget/concepts/package-installation-process

[johnkors]

@nkolev92 / @rrelyea , could you comment how nuget hinder the vulnerability mentioned in this article?

https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

E.g. If i as an attacker upload a package to nuget.org with an the same id (using the same version or even as a newer patch version, so they use that when they patch) that I know an organisation only have in their internal http feed, could I hijack what they get in their restore process?

If nuget restore is choosing the feeds randomly or just by which is faster, couldn't that lead to my exploit package being restored?

[JonDouglas]

@johnkors Hi John,

See the following issue.

#10566 (comment)

Feel free to look through the resources currently available.

[SeymourRu]

Order is important. There are many cases when order literally matters so this situation should be revised for a good.

[nkolev92]

@SeymourRu Please take a look at https://github.com/NuGet/Home/blob/dev/proposed/2021/PackageNamespaces.md.

[SeymourRu]

@SeymourRu Please take a look at https://github.com/NuGet/Home/blob/dev/proposed/2021/PackageNamespaces.md.

@nkolev92 Yeah, thank you, this looks really promising!
Approximately how long it will take to be released?

[nkolev92]

@SeymourRu

Visual Studio 2022 preview 3 will be released within the next few weeks and it will have restore support for the feature.
We're actively working on the other parts.

[MichaelKetting]

I just learned that the originally documented behavior regarding source ordering was invalidated in 2019 after I spent some time hunting down the reason why my process with prioritized nuget sources no longer works.

My use case is that I have different package configurations uploaded on nuget.org and private feeds (e.g. release/debug and packages with symbols includes and in separate packages) and am using the correct source for the correct usecase. Now, the packages I have on nuget.org are suddenly restored instead of those on the private feed.

I have looked into the Package Namespace feature and I can probably use this but it does significantly complicate my scenario and tooling involved.

[nkolev92]

Please refer to https://devblogs.microsoft.com/nuget/introducing-package-source-mapping/