Paket restore fails randomly when "msbuilding" solution in parallel

[dzendras]

There seems to be a problem with paket restore when using msbuild/p:BuildInParallel=true switch.
It happens on our build machine randomly, but frequently (every day):

C:\SolutionDir.paket\paket.targets(36,5): error MSB3073: The command ""C:\SolutionDir.paket\paket.exe" restore --references-files "C:\SolutionDir\FooProject\paket.references" " exited with code 1. [C:\SolutionDir\FooProject\FooProject.csproj]

When using paket restore with --log-file (BTW why isn't it available on the paket's commands list displayed when you run paket without any arguments?) I could see the cause of the failures:

packages folder is locked by paket.exe (PID = 4220). Waiting...

Paket failed with:
The process cannot access the file 'C:\SolutionDir\packages\paket.locked' because it is being used by another process.

The workaround is to run paket restore before running msbuild.
Is it possible to make paket support such a scenario out of the box?

[forki]

We tried to support this scenario, but there is no "before solution build"
event in MSBuild. So everything hooks into project build events.
The only way out was to try to serialize the requests and therefore all
paket.exe are waiting for the first to finish.

It seems the timeout is too short.
On Jun 3, 2015 11:52 AM, "Jedrzej Grabowski" [email protected]
wrote:

There seems to be a problem with paket restore when using
msbuild/p:BuildInParallel=true switch.
It happens on our build machine randomly, but frequently (every day):

C:\SolutionDir.paket\paket.targets(36,5): error MSB3073: The command
""C:\SolutionDir.paket\paket.exe" restore --references-files
"C:\SolutionDir\FooProject\paket.references" " exited with code 1.
[C:\SolutionDir\FooProject\FooProject.csproj]

When using paket restore with --log-file (BTW isn't it available on the
paket's commands list displayed when you run paket without any arguments?)
I could see the cause of the failures:

packages folder is locked by paket.exe (PID = 4220). Waiting...

Paket failed with: The process cannot access the file
'C:\SolutionDir\packages\paket.locked' because it is being used by another
process.

The workaround is to run paket restore before running msbuild.
Is it possible to make paket supports such a scenario out of the box?

—
Reply to this email directly or view it on GitHub
#858.

[Stift]

I just recommend to do a paket restore before you start msbuild and start msbuild with /p:RestorePackages=false. This works fine and fast.

[dzendras]

@forki: I was pretty sure that the extra complexity was not worth it. Just asked to confirm :)
@Stift: The solution seems reasonable.

[blumu]

Just hitting the same issue. Just out of curiosity: is it really necessary for paket to put a read access lock on this file when it just needs to restore packages?

[forki]

@blumu the lock is needed since we change the packages folder. If we don't serialize the requests then you get write conflicts there.

[blumu]

@forki Is it not an overkill to have a global lock? Would not it be more appropriate to only lock any package directory that needs to be updated? That would significantly reduce the chance of lock contention and make this issue go away completely when the package folder is up to date.

[forki]

yes in that case it would be enough to lock per nuget package.

issue go away completely when the package\ folder is up to date

that should already be no issue. since lock time would be really short and no timeout occurs.

[blumu]

Large nuget packages can take some time to download so that would not completely eliminate contentions. It would still reduce them significantly though.

[forki]

TBH I don't really think it's worth it. If you restore before build then everything should be fine.

[blumu]

Maybe. I would argue that restoring packages before building is just a workaround, not an actual fix. Paket.exe should ideally be just as bullet proof as nuget.exe.

[forki]

How is nuget bullet proof here? What are they doing to solve this?
On Mar 10, 2016 19:41, "William Blum" [email protected] wrote:

Maybe. I would argue that restoring packages before building is just a
workaround, not an actual fix. Paket.exe should ideally be just as bullet
proof as nuget.exe.

—
Reply to this email directly or view it on GitHub
#858 (comment).

[blumu]

Ok maybe I did not choose the right terminology :-) I hope I did not offend anyone here. So nuget may not be "bullet proof" but maybe slightly more robust? I'm not sure how they implement it in nuget.exe but I never had to worry about this until I tried paket. I have many large nuget dependencies in my project and nuget manages to restore packages automatically at build time without a glitch. It would be preferable if paket could handle package restore just as well. Also the current workaround is not obvious and not clearly documented and I'm guessing many new paket users hit the same issue.

Auxiliary question: how would you use the workaround when building under Visual Studio? Do I need to configure the additional switch for every single project in my solution?

[forki]

I looked at the code that acquires the lock. It waits for 25min. Is that what you see? Restore failing after 25min? So I guess something else is wrong.

I might have an idea where this race condition comes from.

[blumu]

No, my build normally takes less than 5 minutes overall (with msbuild running on 10 parallel threads).

[forki]

Ok please retry with 2.52.2 - I changed it slightly and also gave a little more verbose output.

[blumu]

Just tried 2.52.2. Paket does not fail and I don't get the msbuild warning MSB3073 anymore, however I still get a bunch of other warnings from paket itself:

error   Could not acquire lock to c:\src\packages\paket.locked.
  The process cannot access the file 'c:\src\packages\paket.locked' because it is being used by    another process.
  Could not acquire lock to c:\src\packages\paket.locked.
  Trials left: 77.

The solution is still suboptimal though, there really should not be a global lock around the package directory, especially when packages are all already up to date.