Large WinRM uploads failing

[dylanmei]

Large file uploads, such as the vmware tools, end abruptly.

Build 'vmware-windows-iso' errored: unexpected EOF

In my case, the temporary file at C:\Users\packer\AppData\Local\Temp repeatedly gets to 15.4 MB (16,211,663 bytes) when it quits.

[dylanmei]

Example output.

    2014/12/03 08:02:42 uploading to temporary file: %TEMP%\winrm-upload-547f33d4-62cb-8a82-a1df-74f31745491b.tmp
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7fff88044282

goroutine 0 [idle]:

goroutine 16 [chan receive]:
github.com/masterzen/winrm/winrm.(*Client).RunWithInput(0xc20805a780, 0xc2165c8000, 0x1f4b, 0x74f318, 0xc20803a008, 0x74f318, 0xc20803a010, 0x74fc40, 0xc20803a000, 0x0, ...)
  /Users/dylan/go/src/github.com/masterzen/winrm/winrm/client.go:120 +0x240
github.com/dylanmei/packer-communicator-winrm/communicator/winrm.(*Communicator).runCommand(0xc20808e640, 0xc2165c8000, 0x1f4b, 0xc21b0a7b60, 0x0, 0x0)
  /Users/dylan/go/src/github.com/dylanmei/packer-communicator-winrm/communicator/winrm/communicator.go:106 +0x107
github.com/dylanmei/packer-communicator-winrm/communicator/winrm.(*fileManager).runCommand(0xc2080c8f60, 0xc2165c8000, 0x1f4b, 0x0, 0x0)
  /Users/dylan/go/src/github.com/dylanmei/packer-communicator-winrm/communicator/winrm/file_manager.go:109 +0x96
github.com/dylanmei/packer-communicator-winrm/communicator/winrm.(*fileManager).Upload(0xc2080c8f60, 0x439170, 0x0, 0x74fc40, 0xc20803a0a0, 0x0, 0x0)
  /Users/dylan/go/src/github.com/dylanmei/packer-communicator-winrm/communicator/winrm/file_manager.go:52 +0x621
github.com/dylanmei/packer-communicator-winrm/communicator/winrm.(*Communicator).Upload(0xc20808e640, 0x439170, 0x0, 0x74fc40, 0xc20803a0a0, 0xc208095ca0, 0x0, 0x0)
  /Users/dylan/go/src/github.com/dylanmei/packer-communicator-winrm/communicator/winrm/communicator.go:120 +0x8e
main.(*FileCommand).Run(0xc20809ac70, 0xc20800e060, 0x1, 0x1)
  /Users/dylan/go/src/github.com/dylanmei/packer-communicator-winrm/main.go:96 +0x409
github.com/rakyll/command.Run()
  /Users/dylan/go/src/github.com/rakyll/command/command.go:155 +0xa1
main.standalone()
  /Users/dylan/go/src/github.com/dylanmei/packer-communicator-winrm/main.go:40 +0x543
main.main()
  /Users/dylan/go/src/github.com/dylanmei/packer-communicator-winrm/main.go:24 +0x62

etc...

[dylanmei]

Underneath all of that ^^

The WS-Management service cannot process the request. This user is allowed a
maximum number of 1500 concurrent operations, which has been exceeded. Close
existing operations for this user, or raise the quota for this user.

[dylanmei]

After sending the ~1496 8k chunks we hit this limit. Why "concurrent"? If there's a resource we can dispose then we should find that. Or yuck, we use multiple sessions.

@sneal do you have advice about sending large files with the packer winrm communicator or winrb/winrm?

[sneal]

@dylanmei Yeah, don't upload large files over WinRM. For VMWare tools, I place them on web server and then download them via a PowerShell script. More detail here in this blog post. I know that's not what you wanted to hear.

It looks like the Go WinRM library isn't properly closing running processes. I'm pretty sure I had this working in my winrm fork, take a look at this commit.

FYI - We're adding GZip support to the Ruby WinRM gem. Its still a WIP.

[dylanmei]

Thanks @sneal, that's good info.

Suddenly VM Tools -- and perhaps any operation requiring large files -- seem better suited as part of a provisioning step anyways. I'll dig into the bug in the WinRM package.

[mefellows]

@sneal I'll double check, I recall adding that commit into the upstream WinRM Go Library but it didn't fix it (Still have this issue lying about).

Uploading files via SOAP really just isn't the best way to go about this sort of thing. I haven't investigated, but I wonder if we could use the in-built web server API as an upload replacement? For now, I'll use a workaround (or just ignore the step altogether).

[sneal]

@mefellows Good suggestion to use the Packer web server for file uploads!

[mefellows]

Draft - seems to work nicely! One downside is that it has to create a temp file on the host first due to the fact a reader is provided not a file handle (at least, I think). I'm sure it can be much improved. But, POC complete.

https://github.com/packer-community/packer-windows-plugins/blob/feature/http-uploader/communicator/winrm/communicator.go#L146-L205

[sneal]

@mefellows Nice job on cranking that out. Not only is it simpler, but it should be 10x faster compared to transferring files over WinRM.

[mefellows]

NOTE: a suggestion from @maxlinc (hashicorp/vagrant#4932 (comment)) could be to leverage the HTTP server to allow rsync over HTTP instead of SSH, which could be useful for bidirectional sync.

[mefellows]

Update: PR #10 implements the local web server however this will not work for remote provisioning (such as to AWS) as it needs to address a local ip. My current thinking is now to implement a small BITS client in Go to be able to communicate in a more native way that doesn't involve SOAP. In fact, BITS server is just an http server anyway so essentially it's in line with the original approach.

[maxlinc]

Any idea how folder redirection works for RDP? It works for me, but I haven't had any figuring out how to use it outside of RDP, or figuring out the underlying protocol to see if it could be used alongside WinRM.

Otherwise BITS sounds like a good approach.

[mefellows]

That's an interesting suggestion @maxlinc, i'll certainly investigate this. The one principal I'm trying to stick by is to not have to download & install any additional software as part of the sync process (e.g. SSH, rsync, ftp etc.). This would seem to fit that.

[dylanmei]

I'm also looking into the concurrent operations issue, which is a problem regardless of file uploading. If that goes well we might follow along with the Ruby gem's GZip approach.

[mefellows]

I definitely like the idea of zipping the contents prior to uploading, i'll add that to the wish list. Thanks for looking at the concurrent aspect of it all.

[dylanmei]

The problems specific to this issue are addressed in 199e29f

All file-copying stuff is moved into github.com/packer-community/winrmcp. I've made sure it

1. uses less memory and
2. keeps using a winrm.Shell until it reaches the MaxConcurrentOperationsPerUser limit, at which point it creates the new winrm.Shell.

The second point makes it mildly quicker than before. A 37mb file went from 3:30 minutes to 2:30 minutes, and a 90mb file from 8:00 to 6:00.

I've only tinkered at it, but running uploads simultaneously makes the directory uploads quite fast. If we can get that stable then we should be able to break up the larger files into parallel uploads as well.

[sneal]

@dylanmei Since MaxConcurrentOperationsPerUser is configurable and the default varies between OS versions, we're planning on catching the associated error and retrying after resetting the connection in the Ruby WinRM gem.

[dylanmei]

@sneal Cool!

Winrmcp reads the setting ahead of time: https://github.com/packer-community/winrmcp/blob/master/winrmcp/cp.go#L56

I can see utilizing other settings to pro-actively warn about MaxMemoryPerShell problems and to enable running multiple shells in parallel.

[sneal]

@mwrock What do you think about reading the WinRM MaxConcurrentOperationsPerUser like @dylanmei is doing here in the Ruby gem? That seems a bit more robust.

[mwrock]

Interesting. That would be ideal. I had tried doing that in the Test-Kitchen implementation and hit a wall with access denied errors. I was unable to read the winrm info on the guest side authenticating remotely with an administrator account. I also tried using plain windows powershell remoting and concluded this was likely a blocked call for remote tokens.

@dylanmei are you using a scheduled task to call via a local context? That would work but was more than I wanted to bite off at the moment. And if you are not, then there is something I was obviously missing.

[mwrock]

Just looked at the code and there is clearly no scheduled task foo going on so I will look at this again. Way better than using exception handling for control logic.

[dylanmei]

@mwrock Well this is curious. I've run winrmcp with this technique only on Win 2012 R2 via Packer vmware and aws. I have fairly familiar WinRM settings (gist).

I may be being too cavalier. If you do find any kind of repro, please do report back here so I can create an issue for winrmcp. I'll do the same if I spot something.

[mefellows]

As an aside, an SMB share is probably worth investigating as an alternative as most hosts and guests support CIFS/SMB shares by default and is the most native Windows sharing mechanism that I can think of. As it turns out BITS looks more trouble that it's worth (involves enabling IIS and the BITS extension for uploads to work which raises some security concerns not to mention complicates setup and cleanup activities).

[mwrock]

Yeah a share could serve as an efficient and convenient transport mechanism. Unfortunately it would not be feasible universally since it assumes the host is addresable by the guest which is not likely to be the case for cloud based scenarios. I think you eluded to that above.

[mefellows]

Thanks @mwrock, could we not create the share on the remote host first and then use standard tools to authenticate and copy the files to the remote share?

[mwrock]

@dylanmei and @sneal: I was just investigating the remote invocation of winrm get winrm/config -format:xml. This does indeed work on my 2012R2/win8.1 VMs. But it fails with:

<f:WSManFault xmlns:f="http://schemas.microsoft.com/wbem/wsman/1/ws
manfault" Code="5" Machine="wrocktest2k8r2"><f:Message>Access is denied. </f:Me
ssage></f:WSManFault>

On win7 and 2k8R2 VMs which, sadly, is were one most needs this info from with its default limit of 15. I've tried several permutations of this command including the navigation of the wsman:\ powershell provider with no luck. I've done a fair amount of googling to get a more solid answer but everything addresses folks trying to enable remoting and not just checking remoting configuration. I also tried temporarily turning off the windows firewall ithout any success.

At any rate, my guess is that the security model changed between powershell 2 and 3 preventing winrm config commands from being remotely invoked on powershell v2 and earlier.