Packer stuck at: "Gracefully halting virtual machine" after Windows sysprep

[kagarlickij]

Hi there, my environment is:
Host platform - macOS 10.12.1
Packer version - 0.11.0 (the same with packer 0.10.2)
Vagrant version - 1.8.6
VMware Fusion Pro version - 8.5.1
VirtualBox version - 5.1.18
Parallels version - 12.0.2

vagrant plugin list
vagrant-aws (0.7.2)
vagrant-parallels (1.6.3)
vagrant-share (1.1.5, system)
vagrant-vmware-fusion (4.0.14)

Repo with all necessary code to reproduce bug - https://github.com/kagarlickijd/packer-demo

If I use "shutdown_command": "shutdown /s /t 10 /f /d p:4:1" it works correctly for VirtualBox, VMware and Parallels.

But I need to perform sysprep instead of just shutdown, so I use shutdown-packer.bat script in shutdown_command section.

It works correctly with VirtualBox, BUT both Parallels and VMware builders are stuck at "Gracefully halting virtual machine" step.

VM is powered down correctly with shutdown-packer.bat script, so it seems like Packer doesn't receive "messages" from Parallels and VMware.

How can this one can be resolved?

Thanks in advance!

[rickard-von-essen]

Please supply the information requested in the issue template:

• Debug log output from PACKER_LOG=1 packer build template.json.
  Please paste this in a gist.
• The simplest example template and scripts needed to reproduce the bug.
  Include these in your gist.

[kagarlickij]

Sure, example gists were added to the initial post, here's log output: https://gist.github.com/kagarlickijd/672e2bd901c7860678f941390c5b01e0

Please be informed that errors like
"vmware-iso : The term 'vmware-iso' is not recognized as the name of a cmdlet, function, script file, or operable
2016/11/08 22:32:38 ui: vmware-iso: program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again."
... are related to another packer 0.11.0 bug - #4044 and they don't appear in packer 0.10.2 as well as don't affect this issue with sysprep in vmware

If you want, I can do the same for Parallels provider, but it's really the same with VMware

[rickard-von-essen]

Could you provide the templet for VirtualBox that works?

[kagarlickij]

@rickard-von-essen sure, here's repo with all necessary code - https://github.com/kagarlickijd/packer-demo

Branch "win-sysprep" uses sysprep which works only with VirtualBox, but not with Parallels and VMware
Branch "win-shutdown" uses shutdown which works with VirtualBox, Parallels and VMware

[rickard-von-essen]

My Windows foo is extremely low but this is what I found so far.

1. Disabling WinRM in the firewall seems to hang everything at that:

==> parallels-iso: Gracefully halting virtual machine...
    parallels-iso:
    parallels-iso: C:\Users\vagrant>echo "Disabling WinRM in Firewall (it will be re-enabled during first boot after sysprep (SetupComplete.cmd will be used, it's created with postSetup.ps1))..."
    parallels-iso: "Disabling WinRM in Firewall (it will be re-enabled during first boot after sysprep (SetupComplete.cmd will be used, it's created with postSetup.ps1))..."
    parallels-iso:
    parallels-iso: C:\Users\vagrant>netsh advfirewall firewall set rule group="Windows Remote Management" new enable=No

1. Keeping WinRM allowed in the FW doesn't change anything besides that I get info that sysprep is running and then it hangs.
2. I have experimented with different combinations of /quit and /shutdown to sysprep.
3. Line 46 which seems to indicate that the WinRM RemoteCmd never returns. But the code for VirtualBox looks the same so I can't explain why it works there.

I'll try and add some extra debugging of the WinRM comm.

[DanHam]

I've been experiencing much the same issue with my Windows builds. After alot of messing around and experimentation I've come to much the same conclusions as Rickard.

I was wondering if the sysprep command is doing something with the WinRM firewall settings or with WinRM itself. This could cause the same apparent behaviour as when we leave in the command to disable the WinRM firewall rules and then run sysprep; We're just seeing the root issue manifest itself one step further down the line...

I've not been able to look at the issue with the Parallels builder as I don't have Parallels. However,
following Rickards pointer above I took a look at the VMware builders shutdown step. Commenting out lines 63, 65 and 66 of builder/vmware/common/step_shutdown.go seems to allow the build to complete.

            ui.Error(err.Error())
            return multistep.ActionHalt
        }

        // Wait for the command to run so we can print std{err,out}
        // We don't care if the command errored, since we'll notice
        // if the vm didn't shut down.
        // cmd.Wait()

        // log.Printf("Shutdown stdout: %s", stdout.String())
        // log.Printf("Shutdown stderr: %s", stderr.String())

        // Wait for the machine to actually shut down
        log.Printf("Waiting max %s for shutdown to complete", s.Timeout)
        shutdownTimer := time.After(s.Timeout)

In other words, not doing cmd.Wait() allows the build to finish. While Rickard may be lacking some Windows foo, I am most definitely lacking Go foo, so that's about as far as I was able to take my debug effort!! Hopefully that will help somehow...

It would be really nice to get a fix in for the v1.0 release!! Many thanks for all your efforts!

Dan

Packer version: 0.11.1.dev
Host platform: Mac OS X 10.11.6
Virtualbox: 5.0.26
VMware Fusion: Professional Version 8.5.1 (4543325)

@kagarlickijd I had to comment out the steps to restart the WinRM service in your enable-WinRM.ps1 script. The restart causes the build to fail for me as Packer has 'Connected to WinRM!' and is uploading the VMware Tools ISO when WinRM is restarted by your script.

[rickard-von-essen]

cmd.Wait() waits for the shutdown_command to "return". It's hints about a solution by not waiting for the command to return, but just issue it and the wait for the VM to shutdown.

The strange thing is that it works for VirtualBox which have the same code as Parallels, where it doesn't work.

[karjona]

Hello,
I would like to report that the exact same problem referenced here also occurs when using the hyperv-iso builder available in this pull request: #2576. The workaround suggested above by @DanHam also works for the Hyper-V builder: commenting the cmd.Wait() line of the hyperv-iso builder lets the packer build finish successfully.

I've only had this problem while trying to build Windows Server 2016 machines and using the Sysprep command as the shutdown_command. Windows Server 2012 R2 builds with the same Sysprep command seem to work just fine.

I would think this indicates a problem with the new sysprep.exe command available in Windows Server 2016 or the way that Windows Server 2016 returns the exit code or the command output. My guess is that this is breaking the code that waits for a response of the shutdown command in some builders, but I have no idea what that change might be in sysprep.exe, outputs or return code.

I hope this adds additional information that can help you guys diagnose the problem.

[taliesins]

https://github.com/taliesins/packer-baseboxes/blob/master/hyperv-windows-2016-serverstandard-amd64.json

Have a look at my insane shutdown approach. Don't forget to look at cmd file that calls powershell.

It appears to work between different Windows versions.

[DanHam]

@mwhooker I noticed you have previously looked at getting rid of cmd.Wait() in #4041 here but then reintroduced it in the same PR on the next commit here. I've been running Packer with cmd.Wait() commented out for a while and haven't suffered any ill effects.

Have a look at my insane shutdown approach.

@taliesins I've managed to avoid this so far :). My thinking is as follows:

• If no other builder waits for the shutdown command, could we not also take the same approach for the VMware and Hyper-v builders and solve the issue that way?
• Using the shutdown command to run sysprep directly is fairly common.
• Removing the problematic cmd.Wait() would allow a consistant approach across all builders - note that the same template works fine for me with the Virtualbox-iso builder

I'm also wondering if #4191 might be related in some way.

Could we get a fix in for this for 0.12.2? Since I'm affected by this, my feeling is that this bug is deserving of a milestone tag!! ;)

[DanHam]

@rickard-von-essen Great! TY!

[DanHam]

👍 Works for me! VMware builder now completing sucessfully.

[R0quef0rt]

I've spent so long working on this issue. I didn't want to custom-build Packer with cmd.wait() commented-out, so I had to find another solution. Here's what's working for me (in Hyper-V and a Server2016 guest):

For my shutdown_command, use this:
"shutdown_command": "C:\\Windows\\system32\\sysprep\\sysprep.exe /unattend:A:\\Autounattend.xml /quiet /generalize /oobe /shutdown",

In the autounattend.xml file, it's VERY IMPORTANT that you use this:

    <settings pass="generalize">
      <component name="Microsoft-Windows-Security-SPP" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <SkipRearm>1</SkipRearm>
      </component>
      <component name="Microsoft-Windows-PnpSysprep" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <PersistAllDeviceInstalls>true</PersistAllDeviceInstalls>
        <DoNotCleanUpNonPresentDevices>true</DoNotCleanUpNonPresentDevices>
      </component>
    </settings>

Sysprep, by default, will remove all device drivers from the system. This completely breaks WinRM functionality when using Packer, and it doesn't allow the sysprep command to return. This setting alone fixed my problem after days/weeks(maybe months?) of troubleshooting.

I hope that helps someone.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.