(BKR-71) Adding support to use WinRM for windows platform

[fgouteroux]

Dear all,

I'm a long beaker user and I have to test puppet deployment on Windows.
But beaker only support SSH with Cygwin and in many companies, it's not allowed.

With my small hands, I've wrote WinRM connection, the Window's world alternative to SSHD that allows you to remotely login securely and execute commands on Windows machines.

With the gem winrm we can execute command on the windows host and with the gem winrm-fs we can copy files.

I've just take the same names as SSH like scp_to beacause the host don't know about the OS (we can do it better).

I try to write some tests in ruby but when I dit it, it's like "I don't know what i'm doing"!

If this enhancement sounds good for the community and if somebody can help me to write tests, I will take time to make it perfect :)

[puppetlabs-jenkins]

Can one of the admins verify this patch?

[liamjbennett]

+1 for the effort here. I can test this out on one of my projects and let you know how I get on.

I can also assist in writing tests or backfilling them if required.

[fgouteroux]

To test it, you have to select an authentication type like the following;

• kerberos
• plaintext
• ssl

Kerberos

To use kerberos, you have to:
1/ install krb5-libs and krb5-workstation package
2/ edit the kerberos configuration:

Example /etc/krb5.conf:

[libdefaults]
    default_realm = ATHENA.MIT.EDU
    default_tkt_enctypes = des3-hmac-sha1 des-cbc-crc
    default_tgs_enctypes = des3-hmac-sha1 des-cbc-crc
    dns_lookup_kdc = true
    dns_lookup_realm = false

[realms]
    ATHENA.MIT.EDU = {
     kdc = kerberos.mit.edu
     kdc = kerberos-1.mit.edu
     kdc = kerberos-2.mit.edu:750
     admin_server = kerberos.mit.edu
     master_kdc = kerberos.mit.edu
     default_domain = mit.edu
    }
    EXAMPLE.COM = {
     kdc = kerberos.example.com
     kdc = kerberos-1.example.com
     admin_server = kerberos.example.com
    }

[domain_realm]
    .mit.edu = ATHENA.MIT.EDU
    mit.edu = ATHENA.MIT.EDU

[logging]
    kdc = SYSLOG:INFO
    admin_server = FILE=/var/kadm5.log

3/ ask a TGT (Ticket-Granting Ticket):

kinit [email protected]

4/ Puts winrm option in the options file

:winrm => {
          :auth  => 'kerberos',
          :realm => 'DOMAIN.COM'
}

5/ Run beaker with the option file : beaker --options-file additional_options.rb

Plaintext

To use plaintext, you have to set the user and password in winrm_opts:

:winrm => {
          :auth => 'plaintext',
          :user => 'myuser',
          :pass => 'mypassword'
}

Ssl

To use ssl, you have to set the user and password in winrm_opts:

:winrm => {
          :auth          => 'ssl',
          :user          => 'myuser',
          :pass          => 'mypassword'
          :ca_trust_path => '/etc/ssl/certs/cert.pem'
}

[anodelman]

@liamjbennett thanks for checking this out! I'll take a look locally after you do the first pass.

[fgouteroux]

@liamjbennett did you find the time to test it ?

[anodelman]

@fgouteroux I'm going to be looking this over this week.

[anodelman]

I'm running the beaker acceptance tests against your code. My findings so far.

Issue 1:
Check port acceptance test doesn't pass, because check port assumes that port 22 is open - which isn't the case for winrm. I updated the test locally to check port 5985 and it passed.

Issue 2:

<WinRM::WinRMWSManFault: [WSMAN ERROR CODE: 2150859173]: <f:WSManFault Code='2150859173' Machine='w2k8r2' xmlns:f='http://schemas.microsoft.com/wbem/wsman/1/wsmanfault'><f:Message>The WS-Management service cannot process the request. This user is allowed a maximum number of 5 concurrent shells, which has been exceeded. Close existing shells or raise the quota for this user. </f:Message></f:WSManFault>>

You get that error if you run more than five commands through, I fixed it locally by putting a 'close' in the 'execute' method and also setting @sid = nil.

Issue 3:
copy_module_to failing. No idea why this is happening yet - doesn't appear to be moving anything to the SUT at all. Will continue diagnoses tomorrow.

[puppetlabs-jenkins]

Can one of the admins verify this patch?

[Iristyle]

On issue 3, @sneal or @mwrock might be of assistance.

The actual code that handles uploads is at https://github.com/WinRb/winrm-fs/blob/master/lib/winrm-fs/core/upload_orchestrator.rb#L32-L53

[anodelman]

I can copy a single file to '.' on a host, I can also copy the contents of a directory to '.' on the host.

I cannot copy a single file to a new directory on a host, nor can I copy a directory to a new directory on a host.

[anodelman]

If we can't get recursive copy of a directory to a directory on a windows host then this support will not be useful to us and we'll have to look further into ssh support.

[liamjbennett]

It's been a while since I have done anything with winrm but if I remember correctly you need to create the file/directory first with New-Item -Force before you overwrite it with Copy-Item -Force.

[sneal]

@anodelman If there are issues with the alpha quality winrm-fs gem I highly encourage anyone to log issues on the project. PRs would be even better.

[Iristyle]

tl; dr - use winrm-transport

@anodelman we can chat about this for a few minutes tomorrow if you'd like, but here are my findings.

I was able to use both winrm-fs and winrm-transport gems successfully to transfer a directory recursively to a Windows host, when writing to the users TEMP directory.

However, winrm-fs started to misbehave when trying to copy to a new directory off of c:\. winrm-transport on the other hand properly recursively created the necessary directory structure and had no problems copying the files over.

winrm-transport

[26] pry(main)> transporter.upload('/Users/Iristyle/source/puppetlabs-sqlserver', "C:/foo/bar")
=> {"1dd2a6e2495c15e29b59f6602878a0c9"=>
  {"src"=>"/Users/Iristyle/source/puppetlabs-sqlserver",
   "src_zip"=>
    "/var/folders/1b/rzv96bhd5h9bwtdnz7g0xmlm0000gn/T/tmpzip-20150518-5068-qen6qo.zip",
   "tmpzip"=>
    "C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\tmpzip-1dd2a6e2495c15e29b59f6602878a0c9.zip",
   "dst"=>"C:\\foo\\bar",
   "size"=>270304,
   "chk_exists"=>"False",
   "src_md5"=>"1dd2a6e2495c15e29b59f6602878a0c9",
   "dst_md5"=>"1dd2a6e2495c15e29b59f6602878a0c9",
   "chk_dirty"=>"True",
   "verifies"=>"True",
   "tmpfile"=>
    "C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\b64-1dd2a6e2495c15e29b59f6602878a0c9.txt",
   "chunks"=>46,
   "xfered"=>360404}}

Verified on the target machine that the new c:\foo\bar directory hierarchy was created and everything copied just fine to the puppetlabs-sqlserver directory inside.

winrm-fs

[27] pry(main)> file_manager.upload('/Users/Iristyle/source/puppetlabs-sqlserver', "C:/bar/foo")
=> 0

Directory structure c:\bar\foo not created on host, and no files copied.

Even if I get a little tricky and use winrm to execute a command to create the directory first, it still fails to copy anything.

[34] pry(main)> x.cmd("md c:\\bar\\foo")
=> {:data=>[], :exitcode=>0}
[35] pry(main)> file_manager.upload('/Users/Iristyle/source/puppetlabs-sqlserver', "C:\\bar\\foo")
=> 0

I suppose one workaround if winrm-fs cannot be swapped out, would be to write everything to a random TEMP directory, then execute a command against the WinRM service instance to move the directory.

Otherwise, I recommend just switching the gems.

@sneal - Is there a reason you haven't joined forces with the winrm-transport folks? Looks like you have similar goals, and it would be awesome to see both teams working to make the most robust gem possible in this space.

[mwrock]

FWIW: I have a little background on how the file copy work here diverged in these two gems. The bulk of the work I had initially contributed to Test-Kitchen in an effort to get files copied to windows test instances in a reasonable amount of time. It seemed like this work had broader value beyond Test-Kitchen 🍴 and so I ported it, via some PRs to the winrm gem 💎 with plans to eventually call to winrm from Test-kitchen.

Around December of last year, testing and development of windows support in Test-kitchen heated up 🔥 and it was more efficient to address bugs and perf concerns directly from Test-Kitchen. @fnichol did alot of this great work partly based on some improvements @sneal had made to my initial contibutions that @sneal ported to winrm-fs. It became clear that Test-Kitchen was not the ideal home for this since it required all test-kitchen users (mostly linux users) to pull down dependencies applicable only to windows so the winrm-transport was born 👶 .

I think there were concerns that this implementation, while optimized for one-time bootstrapping scenarios, may not be ideal for many file copying scenarios. I think many would argue that winrm itself is not ideally suited to any file copy activity except for those who are sick in spirit. 🙏 I also think @fnichol wanted to release Test-Kitchen with windows support within a matter of days of porting to winrm-transport. So a merge of these two works was likely premature.

I definitely think this could be merged at some point soon as things are proving to be more baked 🍞 . I had hoped to be able to contribute more to this effort but just have not had time lately.

This all said, if I did have more time, I would be more inclined to investigate implementing windows file copy via the powershell remoting protocol which is an extension to winrm but supports true streaming and eliminates the 8K buffer limitation that will forever criple pure winrm as a viable file copy mechanism. Unfortunately there is only one implementation right now (powershell on windows) and the spec is not at all clear so some reverse engineering is needed or a partnership with a MS team with better access to the current implementation.

[Iristyle]

Bummer - just noticed that winrm-transport doesn't have download support. I have a branch up that uses both gems - winrm-transport for uploads and winrm-fs for downloads /cc @fnichol

See the latest commit Iristyle@b2cb98d on my branch:
https://github.com/Iristyle/beaker/commits/pr/805

[sneal]

@Iristyle I released winrm-fs 0.2.0 which includes some prior unreleased fixes, some brand new fixes, and additional tests. The use case you mentioned above now works. WRT to directory uploads you should be aware of issue 9

[Iristyle]

@sneal thanks for the update. It appears that that version of the gem has fixed the upload issue that I cited.

However, I've hit the next issue testing the upload of a 54MB file. It seems like the error I received could be a bug / improvement to be made - but I'm not sure if you have hard upper limits in mind for file sizes at the moment? I filed issue WinRb/winrm-fs#15 in winrm-fs as well for this problem (though I haven't dug into the code to see if this is a winrm or winrm-fs issue):

WinRM::WinRMWMIError: [WMI ERROR CODE: 2150859174]:

<p:MSFT_WmiError b:IsCIM_Error='true' xmlns:b='http://schemas.dmtf.org/wbem/wsman/1/cimbinding.xsd' xmlns:cim='http://schemas.dmtf.org/wbem/wscim/1/common' xmlns:p='http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/MSFT_WmiError' xsi:type='p:MSFT_WmiError_Type' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'>
    <p:CIMStatusCode xsi:type='cim:cimUnsignedInt'>27</p:CIMStatusCode>
    <p:CIMStatusCodeDescription xsi:nil='true' xsi:type='cim:cimString'/>
    <p:ErrorSource xsi:nil='true' xsi:type='cim:cimString'/>
    <p:ErrorSourceFormat xsi:type='cim:cimUnsignedShort'>0</p:ErrorSourceFormat>
    <p:ErrorType xsi:type='cim:cimUnsignedShort'>0</p:ErrorType>
    <p:Message xsi:type='cim:cimString'>The WS-Management service cannot process the request. This user is allowed a maximum number of 1500 concurrent operations, which has been exceeded. Close existing operations for this user, or raise the quota for this user. </p:Message>
    <p:MessageID xsi:type='cim:cimString'>HRESULT 0x803381a6</p:MessageID>
    <p:OtherErrorSourceFormat xsi:nil='true' xsi:type='cim:cimString'/>
    <p:OtherErrorType xsi:nil='true' xsi:type='cim:cimString'/>
    <p:OwningEntity xsi:nil='true' xsi:type='cim:cimString'/>
    <p:PerceivedSeverity xsi:type='cim:cimUnsignedShort'>0</p:PerceivedSeverity>
    <p:ProbableCause xsi:type='cim:cimUnsignedShort'>0</p:ProbableCause>
    <p:ProbableCauseDescription xsi:nil='true' xsi:type='cim:cimString'/>
    <p:error_Category xsi:type='cim:cimUnsignedInt'>30</p:error_Category>
    <p:error_Code xsi:type='cim:cimUnsignedInt'>2150859174</p:error_Code>
    <p:error_Type xsi:type='cim:cimString'>HRESULT</p:error_Type>
    <p:error_WindowsErrorMessage xsi:type='cim:cimString'>The WS-Management service cannot process the request. This user is allowed a maximum number of 1500 concurrent operations, which has been exceeded. Close existing operations for this user, or raise the quota for this user. </p:error_WindowsErrorMessage>
</p:MSFT_WmiError>

from /Users/Iristyle/source/beaker/.bundle/gems/ruby/2.0.0/gems/winrm-1.3.3/lib/winrm/http/response_handler.rb:75:in `raise_if_wmi_error'

[Iristyle]

Ah, looks like you might already be aware of this particular problem @sneal based on this thread packer-community/packer-windows-plugins#3

[anodelman]

To wrap up:

• this patch needs to be updated to handle executing more than 5 windows commands.
• winrm-fs does not currently correctly handle recursive directory copy, and will need to be further evaluated.

[sneal]

@anodelman recursive directory copy does work in winrm-fs >= 0.2.0

[Iristyle]

Agreed with @sneal that the recursive directory copy worked just fine in >= 0.2.0.

The blocker issue here for us is the exhausted 1500 concurrent operations and what the best strategy for addressing that is.

[anodelman]

Sorry for the confusion here. So there is still a pending issue with winrm-fs to be resolved before this can be considered mergeable.

[anodelman]

These local_opts are not used in this method.

[anodelman]

@Iristyle I think that we should close this out and replace it with whatever work you get going - @fgouteroux this isn't mergeable in it's current state.

[fgouteroux]

@anodelman this isn't mergeable due to the winrm-fs limitation ?

The workaround "increasing your operations per user quota" isn't a solution ? If we increase the maxConcurrentOperationsPerUser just before upload/download a file ?

@Iristyle @sneal what about BITS (Background Intelligent Transfert Service) with Windows Powershell ? I use it to download big file in puppet manifests with one command: 'Import-Module BitsTransfert; Start-BitsTransfert -source "src_file" -destination "dest_file"'
Maybe we can add an option in the winrm-fs module ?

[anodelman]

In it's current state the patch does not successfully pass beaker acceptance tests for the host layer - so you cannot successfully copy information to/from hosts.

@Iristyle is working, using your patch as a base, to get this working.

Thanks for your effort here in getting the initial work figured out! We'll re-open once we have something that passes tests and ensure that you get credit in the comments.