Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSH bug?? Connection refused Failed to connect to new control master . #309

Closed
nabidul opened this issue Mar 24, 2017 · 22 comments
Closed

Comments

@nabidul
Copy link

nabidul commented Mar 24, 2017

I was trying to run this from within a Docker container in Centos. I suspected that might be an issue and when I switched to using a VM it worked fine.

In short,

debug1: Authentication succeeded (publickey).
nAuthenticated to 10.3.3.16 ([10.3.3.16]:22).
debug1: setting up multiplex master socket
debug3: muxserver_listen: temporary control path /root/.ansible/cp/ansible-ssh-10.3.3.16-22-ansible.WKmgczTyEdVwJpd4
debug2: fd 4 setting O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug1: channel 0: new [/root/.ansible/cp/ansible-ssh-10.3.3.16-22-ansible]
debug3: muxserver_listen: mux listener channel 0 fd 4
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x08
debug1: control_persist_detach: backgrounding master process
debug2: control_persist_detach: background process is 850
Control socket connect(/root/.ansible/cp/ansible-ssh-10.3.3.16-22-ansible): **Connection refused
Failed to connect to new control master ", 
    "unreachable": true**
}

What could be happening? Thank you in advance.


Centos info
CentOS Linux release 7.2.1511 (Core) 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.2.1511 (Core) 
CentOS Linux release 7.2.1511 (Core) 
ANSIBLE VERSION

ansible 2.2.1.0

SSH Config
 #file: ssh_config
IdentityFile ~/.ssh/id_rsa
Inventory
[all]
node1

[all:vars]
ansible_connection=ssh
ansible_ssh_user=ansible
Ping with ansible:
node1 | UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh: Control socket connect(/root/.ansible/cp/ansible-ssh-10.3.3.16-22-ansible): Connection refused\r\nFailed to connect to new control master\r\n", 
    "unreachable": true
}
-Verbose
<p>Loading callback plugin minimal of type stdout, v2.0 from /usr/lib/python2.7/site-packages/ansible/plugins/callback/init.pyc
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/system/ping.py
&lt;10.3.3.16> ESTABLISH SSH CONNECTION FOR USER: ansible
&lt;10.3.3.16> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ansible -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r 10.3.3.16 '/bin/sh -c '"'"'( umask 77 &amp;&amp; mkdir -p "echo ~/.ansible/tmp/ansible-tmp-1490360816.2-240348019772084" &amp;&amp; echo ansible-tmp-1490360816.2-240348019772084="echo ~/.ansible/tmp/ansible-tmp-1490360816.2-240348019772084" ) &amp;&amp; sleep 0'"'"''
node1 | UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: OpenSSH<em>7.4p1, LibreSSL 2.4.4
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for 
debug1: Reading configuration data /etc/ssh/ssh</em>config
debug1: auto-mux: Trying existing master
debug1: Control socket "/root/.ansible/cp/ansible-ssh-10.3.3.16-22-ansible" does not exist
debug2: resolving "10.3.3.16" port 22
debug2: ssh<em>connect</em>direct: needpriv 0
debug1: Connecting to 10.3.3.16 [10.3.3.16] port 22.
debug2: fd 3 setting O<em>NONBLOCK
debug1: fd 3 clearing O</em>NONBLOCK
debug1: Connection established.
debug3: timeout: 10000 ms remain after connect
debug1: permanently<em>set</em>uid: 0/0
debug1: identity file /root/.ssh/id<em>rsa type 1
debug1: key</em>load<em>public: No such file or directory
debug1: identity file /root/.ssh/id</em>rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH<em>7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH</em>7.3
debug1: match: OpenSSH<em>7.3 pat OpenSSH compat 0x04000000
debug2: fd 3 setting O</em>NONBLOCK
debug1: Authenticating to 10.3.3.16:22 as 'ansible'
debug3: hostkeys<em>foreach: reading file "/root/.ssh/known</em>hosts"
debug3: record<em>hostkey: found key type ECDSA in file /root/.ssh/known</em>hosts:1
debug3: load<em>hostkeys: loaded 1 keys from 10.3.3.16
debug3: order</em>hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2<em>MSG</em>KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2<em>MSG</em>KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected],aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: [email protected],zlib,none
debug2: compression stoc: [email protected],zlib,none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first<em>kex</em>follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first<em>kex</em>follows 0 
debug2: reserved 0 
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: compression: none
debug1: kex: client->server cipher: [email protected] MAC: compression: none
debug3: send packet: type 30
debug1: expecting SSH2<em>MSG</em>KEX<em>ECDH</em>REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:Dha6CJyI4id6qAaSyM0AhG+8ZG6U6yHpm5xf4JAxkms
debug3: hostkeys<em>foreach: reading file "/root/.ssh/known</em>hosts"
debug3: record<em>hostkey: found key type ECDSA in file /root/.ssh/known</em>hosts:1
debug3: load<em>hostkeys: loaded 1 keys from 10.3.3.16
debug1: Host '10.3.3.16' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known</em>hosts:1
debug3: send packet: type 21
debug2: set<em>newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2</em>MSG<em>NEWKEYS sent
debug1: expecting SSH2</em>MSG<em>NEWKEYS
debug3: receive packet: type 21
debug1: SSH2</em>MSG<em>NEWKEYS received
debug2: set</em>newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /root/.ssh/id<em>rsa (0x7f23694372a0)
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2</em>MSG<em>EXT</em>INFO received
debug1: kex<em>input</em>ext<em>info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service</em>accept: ssh-userauth
debug1: SSH2<em>MSG</em>SERVICE<em>ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey
debug3: authmethod</em>lookup publickey
debug3: remaining preferred: ,gssapi-keyex,hostbased,publickey
debug3: authmethod<em>is</em>enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id<em>rsa
debug3: send</em>pubkey<em>test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug2: input</em>userauth<em>pk</em>ok: fp SHA256:uqPlK4SsqV/skWF1FSJ9VVcZBNQH78bFMd61zO0YqBs
debug3: sign<em>and</em>send<em>pubkey: RSA SHA256:uqPlK4SsqV/skWF1FSJ9VVcZBNQH78bFMd61zO0YqBs
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to 10.3.3.16 ([10.3.3.16]:22).
debug1: setting up multiplex master socket
debug3: muxserver</em>listen: temporary control path /root/.ansible/cp/ansible-ssh-10.3.3.16-22-ansible.WKmgczTyEdVwJpd4
debug2: fd 4 setting O<em>NONBLOCK
debug3: fd 4 is O</em>NONBLOCK
debug3: fd 4 is O<em>NONBLOCK
debug1: channel 0: new [/root/.ansible/cp/ansible-ssh-10.3.3.16-22-ansible]
debug3: muxserver</em>listen: mux listener channel 0 fd 4
debug2: fd 3 setting TCP<em>NODELAY
debug3: ssh</em>packet<em>set</em>tos: set IP<em>TOS 0x08
debug1: control</em>persist<em>detach: backgrounding master process
debug2: control</em>persist_detach: background process is 850
Control socket connect(/root/.ansible/cp/ansible-ssh-10.3.3.16-22-ansible): Connection refused
Failed to connect to new control master
",
"unreachable": true
}</p>
@strangeman
Copy link
Contributor

Please, can you format your code with Markdown (https://guides.github.com/features/mastering-markdown/)? It hard to read.
Also, you can paste big logfiles to the pastebin services.

@fxfitz
Copy link

fxfitz commented Aug 10, 2017

I'm seeing this problem as well. Is anyone else having this problem?

@u2bo
Copy link

u2bo commented Aug 11, 2017

i get the same error! ansible version is ansible 2.2.1.0 how resolve it? openssh-client was installed

@fxfitz
Copy link

fxfitz commented Aug 11, 2017

I was able to resolve this problem by upgrading all packages, specifically the kernel, and rebooting. Hope that helps.

@Echobob
Copy link
Contributor

Echobob commented Aug 11, 2017

For those still affected, can you paste your ansible.cfg settings for the following sections:

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s

[accelerate]

@u2bo
Copy link

u2bo commented Aug 11, 2017

@fxfitz @Echobob thanks friends! my ansible was packed in docker .i update the centos host kernel from 3.10.237 to 3.10.0-514 ,resolve docker warning and reboot it then works.

@fxfitz
Copy link

fxfitz commented Aug 11, 2017

@Echobob Just as an FYI, I don't have anything set in those sections.

@slikk66
Copy link

slikk66 commented Oct 24, 2018

Just ran into this today, basically it seems if you run ansible inside a docker container you're going to have a bad time. I was using packer 1.3.1 and ansible 2.7.0 on Ubuntu 18.04 container on a mac osx host and it didn't work. My working ansible config now has:

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s
control_path = /dev/shm/cp%%h-%%p-%%r

Could not have been more difficult to debug.

@AndrewFarley
Copy link

@slikk66 Thank you so much :) . Saves me from tons more debugging time! Would give 100 thumbs up if I could. :P

@SansGuidon
Copy link

SansGuidon commented Nov 14, 2018

clearing ssh_args value did the trick for me.
at the opposite, trying the suggested -C -o ControlMaster=auto -o ControlPersist=600s or anything similar was causing me issues

@ayesha54
Copy link

@slikk66 Thank you so much :)
It worked

@alfchee
Copy link

alfchee commented Jun 29, 2020

Just ran into this today, basically it seems if you run ansible inside a docker container you're going to have a bad time. I was using packer 1.3.1 and ansible 2.7.0 on Ubuntu 18.04 container on a mac osx host and it didn't work. My working ansible config now has:

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s
control_path = /dev/shm/cp%%h-%%p-%%r

Could not have been more difficult to debug.

Thanks, this worked for me, I'm running Ansible Playbook from WSL with Ubuntu 18.04, and this error was giving me problems.

@spurin
Copy link

spurin commented Jul 5, 2020

@alfchee I've also just encountered this issue. An alternative way you can fix this whilst running in a container is by editing config/base.yml in your Ansible installation path and changing the default path for the control path dir, i.e.

ANSIBLE_SSH_CONTROL_PATH_DIR:
  # TODO: move to ssh plugin
  default: ~/.ansible/cp

Change to

ANSIBLE_SSH_CONTROL_PATH_DIR:
  # TODO: move to ssh plugin
  default: /dev/shm

Standard disclaimers apply, some might prefer the config driven approach and YMMV. For me, I'm using Ansible in a container for experimentation/dev/learning purposes only, so, I prefer the change once, and have it the standard without ansible.cfg overrides.

@emranbm
Copy link

emranbm commented May 24, 2021

Just ran into this today, basically it seems if you run ansible inside a docker container you're going to have a bad time. I was using packer 1.3.1 and ansible 2.7.0 on Ubuntu 18.04 container on a mac osx host and it didn't work. My working ansible config now has:

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s
control_path = /dev/shm/cp%%h-%%p-%%r

Could not have been more difficult to debug.

Thanks a lot. Same problem with ansible 2.9 docker image on an ubunut 16.04 host. And it fixed.

@VigneshAjay98
Copy link

VigneshAjay98 commented May 4, 2022

Screenshot 2022-05-04 143722

I faced the above issue in jenkins docker container. So i did this as mentioned by @alfchee and its solved.
Screenshot 2022-05-04 144153

Now its working fine!!!!!!!
image

@spurin
Copy link

spurin commented May 4, 2022

Hi @VigneshAjay98

Whilst this will fix it, it will involve manipulating the Ansible configuration file related to your setup each time (if you’re using specific Ansible configuration files for different playbooks/areas).

In my course, Dive Into Ansible, I provide container based images for Ansible and the approach I take instead is to update Ansible in the container image so it’s using the correct path internally. Therefore allowing Ansible to run as you’d expect.

I’ve got 2 commands that you can run which will fix this, without then needing to patch files.

See the following, lines 25 - 30. If you paste those it will resolve the issue in the container image -

https://github.com/spurin/diveintoansible-images/blob/ansible/Dockerfile

@harahauk
Copy link

I'm working in WSL1 on AlmaLinux on a Insider Beta Build of Win 11 (22H2, 22623.746) and I suddenly started getting these errors. @slikk66 's posted solution did the trick and saved me from madness

@ijpatricio
Copy link

Thank you @slikk66

@tabish-javed
Copy link

tabish-javed commented Oct 28, 2023

@alfchee I've also just encountered this issue. An alternative way you can fix this whilst running in a container is by editing config/base.yml in your Ansible installation path and changing the default path for the control path dir, i.e.

ANSIBLE_SSH_CONTROL_PATH_DIR:
  # TODO: move to ssh plugin
  default: ~/.ansible/cp

Change to

ANSIBLE_SSH_CONTROL_PATH_DIR:
  # TODO: move to ssh plugin
  default: /dev/shm

Standard disclaimers apply, some might prefer the config driven approach and YMMV. For me, I'm using Ansible in a container for experimentation/dev/learning purposes only, so, I prefer the change once, and have it the standard without ansible.cfg overrides.

Path to base.yml;
/usr/local/lib/python3.10/dist-packages/ansible/config/base.yml

@spurin - I came here following your Ansible Course on Udemy. Please update there as well.
By the way the section ANSIBLE_SSH* doesn't exists...

@tabish-javed
Copy link

Hi @VigneshAjay98

Whilst this will fix it, it will involve manipulating the Ansible configuration file related to your setup each time (if you’re using specific Ansible configuration files for different playbooks/areas).

In my course, Dive Into Ansible, I provide container based images for Ansible and the approach I take instead is to update Ansible in the container image so it’s using the correct path internally. Therefore allowing Ansible to run as you’d expect.

I’ve got 2 commands that you can run which will fix this, without then needing to patch files.

See the following, lines 25 - 30. If you paste those it will resolve the issue in the container image -

https://github.com/spurin/diveintoansible-images/blob/ansible/Dockerfile

Thanks @spurin ! I executed those two lines inside the container, and this problem now resolved. So, updating ansible in container to latest version isn't a good idea ?

@spurin
Copy link

spurin commented Oct 28, 2023

Hi @tabish-javed, I have this change as part of the Dockerfile for the Ansible lab image so essentially, I install and patch. As you've seen, sadly if you update the version of ansible whilst in the running container, you'll lose this patch.

This thread has some ways of overriding this at an ansible.cfg level but personally i'm not a fan, this patch allows Ansible to run as expected whilst in a container and is a more real-life outcome 👍

@spurin
Copy link

spurin commented Oct 28, 2023

And also @tabish-javed - the location of that change has changed as Ansible has refactored and moved libraries around, hence the two variations, one for older versions and the other for current versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests