keyfile query parameter doesn't work in connection string when ~/.ssh/config is empty

[tlhakhan]

System Information

Linux distribution

Debian GNU/Linux 12

Terraform version

$ terraform -v
Terraform v1.9.6
on linux_arm64
+ provider registry.terraform.io/dmacvicar/libvirt v0.8.0

Provider and libvirt versions

$ terraform providers 

Providers required by configuration:
.
└── provider[registry.terraform.io/dmacvicar/libvirt] 0.8.0

---

Description of Issue/Question

I observed the keyfile query param is ignored, and only the ~/.ssh/id_rsa is added to the list of key files, when ~/.ssh/config is an empty file.

I expect the keyfile provided in the query param to be included in the list of key files, even when ~/.ssh/config file is just an empty file.

Problem Setup

The system setup:

• The ~/.ssh/config is an empty file. The file must exist.
• Create an SSH private key ansible.key in the same folder as the Terraform workspace.
• A remote host running libvirtd.

The main.tf file:

terraform {
  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
      version = "0.8.0"
    }
  }
  backend "local" {
    path = "local.tfstate"
  }
}

provider "libvirt" {
  uri = "qemu+ssh://[email protected]/system?keyfile=ansible.key&sshauth=privkey&no_verify=1"
}

data "libvirt_node_info" "node" {}

Steps to Reproduce Issue

Run terraform plan with the above system setup.

$ tf plan

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: failed to connect: failed to connect to remote host 'vhost-1.lan': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
│ 
│   with provider["registry.terraform.io/dmacvicar/libvirt"],
│   on main.tf line 13, in provider "libvirt":
│   13: provider "libvirt" {
│ 
╵

Relevant logs

• Ran the following TF_LOG=debug tf plan
• Extracted the useful log lines below

2024-09-29T11:53:10.625Z [DEBUG] provider.terraform-provider-libvirt_v0.8.0: plugin address: address=/tmp/plugin103822099 network=unix timestamp=2024-09-29T11:53:10.625Z
2024-09-29T11:53:10.636Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [DEBUG] Configuring provider for 'qemu+ssh://[email protected]/system?keyfile=ansible.key&sshauth=privkey&no_verify=1': &{map[uri:0x40002a6dc0] 0x40000a8960 <nil> 0x40000c3180 map[] <nil> {{<nil>} <nil>} 0x400009c400 0x400000c5b8 0x40000fa680 false {{{} 1} {0 0}} false false}: timestamp=2024-09-29T11:53:10.636Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [INFO] establishing ssh connection to 'vhost-1.lan': timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [DEBUG] auth methods for vhost-1.lan: privkey: timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [DEBUG] found no ssh keys, using default keypath: timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [DEBUG] ssh identity files for host 'vhost-1.lan': [${HOME}/.ssh/id_rsa]: timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [DEBUG] Reading ssh key '${HOME}/.ssh/id_rsa': timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [INFO] SSH connecting to 'vhost-1.lan' (vhost-1.lan): timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.792Z [ERROR] provider.terraform-provider-libvirt_v0.8.0: Response contains error diagnostic: @module=sdk.proto diagnostic_summary="failed to connect: failed to connect to remote host 'vhost-1.lan': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain" tf_req_id=3cc81bda-7e2c-db96-3131-50901aeb5b17 @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail="" diagnostic_severity=ERROR tf_proto_version=5.6 tf_provider_addr=provider tf_rpc=Configure timestamp=2024-09-29T11:53:10.792Z
2024-09-29T11:53:10.792Z [ERROR] vertex "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" error: failed to connect: failed to connect to remote host 'vhost-1.lan': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

---

Relevant code section:

The bug appears to be here in this code block (v0.8.0).

terraform-provider-libvirt/libvirt/uri/ssh.go

Lines 37 to 59 in 1a88931

	// keyfile order of precedence:
	// 1. load uri encoded keyfile
	// 2. load override as specified in ssh config
	// 3. load default ssh keyfile path
	sshKeyPaths := []string{}
	sshKeyPath := q.Get("keyfile")
	if sshKeyPath != "" {
	sshKeyPaths = append(sshKeyPaths, sshKeyPath)
	}
	keyPaths, err := sshcfg.GetAll(target, "IdentityFile")
	if err != nil {
	log.Printf("[WARN] unable to get IdentityFile values - ignoring")
	} else {
	sshKeyPaths = append(sshKeyPaths, keyPaths...)
	}
	if len(keyPaths) == 0 {
	log.Printf("[DEBUG] found no ssh keys, using default keypath")
	sshKeyPaths = []string{defaultSSHKeyPath}
	}
	log.Printf("[DEBUG] ssh identity files for host '%s': %s", target, sshKeyPaths)

Connection string for reference:

provider "libvirt" {
  uri = "qemu+ssh://[email protected]/system?keyfile=ansible.key&sshauth=privkey&no_verify=1"
}

In this section, the keyfile query param value in the connection string is retrieved and added to sshKeyPaths.

terraform-provider-libvirt/libvirt/uri/ssh.go

Lines 42 to 45 in 1a88931

	sshKeyPath := q.Get("keyfile")
	if sshKeyPath != "" {
	sshKeyPaths = append(sshKeyPaths, sshKeyPath)
	}

In this section, the ~/.ssh/config file is scanned for any instance of IdentityFile.

terraform-provider-libvirt/libvirt/uri/ssh.go

Lines 47 to 52 in 1a88931

	keyPaths, err := sshcfg.GetAll(target, "IdentityFile")
	if err != nil {
	log.Printf("[WARN] unable to get IdentityFile values - ignoring")
	} else {
	sshKeyPaths = append(sshKeyPaths, keyPaths...)
	}

In this section, the keyfile query param value is overwritten because ~/.ssh/config file is empty and then initializes with the defaultSSHKeyPath which is just ${HOME}/.ssh/id_rsa.

terraform-provider-libvirt/libvirt/uri/ssh.go

Lines 54 to 57 in 1a88931

	if len(keyPaths) == 0 {
	log.Printf("[DEBUG] found no ssh keys, using default keypath")
	sshKeyPaths = []string{defaultSSHKeyPath}
	}

Workaround

With the above understanding of the bug, there is one happy path. If the ~/.ssh/config includes any instance of IdentityFile for the given host, then it will also accept the query param keyfile.

# cat ~/.ssh/config
Host vhost-1.lan
  IdentityFile ~/.ssh/some-random.key

In the terraform log below, see that it reads the ansible.key from the connection string, and the ~/.ssh/some-random.key from the ~/.ssh/config. It then connects to vhost-1 successfully.

[INFO] establishing ssh connection to 'vhost-1.lan': timestamp=2024-09-29T12:07:30.325Z
[DEBUG] auth methods for vhost-1.lan: privkey: timestamp=2024-09-29T12:07:30.325Z
[DEBUG] ssh identity files for host 'vhost-1.lan': [ansible.key ~/.ssh/some-random.key]: timestamp=2024-09-29T12:07:30.325Z
[DEBUG] Reading ssh key 'ansible.key': timestamp=2024-09-29T12:07:30.325Z
[DEBUG] Reading ssh key '~/.ssh/some-random.key': timestamp=2024-09-29T12:07:30.325Z
[ERROR] Failed to parse ssh key: ssh: no key found: timestamp=2024-09-29T12:07:30.326Z
[INFO] SSH connecting to 'vhost-1.lan' (vhost-1.lan): timestamp=2024-09-29T12:07:30.326Z
[INFO] libvirt client libvirt version: 10000000: timestamp=2024-09-29T12:07:30.524Z

$ tf plan
data.libvirt_node_info.node: Reading...
data.libvirt_node_info.node: Read complete after 0s [id=2357352559]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

[dmacvicar]

@memetb

[korncola]

discovered the same, as I dont use RSA keys anymore, as the are legacy. The provider should at least do 2 things:

• also check for $HOME/.ssh/id_ed25519 if no id_rsa is in place
• not ignore connection string at any condition

[memetb]

@korncola can you clarify your second point: did you encounter a situation where the connection string was ignored? If so, can you please document?

Augmenting default behaviour re: id_rsa is an easy fix. I'll make a PR for that. @dmacvicar do you want a new issue associated with it?

[dmacvicar]

Yes, lets keep separate issues separate 🙏

[korncola]

@korncola can you clarify your second point: did you encounter a situation where the connection string was ignored? If so, can you please document?

Thanks for your quick response 👍

I have no ~/.ssh/id_rsa and no ~/.ssh/config and use this connection string:

provider "libvirt" {
  uri = "qemu+ssh://[email protected]/system?keyfile=/Users/korncola/id_ed25519&sshauth=privkey"
}

getting following error: (TF_LOG=DEBUG), which show that my connection string paramters are ignored.

The moment i create a ~/.ssh/config with an aliased host entry the connection string parameters are picked up and connection works.

But this should not be necessary as the connection string contains already all relevant information. IMHO even the parameters should not be necessary, as key auth is the first to try for every ssh client (except told otherwise explicitly) and also check for keys like id_ed25519. Thanks for putting this second thing on your roadmap 👍

2024-10-16T21:01:45.774+0200 [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/10/16 21:01:45 [DEBUG] Configuring provider for 'qemu+ssh://[email protected]/system?keyfile=/Users/korncola/.ssh/id_ed25519&sshauth=privkey': &{map[uri:0x140001d3e00] 0x140002594a0 <nil> 0x1400049a780 map[] <nil> {{<nil>} <nil>} 0x1400008bd80 0x1400033aeb8 0x1400034fd40 false {{{} 1} {0 0}} false false}: timestamp="2024-10-16T21:01:45.774+0200"
2024-10-16T21:01:45.774+0200 [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/10/16 21:01:45 [INFO] establishing ssh connection to 'kvm.example.com': timestamp="2024-10-16T21:01:45.774+0200"
2024-10-16T21:01:45.775+0200 [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/10/16 21:01:45 [DEBUG] Using known hosts file '/Users/korncola/.ssh/known_hosts' for target 'kvm.example.com': timestamp="2024-10-16T21:01:45.775+0200"
2024-10-16T21:01:45.775+0200 [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/10/16 21:01:45 [DEBUG] auth methods for kvm.example.com: privkey: timestamp="2024-10-16T21:01:45.775+0200"
2024-10-16T21:01:45.775+0200 [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/10/16 21:01:45 [DEBUG] found no ssh keys, using default keypath: timestamp="2024-10-16T21:01:45.775+0200"
2024-10-16T21:01:45.775+0200 [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/10/16 21:01:45 [DEBUG] ssh identity files for host 'kvm.example.com': [${HOME}/.ssh/id_rsa]: timestamp="2024-10-16T21:01:45.775+0200"
2024-10-16T21:01:45.775+0200 [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/10/16 21:01:45 [DEBUG] Reading ssh key '${HOME}/.ssh/id_rsa': timestamp="2024-10-16T21:01:45.775+0200"
2024-10-16T21:01:45.775+0200 [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/10/16 21:01:45 [ERROR] Failed to read ssh key '${HOME}/.ssh/id_rsa': open /Users/korncola/.ssh/id_rsa: no such file or directory: timestamp="2024-10-16T21:01:45.775+0200"
2024-10-16T21:01:45.775+0200 [ERROR] provider.terraform-provider-libvirt_v0.8.0: Response contains error diagnostic: diagnostic_severity=ERROR diagnostic_summary="failed to connect: could not configure SSH authentication methods" tf_req_id=ae022ac5-c7f7-a4ed-55da-bb391b9e59d4 tf_rpc=Configure @module=sdk.proto diagnostic_detail="" tf_proto_version=5.6 tf_provider_addr=provider @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/diag/diagnostics.go:58 timestamp="2024-10-16T21:01:45.775+0200"
2024-10-16T21:01:45.775+0200 [ERROR] vertex "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" error: failed to connect: could not configure SSH authentication methods
2024-10-16T21:01:45.775+0200 [WARN]  Planning encountered errors, so plan is not applyable
╷
│ Error: failed to connect: could not configure SSH authentication methods
│ 
│   with provider["registry.terraform.io/dmacvicar/libvirt"],
│   on test.tf line 1, in provider "libvirt":
│    1: provider "libvirt" {
│ 
╵
2024-10-16T21:01:45.777+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-10-16T21:01:45.777+0200 [INFO]  provider: plugin process exited: plugin=.terraform/providers/registry.terraform.io/dmacvicar/libvirt/0.8.0/darwin_arm64/terraform-provider-libvirt_v0.8.0 id=39945
2024-10-16T21:01:45.777+0200 [DEBUG] provider: plugin exited

[darachm]

I believe I've also got a similar problem as described here.
Trying to setup by using something like:
uri = "qemu+ssh://[email protected]/system?keyfile=/home/zed/.ssh/user"
and got error:
Error: failed to connect: could not configure SSH authentication methods
re-running with TF_LOG=DEBUG (thanks korncola !), and I see that it is trying to use: "[${HOME}/.ssh/id_rsa]" .
If I create a symlink from ~/.ssh/id_rsa to the right key, then it works.

FYI, I doubt it matters, but I've got a host of other issues on the remote device as I'm brand new to this VM business and trying to set that all up now