No interfaces obtained via qemu-agent #891

matthijssjansen · 2021-10-06T14:59:34Z

System Information

Linux distribution

Ubuntu 20.04

Terraform version

1.0.6

Provider and libvirt versions

0.6.2

Checklist

Is your issue/contribution related with enabling some setting/option exposed by libvirt that the plugin does not yet support, or requires changing/extending the provider terraform schema?
- Make sure you explain why this option is important to you, why it should be important to everyone. Describe your use-case with detail and provide examples where possible.
- If it is a very special case, consider using the XSLT support in the provider to tweak the definition instead of opening an issue
- Maintainers do not have expertise in every libvirt setting, so please, describe the feature and how it is used. Link to the appropriate documentation
Is it a bug or something that does not work as expected? Please make sure you fill the version information below:

Description of Issue/Question

Setup

Terraform:
cloudedge.txt

Cloudinit:
network_config_static.txt
cloud_init.txt

Steps to Reproduce Issue

I want to create VMs and expose them to the local network via a network bridge br0. However, using Terraform the VMs are stuck on the following (from TF_LOG=DEBUG terraform apply)

2021-10-06T13:45:01.302Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] sending command to qemu-agent
2021-10-06T13:45:01.307Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] qemu-agent response: {"return":[{"name":"lo","ip-addresses":[{"ip-address-type":"ipv4","ip-address":"127.0.0.1","prefix":8},{"ip-address-type":"ipv6","ip-address":"::1","prefix":128}],"statistics":{"tx-packets":6,"tx-errs":0,"rx-bytes":416,"rx-dropped":0,"rx-packets":6,"rx-errs":0,"tx-bytes":416,"tx-dropped":0},"hardware-address":"00:00:00:00:00:00"},{"name":"eth0","ip-addresses":[{"ip-address-type":"ipv6","ip-address":"fe80::5054:ff:fe7e:962e","prefix":64}],"statistics":{"tx-packets":29,"tx-errs":0,"rx-bytes":27610,"rx-dropped":5,"rx-packets":224,"rx-errs":0,"tx-bytes":7838,"tx-dropped":0},"hardware-address":"52:54:00:7e:96:2e"}]}
2021-10-06T13:45:01.307Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] Parsed response {Interfaces:[{Name:lo Hwaddr:00:00:00:00:00:00 IPAddresses:[{Type:ipv4 Address:127.0.0.1 Prefix:8} {Type:ipv6 Address:::1 Prefix:128}]} {Name:eth0 Hwaddr:52:54:00:7e:96:2e IPAddresses:[{Type:ipv6 Address:fe80::5054:ff:fe7e:962e Prefix:64}]}]}
2021-10-06T13:45:01.307Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] Interfaces obtained via qemu-agent: []
2021-10-06T13:45:01.307Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] no interfaces could be obtained with qemu-agent: falling back to the libvirt API
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] Interfaces info obtained with libvirt API:
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: ([]libvirt.DomainInterface) {
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: }
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] ifaces with addresses: []
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] 52:54:00:7E:96:2E doesn't have IP address(es) yet...
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [DEBUG] IP address not found for iface=52:54:00:7E:96:2E: will try in a while
2021-10-06T13:45:01.308Z [DEBUG] provider.terraform-provider-libvirt-local: 2021/10/06 13:45:01 [TRACE] Waiting 10s before next try

Resulting in

Error: Error: couldn't retrieve IP address of domain.

The VMs have qemu-guest-agent running on them, which gives back this list of network interfaces available on the VM, but this list is missing something / incorrect according to Terraform. The VM ends up having no IP (using virsh console to login to the VM, followed by `hostname -i```.

Additional information:

I have tried many permutations of the given terraform and cloudinit file, including other terraform-provider-libvirt versions, other operating systems for the VM, using static IPs and / or dynamic IPs via DHCP. At this point I am out of ideas and would like some help in figuring out what goes wrong.

I am able to create VMs with static IPs and a network bridge using the Qemu CLI (without Terraform).

The text was updated successfully, but these errors were encountered:

mxkle · 2021-10-10T10:11:17Z

Hi,

i think i have the same problem. I try to deploy a debian vm with this plugin but it runs into a timeout while trying to get the ip of the vm. On my dhcp server i can see that the vm successfully obtained an address but terraform isnt recognizing it.

my tf file:

terraform {
  required_version = ">= 0.13"
    required_providers {
      libvirt = {
        source  = "dmacvicar/libvirt"
        version = "0.6.11"
      }
    }
}

resource "libvirt_domain" "virt-machine" {
  count  = var.vm_count
  name   = format("${var.vm_hostname_prefix}%02d", count.index + var.index_start)
  memory = var.memory
  vcpu   = var.vcpu
  autostart  = var.autostart
  qemu_agent = true
  
  cloudinit = element(libvirt_cloudinit_disk.commoninit.*.id, count.index)

  network_interface {
    bridge         = var.bridge
    wait_for_lease = true
    hostname       = format("${var.vm_hostname_prefix}%02d", count.index + var.index_start)
  }

  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }

  console {
    type        = "pty"
    target_type = "virtio"
    target_port = "1"
  }

  disk {
    volume_id = element(libvirt_volume.volume-qcow2.*.id, count.index)
  }

  graphics {
    type        = "spice"
    listen_type = "address"
    autoport    = true
  }
}

while terraform tries to obtain the ip of the vm, i can already find it via virsh:

virsh qemu-agent-command nomad01 '{"execute": "guest-network-get-interfaces"}'
{
  "return": [
    {
      "name": "lo",
      "ip-addresses": [
        {
          "ip-address-type": "ipv4",
          "ip-address": "127.0.0.1",
          "prefix": 8
        },
        {
          "ip-address-type": "ipv6",
          "ip-address": "::1",
          "prefix": 128
        }
      ],
      "statistics": {
        "tx-packets": 0,
        "tx-errs": 0,
        "rx-bytes": 0,
        "rx-dropped": 0,
        "rx-packets": 0,
        "rx-errs": 0,
        "tx-bytes": 0,
        "tx-dropped": 0
      },
      "hardware-address": "00:00:00:00:00:00"
    },
    {
      "name": "ens3",
      "ip-addresses": [
        {
          "ip-address-type": "ipv4",
          "ip-address": "192.168.2.117",
          "prefix": 24
        },
        {
          "ip-address-type": "ipv6",
          "ip-address": "fe80::5054:ff:fe62:d25f",
          "prefix": 64
        }
      ],
      "statistics": {
        "tx-packets": 56,
        "tx-errs": 0,
        "rx-bytes": 70690,
        "rx-dropped": 5,
        "rx-packets": 743,
        "rx-errs": 0,
        "tx-bytes": 8065,
        "tx-dropped": 0
      },
      "hardware-address": "52:54:00:62:d2:5f"
    },
    {
      "name": "docker0",
      "ip-addresses": [
        {
          "ip-address-type": "ipv4",
          "ip-address": "172.17.0.1",
          "prefix": 16
        }
      ],
      "statistics": {
        "tx-packets": 0,
        "tx-errs": 0,
        "rx-bytes": 0,
        "rx-dropped": 0,
        "rx-packets": 0,
        "rx-errs": 0,
        "tx-bytes": 0,
        "tx-dropped": 0
      },
      "hardware-address": "02:42:3e:b2:81:46"
    }
  ]
}

in my cloud init file i also install, start and enable the qemu-guest-agent.service.
before terraform aborts i can connect to the vm via ssh and verify that its running:

$ sudo systemctl status qemu-guest-agent.service 
● qemu-guest-agent.service - QEMU Guest Agent
     Loaded: loaded (/lib/systemd/system/qemu-guest-agent.service; static)
     Active: active (running) since Sun 2021-10-10 11:35:23 CEST; 23min ago
   Main PID: 455 (qemu-ga)
      Tasks: 2 (limit: 529)
     Memory: 2.3M
        CPU: 5ms
     CGroup: /system.slice/qemu-guest-agent.service
             └─455 /usr/sbin/qemu-ga

Oct 10 11:35:23 nomad01 systemd[1]: Started QEMU Guest Agent.

and here is the output from the terraform apply command:

2021-10-10T10:07:14.825Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] waiting for network address for iface=52:54:00:62:D2:5F
2021-10-10T10:07:14.826Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] Not implemented
2021-10-10T10:07:14.826Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] no interfaces could be obtained with qemu-agent: falling back to the libvirt API
2021-10-10T10:07:14.826Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] Interfaces info obtained with libvirt API:
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: ([]libvirt.DomainInterface) <nil>
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] ifaces with addresses: []
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] 52:54:00:62:D2:5F doesn't have IP address(es) yet...
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [DEBUG] IP address not found for iface=52:54:00:62:D2:5F: will try in a while
2021-10-10T10:07:14.827Z [DEBUG] provider.terraform-provider-libvirt_v0.6.11: 2021/10/10 10:07:14 [TRACE] Waiting 10s before next try

I found this issue here: #710 its closed because there was already a fix for a problem with channel bindings.

remoe · 2021-10-10T14:02:10Z

qemu agent is not supported in latest version: #879

0.6.3 : works - https://github.com/dmacvicar/terraform-provider-libvirt/blob/v0.6.3/libvirt/domain.go#L157
v0.6.9-pre1 : not implemented! https://github.com/dmacvicar/terraform-provider-libvirt/blob/v0.6.9-pre1/libvirt/domain.go#L159

mxkle · 2021-10-10T15:15:17Z

Oh thanks for the info remoe, i somehow missed that. Sorry!
I will switch to version 0.6.3 and try agian.

remoe · 2021-10-10T15:26:03Z

@mxkle , but this version would not work with terraform 1.x

everflux · 2021-11-21T21:10:44Z

Is there a version that supports the qemu-agent and works with terraform 1.0.x?

remoe · 2021-11-21T21:23:24Z

@everflux, no wait for https://github.com/dmacvicar/terraform-provider-libvirt/pull/ 873 is reviewed and merged in.

everflux · 2021-11-22T11:59:40Z

The PR is reviewed but it looks like it will not be merged, it is 1,5h years old and conflicts with the current main.
I can understand the requested changes by @dmacvicar but I am afraid that it won't materialize.

dmacvicar · 2021-12-12T01:37:32Z

I have manually rebased and committed #873

I don't like the 120M image for the test, so the test is pending and I opened #913 to revisit it.

remoe mentioned this issue Oct 10, 2021

Output IP addresses not returned from Qemu Guest Agent #893

Closed

dmacvicar closed this as completed Dec 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No interfaces obtained via qemu-agent #891

No interfaces obtained via qemu-agent #891

matthijssjansen commented Oct 6, 2021

mxkle commented Oct 10, 2021 •

edited

Loading

remoe commented Oct 10, 2021 •

edited

Loading

mxkle commented Oct 10, 2021

remoe commented Oct 10, 2021

everflux commented Nov 21, 2021

remoe commented Nov 21, 2021

everflux commented Nov 22, 2021

dmacvicar commented Dec 12, 2021

No interfaces obtained via qemu-agent #891

No interfaces obtained via qemu-agent #891

Comments

matthijssjansen commented Oct 6, 2021

System Information

Linux distribution

Terraform version

Provider and libvirt versions

Checklist

Description of Issue/Question

Setup

Steps to Reproduce Issue

Additional information:

mxkle commented Oct 10, 2021 • edited Loading

remoe commented Oct 10, 2021 • edited Loading

mxkle commented Oct 10, 2021

remoe commented Oct 10, 2021

everflux commented Nov 21, 2021

remoe commented Nov 21, 2021

everflux commented Nov 22, 2021

dmacvicar commented Dec 12, 2021

mxkle commented Oct 10, 2021 •

edited

Loading

remoe commented Oct 10, 2021 •

edited

Loading