Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New 'vapp properties' do not work with Ubuntu 14.04/16.04 OVA #381

Closed
sdemura opened this issue Feb 2, 2018 · 15 comments
Closed

New 'vapp properties' do not work with Ubuntu 14.04/16.04 OVA #381

sdemura opened this issue Feb 2, 2018 · 15 comments
Labels
bug Type: Bug

Comments

@sdemura
Copy link

sdemura commented Feb 2, 2018

Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform
issue trackers reserved for bug reports and feature requests. For general usage
questions, please see: https://www.terraform.io/community.html.

Terraform Version

terraform-vsphere$ terraform -v
Terraform v0.11.3
+ provider.vsphere v1.3.1

vSphere Provider Version

terraform-vsphere$ terraform providers
.
└── provider.vsphere 1.3.1

Affected Resource(s)

Please list the resources as a list, for example:

  • vsphere_virtual_machine

Terraform Configuration Files

resource "vsphere_virtual_machine" "vm" {
  name             = "terraform-xenial"
  resource_pool_id = "${data.vsphere_resource_pool.pool.id}"
  datastore_id     = "${data.vsphere_datastore.datastore.id}"

  num_cpus = 2
  memory   = 1024
  guest_id = "ubuntu64Guest"

  network_interface {
    network_id = "${data.vsphere_network.network.id}"
  }

  disk {
    name = "terraform-xenial.vmdk"
    size = 40
  }

  vapp {
    properties {
      "password" = "super-secure-password"
      "hostname" = "custom-hostname"
    }
  }

  clone {
    template_uuid = "${data.vsphere_virtual_machine.template.id}"
  }
}

Expected Behavior

VM should have been created, and configured with the provided vApp properties.

Actual Behavior

VM hostname and password are not changed.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform apply

Important Factoids

Using govc import.ova -options=xenial.json works as expected, where xenial.json looks like:

{
    "DiskProvisioning": "thin",
    "IPAllocationPolicy": "dhcpPolicy",
    "IPProtocol": "IPv4",
    "InjectOvfEnv": false,
    "Name": null,
    "NetworkMapping": [
        {
            "Name": "VM Network",
            "Network": ""
        }
    ],
    "PowerOn": true,
    "PropertyMapping": [
        {
            "Key": "instance-id",
            "Value": "id-ovf"
        },
        {
            "Key": "hostname",
            "Value": "custom-hostname"
        },
        {
            "Key": "seedfrom",
            "Value": ""
        },
        {
            "Key": "public-keys",
            "Value": ""
        },
        {
            "Key": "user-data",
            "Value": ""
        },
        {
            "Key": "password",
            "Value": "super-secure-password"
        }
    ],
    "WaitForIP": false
}

When deploying via vSphere GUI, these options are configurable and work as well.

References

vApp options were added in #303

@sdemura sdemura changed the title New 'vapp' options do not work with Ubuntu 14.04/16.04 OVA New 'vapp properties' do not work with Ubuntu 14.04/16.04 OVA Feb 2, 2018
@vancluever vancluever added the bug Type: Bug label Feb 2, 2018
@sdemura
Copy link
Author

sdemura commented Feb 2, 2018

I have been able to get this work.

In order to inject ovfEnv variables into a vSphere VM, an CDROM device with an appropriate ISO needs to be attached. The govc and vSphere GUI do this automatically for you, but terraform does not.

If I add a CDROM, then the VM boots as expected with the desired configurations:

resource "vsphere_virtual_machine" "vm" {
  name             = "terraform-xenial"
  resource_pool_id = "${data.vsphere_resource_pool.pool.id}"
  datastore_id     = "${data.vsphere_datastore.datastore.id}"

  num_cpus = 2
  memory   = 1024
  guest_id = "ubuntu64Guest"

  network_interface {
    network_id = "${data.vsphere_network.network.id}"
  }

  disk {
    name = "terraform-xenial.vmdk"
    size = 40
  }

  vapp {
    properties {
      password    = "super-secure-password"
      hostname    = "custom-hostname"
    }
  }

  cdrom {
    datastore_id = "${data.vsphere_datastore.datastore.id}"
    path         = "terraform-xenial/_ovfenv-terraform-xenial.iso"
  }

  clone {
    template_uuid = "${data.vsphere_virtual_machine.template.id}"
  }
}

@vancluever
Copy link
Contributor

Hey @sdemura, thanks for opening this. Funny enough I was looking at this last night!

I'll investigate this a bit more to see why, but I didn't have much success getting this working 100% by just doing a OVF import from govc and just booting the VM without setting any of the options. Not too sure why, but on my end the OVF would not boot properly - I got an error upon startup and then it hung for a significant period of time waiting for fd0. I'll work on repros using the path that you mentioned too to see what's different, however right off the bat I did notice that the Ubuntu OVF uses the ISO transport and not the guestinfo transport like CoreOS does. Not too sure if that has anything to do with it though.

Will update when I have more info.

Thanks!

@sdemura
Copy link
Author

sdemura commented Feb 2, 2018

@vancluever looks like we posted at the same time. see above ^

@vancluever
Copy link
Contributor

@sdemura I saw! Glad you figured it out and I was suspecting this might be the case.

Sounds like we need to prioritize some of the CDROM update work to ensure that OVF templates with ISO transports work as expected. We have other work that is pending here (#300 and #358), so maybe we can group this all together.

@charandas
Copy link

charandas commented Feb 7, 2018

Great. Its quite a coincidence, I have been grappling, struggling and learning from similar work with ubuntu OVA images in the last week as well.

Did anyone notice a method for providing network config through user-data?

This gist seems to imply, that the OVF transport of ISO at least supports network-config as independent from user-data.

If that is so, why does the current ubuntu 16.04 OVA envelope does not have a key for it.

EDIT: maybe as this page mentions,

instance-id: iid-abcdefg
network-interfaces: |
  iface eth0 inet static
  address 192.168.1.10
  network 192.168.1.0
  netmask 255.255.255.0
  broadcast 192.168.1.255
  gateway 192.168.1.254
hostname: myhost

Its in meta-data implying that the key is network-interfaces.

@charandas
Copy link

charandas commented Feb 22, 2018

In #74, @jason-azze suggested a trick to create an empty CDROM device, which works. I am noticing however that the cloud init bootstrap over ISO doesn't work over that.

If I shutdown the machine, changing it to a client device from datastore ISO, and reboot, it works.

@vancluever is there a roadmap or expected date when empty client device CDROM support is coming to this provider - last I heard, you said its just a matter of time. Excited about being able to bootstrap my VMs end-to-end without manual tinkering.

@sdemura
Copy link
Author

sdemura commented Mar 5, 2018

@charandas did you base64 encode your data?

Here's an example where I added user-data:

  vapp {
    properties {
      user-data = "${base64encode("#cloud-config
ssh_pwauth:   0
disable_root: 1
runcmd: 
  - ['hostnamectl', 'set-hostname', 'my-hostname']
ssh_authorized_keys:
  - <ssh key here>")
      }"
    }
  }

@charandas
Copy link

Yes @sdemura. The problem isn’t that the user data is invalid. In my countless experiments, the ovf transport hasn’t worked for me on my Ubuntu boxes.

But ovf iso transport works everytime. I just check the option to make available the vapp properties as a cdrom in the template itself on vsphere.

This has made me think that cloud-init on vsphere cannot read the data out unless it’s from the cdrom.

Maybe you know a way about that, if you do, please share.

@vancluever
Copy link
Contributor

@charandas not that I can speak from 100% experience on the Ubuntu template, but there is a guestinfo transport specifically in cloud-init designed to read user data and other cloud-config keys from guestinfo.ovfEnv which is usually fed in by vApp properties. You can't edit this key manually - vSphere will overwrite every time you attempt to.

OTOH, if a template does not have a vApp configuration defined, it will fall back to the specific guestinfo keys directly. These can be specified in the advanced configuration of a virtual machine or by using extra_config in the vsphere_virtual_machine resource.

This work was initially contributed here, if I understand correctly.

Of course, cloud-init has been re-implemented a few times by different distros, so these rules may not apply 100%, but I know for sure this works in this fashion on the CoreOS re-implementation.

Hopefully we can work on getting this completely resolved soon the correct way for Ubuntu - delivery via the ISO transport. We are working on a number of updates to CDROM device handling, of which this will be one of them.

Thanks!

@charandas
Copy link

charandas commented Mar 6, 2018

Thanks @vancluever for the detailed answer. This works for me on coreos without needing the CDROM gymnastics also.

My template AFAIK for ubuntu (like my coreos ones) has vApp configuration keys, and they are defined with the help of this provider when provisioning (using the new vapp key).

cloud-init however fails to see it unless I expose that vApp definition in form of the CDROM on ubuntu. To elaborate, I am using a feature in vsphere to provide guestinfo.ovfEnv on the first CDROM device.

Since that works, I have decided to not fight the tigers, and roll with the CDROM option. If in future however, I see the guestinfo.ovfEnv working directly for me, I would be more than happy to switch to it (my guess is, it depends on cloud-init code in ubuntu).

Thanks much!

@eperdeme
Copy link

eperdeme commented Mar 12, 2018

Howdi,

is this fix for this on;
52e17d0

Looking at that commit it would allow you to specify ClientDevice as CDROm to not force an ISO only mount.

From my testing providing you have a CDROM on the VM which is in ClientDevice mode, then vSphere will pass through the OVF properties in the ISO format in to the running VM, which cloud-init picks up fine.

** Just compiled the provider from master and looks good;

  cdrom {
    client_device = true
  #  datastore_id = "${data.vsphere_datastore.datastore.id}"
  #  path         = ""
  }
	vapp {
    properties {
      "user-data" = "${base64encode("
#cloud-config
yum_repos:
  cx_repo:
   baseurl: \"https://yum-repo01.dev.mycompany.plc/pub/repo\"
   enabled: true
   gpgcheck: false
   name: cx_repo
   sslverify: false
")
	}"
    }
  }

@bill-rich
Copy link
Contributor

Hi @eperdeme, support for mapping CDROMs to client devices does partially resolve this issue. When a CDROM with client_device is configured and vapp options are specified, the vApp config ISO will be created and connected to the CDROM. This will work for when ISO transport is needed, but results in a diff between the config and the state since the CDROM is then mapped to a datastore ISO. We are working on how best to handle this so that there's a better user experience, but in the meantime, creating a CDROM with client_device will work.

@mcanevet
Copy link

@charandas did you find a way to configure network using vapp properties?

@charandas
Copy link

@mcanevet I am not sure what exactly you are after, the cloud-config part or the terraform-vsphere integration part. For the latter, ensure, that the OVF environment transport allows ISO as shown in the screenshot. You would need to convert the template into a VM to access that dialog. Once done, convert it back to a template.

screen_shot_2018-03-28_at_1 10 58_pm

For the cloud-init itself, something like this pseudo-code (depending on your tools - I use jsonnet prior to terraform) worked for me. You would have to base64 encode this before passing to terraform.


local config  = @"network:\n    version: 2\n    renderer: networkd\n    ethernets:\n      ens192:\n        dhcp4: no\n        addresses: [$${ip_address}]\n        gateway4: $${gateway}\n        nameservers:\n          addresses: [$${dns_0}]\n"

#cloud-config
runcmd:
- 'echo 127.0.0.1 `hostname` >> /etc/hosts'
- 'rm -f /etc/netplan/50-cloud-init.yaml && echo "network: {config: disabled}" > /etc/cloud/cloud.cfg.d/99-disable-network-config && printf "%s" > /etc/netplan/40-static-ens192.yaml'  % [config];
- netplan apply

@bill-rich
Copy link
Contributor

Thanks for the feedback on this issue everyone!

Support for delivering vApp properties using ISO transport has been added in GH-440. A CDROM with client_device is still required, but a diff will not be generated when the CDROM has the vApp ISO inserted. A check was also added to ensure a client_device CDROM is present when a virtual machine is configured to use vApp ISO transport. That should help avoid situations where the virtual machine isn't receiving the properties, but it isn't clear why. I believe GH-440 should cover the use cases that were brought up here.

@ghost ghost locked and limited conversation to collaborators Apr 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Type: Bug
Projects
None yet
Development

No branches or pull requests

6 participants