OVA upload support for vsphere provider

[hashibot]

This issue was originally opened by @chinmayb as hashicorp/terraform#11815. It was migrated here as part of the provider split. The original body of the issue is below.

---

Terraform #Version

terraform v0.8.6

Affected Resource(s)

• vsphere

Expected Behavior

template - (Required if size and bootable_vmdk_path not provided) Template for this disk.
disk { template } field should take the OVA location ( remote URL or a file path). Currently takes the template which is present in the vCenter.

or

Introduce a new resource "vsphere_ova" to allow uploading the OVA to the vSphere.

Actual Behavior

Currently OVA images are not supported.

[alekssaul]

I was thinking about how this could potentially work, and figured I'd jot it down into the issue here.

We have a use case for uploading OVA's as well. Currently, working around it by importing the OVA "by hand" and converting into a template, than referencing it as a disk template.

Using an OVA as a field in disk {template} would probably work, however if you have multiple VMs to create with the OVA, this would most likely require an OVA upload action for each VM creation (slow).

I was thinking an introduction of a new "VM template" resource may be a good approach. Bundled with ability to create the VM Template from OVA, VM Template can be created once and referenced as resource "vsphere_virtual_machine" {disk {template}} once in vCenter.

PS: It looks like uploading of Remote URL for OVA is not supported in govmomi (which terraform-provider-vsphere uses), so I imagine URL based uploads would have to wait for upstream support.

[alekssaul]

@chinmayb, I am playing with some ideas on this.

So far, I have a vsphere_template resource that converts a VM to template and template back to vm (on destroy).

resource "vsphere_template" "test" {
  name = "test"
  virtualmachine = "test"
  datacenter = "nyc"
  cluster = "cluster01"
}

my current thought process is to add ova_file and ova_url parameters to the flow, so

• OVA gets uploaded to vCenter
• Default object is a Virtual Machine
• Virtual Machine gets converted to template
• can be referenced to generate bunch of Virtual Machines

Does this sounds like a solution that would be helpful for your use-case as well?

[mkuzmin]

Are you sure this should be a Terraform's feature?
In HashiCorp tool stack there is a clear separation of concerns: Packer prepares base templates, Terraform launches instances.

I'm developing a new native vSphere builder for Packer, maybe this functionality better fits there.

[alekssaul]

Hi @mkuzmin ,

Thank you for responding. I am not sure, I am hoping you can provide guidance.

My main goal for the feature is not to create(pack) the OVA, but to produce a vSphere VM Template object out of an OVA. My understanding from the HashiCorp stack is Packer will produce the actual OVA image, but would not import it to vSphere as a VM template that can be referenced by terraform, hence my idea on the feature.

I did not find a reasonable way to take an OVA image and produce Virtual Machines in vSphere via Terraform natively. The process I've been following was,

1. Download OVA image
2. Upload OVA image into vSphere
3. Convert OVA image to a vSphere Virtual Machine template
4. Use template as a Virtual Machine's Disk template with Terraform

My main use case is to Download/upload CoreOS Container Linux (already published OVA/OVF format) and generate a Kubernetes cluster out of it via terraform.

Please let me know if there is a better approach.

Update: The feature I had in mind is aligned with something like; aws_ami or aws_ami_from_instance resources in Terraform AWS provider

[mkuzmin]

I see several possible solutions (all don't relate to Terraform):

1. Use ovftool or govc to upload OVF.
2. Develop vsphere-ovf builder in Packer. It will get binary files, upload them to vSphere, and produce VM templates. In future this may become a feature of my Packer plugin.
3. Do not use OVF sources at all. Install CoreOS right from ISO-image, and add VMware tools explicitly.

I'd prefer 3rd solution, but at the moment my Packer builder does not support creating new VMs from ISO-files yet. Official builder can do that, but it requires using VMware Player.

[alekssaul]

Thanks @mkuzmin ,

My concern with the option 1 is that it requires another tool, to manage the lifecycle. So provisioning a complete cluster essentially requires terraform + scripting and tooling, and doesn't plug into terraform lifecycle. As an example; consider user executes option 1, and has a cluster. If an operator accidentally removes the template from vCenter, further terraform apply actions such as us "scaling the cluster up" with terraform will fail. Whereas if terraform was to manage the process, terraform can re-create the template and spin up new instances.

Option 3, is not quite simple in CoreOS world as the end user needs to track tested version of VMware tools for the particular image, package the vmware tools in a container, distribute it to the OS, create systemd unit to run it in host namespace etc. CoreOS OVA image ships the tools in OEM partition. My understanding is the OEM partition is intended for the platform tools and not meant to be end user managed. It may also break workflow that terraform may require, as (to my best knowledge) Terraform expect the Virtual Machine resource to be a managed object once it's powered on. In this scenario, Virtual Machine will come up "empty", and the OS will require an installation script and a reboot before it can be a managed resource. I believe Option 3 is best suited for a tool like packer. The intend is not for terraform to create the OS images, but consume them.

I believe Option 2 is a good approach. Looking at previous examples for AWS, it seems like packer-aws "builds an AMI by launching an EC2 instance from a source AMI, provisioning that running machine, and then creating an AMI from that machine.". However "The builder does not manage AMIs. Once it creates an AMI and stores it in your account, it is up to you to use, delete, etc. the AMI.". I believe that's the sweet spot for Terraform. Terraform provider for AWS provides aws-ami resource and aws-ami-from-instance resources, Openstack provider has openstack_images_image_v2 resource that are akin to what I had in mind.

Based on examples of Terraform/Packer implementations in other platforms (AWS and Openstack in particular). It sounds reasonable for Packer-vsphere to implement means to create OVF images and terraform provider to implement vsphere-template resources to consume them.

[johnmarcou]

IMHO, vpshere_template is to vSphere provider what docker_registry_image is to Docker provider.

Both are downloading "images" to run instances, especially with the ova_url argument. So I think this template/ova/image downloading feature makes perfect sense.

[qubitrenegade]

We too have a requirement to either "deploy from" ova or "deploy an ova" as a template.

We have a number of vendor provided appliances that come on OVAs. Currently our workflow is to manually deploy the ova, fill out any fields, convert it to a template, then we can continue using Terraform to deploy instances from the manually generated template.

The other way we consume OVAs is we have a packer process that generates the ova and uploads it to artifactory. Artifactory then takes care of mirroring that image to all the remote sites. Then we manually register the OVA as a template and can continue using Terraform.

Thing is, packer already has the functionality to upload an ova and register it as a template, however, packer must be the one to generate the OVA as there's no way to tell packer to take a generated artifact and just run post-processors on it.

I definitely think it would be a useful feature of Terraform to be able to consume an OVA, however, it really seems like it would be a job better suited for packer if the end goal is a template registered in vSphere... however, if Packer will never have the ability to run post-processors without running a builder then I think it's an absolutely necessary feature for terraform. As others have pointed out above, it's not unprecedented for a Terraform Provider to consume an image and register it.

Anyway, that's just my $0.02. We would definitely like to see and would benefit greatly from this functionality.

Thanks!

• Q

[vancluever]

Hey @qubitrenegade and everyone else - this is definitely one of the three major workflows that we want to support in the new vsphere_virtual_machine resource, so stay tuned!

[vancluever]

Hey all, just an update here: PR #303 will be hopefully merged in the next couple of weeks, at which point we will have OVA support via the vApp properties interface.

Where we go from there I'm still trying to decide, as this PR closes the hole in the feature set specific to vsphere_virtual_machine. Since you would now be able to clone a template that was the result of an OVF import, and set the appropriate properties pre-power on, you are essentially getting the same functionality you would be getting on the import dialog in the console.

The last question that needs to be answered is - do we support upload or not? I'm torn on this one, and am leaning towards no, for the following reasons:

• OVFs are much larger than other kinds of items that we support uploading from local file in other providers (ie: AWS Lambda articfacts). This runs a higher risk of TF runs failing due to some sort of problem or the other.
• We would not be doing much with the OVF post-import other than either registering it as a virtual machine or template, and the resource would be prone to failure if the OVF template was to go away.
• As mentioned, Packer already has this functionality. Another tool that fits well into the TF ecosystem is govc, which also has this functionality via govc import.ova, which works if you just run it without supplying the property JSON file and then mark the imported OVA as a template afterwards. This is actually a workflow that I was planning on working with after we have r/virtual_machine: add vApp properties #303 in, to take advantage of Ignition in the CoreOS OVA.

So in summary, I think I agree with @qubitrenegade here and others who say that OVA uploading may not be a good idea. In this case, and in a packer/Terraform pattern, the OVA actually the "artifact", and that has never been a thing in providers like AWS, etc.

As for the data source, we now have vsphere_virtual_machine, which I think serves the purposes discussed above.

More info on this as we get closer to the 1.3.0 release which is where I'm hoping the vApp property support will land in.

[qubitrenegade]

do we support upload or not?

To be honest, the general response from the Packer community has been... rather insulting and combative regarding this issue, i.e.: the lack of a method to register an OVA with vSphere if we're not building a fresh OVA every time we run packer... Which to our organization presents a serious gap in our automation workflow.

For technical reasons, I still am not convinced that Terraform is the appropriate tool... but from a purely political standpoint, the Packer community refuses to even discuss this issue while you @vancluever have been a pleasure to work with and are willing to have the hard conversations...

So, perhaps there's a case from the political angle for including this feature because you're the only one willing to have this conversation, instead of telling me to go use another tool...

[Lyoness]

Thanks for your work into this @vancluever !

[charandas]

That's excellent and I have been using #303 with coreos projects.

With ubuntu and cloud-init, its not working as fluidly, as the OVF transport used by default does not work for cloud-init. However, using the CDROM transport works.

@vancluever or anyone else, if you have some details on why cdrom option for terraform plugin's vsphere_virtual_machine resource does not support an empty cdrom device, please advise. Even if I add a cdrom client device to the underlying template, terraform scraps that during cloning.

I currently cannot fully automate the bringup of my VMs, as I am relying on vsphere to attach the OVF env on the first empty cdrom device, which I cannot create on my new VM using terraform.

I can create a separate issue as well.

[vancluever]

Hey @charandas, the vApp option delivery over ISO quirks are currently being tracked in #381, and a workaround is documented there as well, so long as you don't need the CDROM drive for anything else.

Transparent vApp option ISO delivery and empty CDROM device support are both things that we want to support, so it's just a matter of time before they are all added.

Thanks!

[jason-azze]

I'm not working with OVAs yet, but I have been able to create empty CDROM devices using this method:

My data source:

data "vsphere_datastore" "iso-datastore" {
  name          = "${var.images_datastore_designator}"
  datacenter_id = "${data.vsphere_datacenter.target-datacenter.id}"
}

My cdrom device instantiation:

  cdrom {
    datastore_id = "${data.vsphere_datastore.iso-datastore.id}"
    path         = "${var.iso_path}"
  }

Now set the value of iso_path to nothing at runtime. I set

iso_path = ""

and images_datastore_designator to some real vSphere datastore at runtime:

images_datastore_designator = "dev-ops/nubrdata01-nu01"

Where dev-ops is a folder that exists and nubrdata01-nu01 is a datastore that exists.

[vancluever]

Hey all, now that #303 has been merged and has been available for a few weeks now, I'm going to close this. For the ISO transport issues please watch #381.

To answer the elephant in the room question on whether or not Terraform should support uploading OVAs, I think at this point in time, the answer is no, as I really do believe that this kind of thing should be the job of Packer or some other tool.

To that end, last week I put together a CoreOS example that does an end-to-end build of a webapp, pushing of the CoreOS OVA to vSphere, and then a Terraform deploy to launch VMs to a 3-node vSphere cluster, one node per machine, using the Ignition provider to configure the instances, and then finally using provisioners to push the built binary to the machine (to avoid a large amount of data sitting in the config data property, whose maximum is 64k anyway). This process uses govc import.ova to import the OVA. You can see the full example currently in PR #383.

Eventually what I'd like to see is Packer be a part of this ecosystem, but there is some work that needs to be done for that to happen, namely it needs a proper vSphere API builder (my personal opinion). Eventually, I could see something like either there being something like vsphere-ova in addition to vsphere-iso, and vsphere-clone, or whatever, which would allow a build path very similar to something like amazon-ebs where the template is imported and then booted just like an AWS instance, with provisioners then connecting to the bootstrapped VM and doing their thing, with the pre-existing vsphere-template post-processor then marking that VM as a template.

Given all of the above, I don't think it really makes sense to try and put Terraform into a box it really doesn't fit in, but rather mitigate the lack of support mentioned above with a well-known tool like govc for the time being, which ultimately may serve the purpose better for people that don't necessarily need to heavily customize an OVA before it's deployed anyway.

Thanks all, and I hope you find the new vApp/OVF/OVA property support useful!

[fredwangwang]

openstack provider has the ability to upload an image https://www.terraform.io/docs/providers/openstack/r/images_image_v2.html. Just curious about how much the difference is between import ova and upload image...
Is there any particular reason supporting the similar operation on openstack but not on vsphere?