investigate using vSphere API

[mwhooker]

There appear to be some limitations with how we work with vSphere, where the solution seems to be using the API directly.

Will use this as a meta-ticket to track the feasibility of that.

• vmware-iso: Failed to build image using distributed switch (vSphere) #2715
• clean up orphaned vm post build using vmware-iso builder and remote_host #2841

[rhass]

➕ 💯

[marema31]

+1

I'm currently use the govc cli tools from govmomi project of vmware for two different usage in the shell-local post-processor (modifying the vmx and mark the vm as a template).

I wanted to try to create a post-processor for this two action using at least govc (like the vsphere post-processor use ovftool) or better directly the API.
I'm not a real dev (mostly sysadmin) and was scared by the idea of writing a builder for that, but the last time I tried to find the corresponding between the vmware-iso builder and govc capabilities, it seems that all actions could be done via this cli (and therefore by the API):

• Import files from local to a datastore (for the iso)
• Create a new VM
• Add parameter to vmx
• Connect/Disconnect floppy/cd rom/devices
• Boot the VM
• Get IP information (via the GuestIPHack advanced parameter)
• Enable/disable/configure VNC for the VM. (I don't know if it modify the firewall accordingly)
• Get status of VM (to wait the shutdown to complete)
• Unregister the VM (and erase data on datastore)
• Import OVA/OVF (not export it seems)
• Use the Vmware tools to import file to the guest (more for provisionner/post-processor)
• Add device (network, disk, etc...)

govc also have a debug mode really handy to determine the privileges needed by a command.

It could resolve the three issues I have with IT security guys that are not found of the usage of SSH directly to an ESXi and the fact that SSH usage implies a local account with admin role.
It may also resolve some issues with distributed vswitch (#2715)

I can help on testing some ideas if you want.

[seanmalloy]

I am also interested in helping with this. At a minimum to help with the testing. I would also like to help write the code assuming I can find the time.

[seanmalloy]

It would also be nice to add a feature to support vCenter content libraries. Content libraries are a new feature starting with vCenter version 6.0.

https://pubs.vmware.com/vsphere-65/index.jsp?topic=%2Fcom.vmware.vsphere.vm_admin.doc%2FGUID-254B2CE8-20A8-43F0-90E8-3F6776C2C896.html

[marema31]

Post of @dave2 from #3329 learn me something to be known

The API is usable only if you have a paid version of ESXi.
This post from stackoverflow give more information on that.

I suppose that for most of us that will not be an issue. But some may need to relies on the current implementation.

[marema31]

I'm trying to mockup a shell mimicking the builder with govc. I have a minimal equivalent
@mwhooker does it make sense to add this to vmware-iso builder or creating an other builder (vsphere ?) since some steps are not necessary with the API ( templating of VMX, separate creation of vmdk, folder creation are all replaced by vm.create, scp and sha1sum of iso arereplaced by datastore.upload, lookup for vnc is just a callto vm.vnc enable that return the URL with port, etc...).

[marema31]

An other solution is to do as #4591 with a new driver in vmware-common to have ability to start from iso or clone another VM.

[rickard-von-essen]

IMHO it's best to create new builders for vsphere, but you can reuse code from builders/vmware/common.

[marema31]

OK, for information, I've begun to do so. I use some part of #4591 to provide the clone capability.

[LizaTretyakova-zz]

Hi there!
I would like to repeat this comment here as well: I have a minimal working builder for creating VMs based on existing ones directly in vSphere (as previously discussed here), you can take a look at my repo. I'm now working on finishing the minimal necessary functionality and expect to be ready to open a PR in two weeks.

[lox]

Nice one @LizaTretyakova, any chance of updating the README with some install and usage instructions?

[taliesins]

Have a look at:
https://github.com/taliesins/vagrant-vsphere/blob/parallel-provisioning/lib/vSphere/driver.rb

for some hints on what can possibly be added. I think Ruby usage of API is practically the same.

[LizaTretyakova-zz]

Hi @lox, I updated the repo, now it should be better.
I also put a sample template to the README. The provisioners section there is not mandatory, it is just mentioned to show the possibility.

[LizaTretyakova-zz]

Hi again!
A quick update on the builder: cloning to alternate host, resource pool and/or datastore is ready. You can still take a look here.

[bugbuilder]

Hi everyone!,

I want to share with you a simple vSphere post-processor to be used with vmware-iso builder, this allows to mark a VM as a template (using vmware/govmomi) leaving it in a path of choice, you can find it at this repo

[rickard-von-essen]

@bugbuilder how about opening a PR with this. It's one of the most requested features.

[bugbuilder]

@rickard-von-essen sound great, I'll work on it.

[mkuzmin]

Hi

There is open PR #4882 for remote builds on VMware vSphere.
I feel very uncomfortable criticizing a work of other people, but have to rise my concerns. There are three issues with this implementation:

• The builder was bootstrapped by copy-pasting existing vmware plugin. That might help to get a working solution faster, but there are downsides too. The code inherits all the past complexity. Desktop VMware Player and clustered vCenter are different mediums with different use cases. Not all features and configuration parameters are adequate in this new environment.
• govc command-line tool is used as a driver. It adds new runtime requirements and complicates installation procedure. Because we don't have a direct access to vSphere API, it limits our ability to fix low-level issues.
• Tests are mocked. But the fact is vSphere API is crazy complicated. Unit tests are fine for checking mandatory configuration parameters, but the fictional backend will never tell you what kind of ridiculous requirement vSphere API has in runtime.

By coincidence @LizaTretyakova and me started to work on the same task at the same point of time in February-March, and now have another implementation of a vSphere plugin at https://github.com/jetbrains-infra/packer-builder-vsphere/.

• It is written from scratch, and grows organically feature by feature.
• We use native vSphere API via govmomi.
• There is a suite of integration tests on a real hardware.

At the moment it supports incremental builds only, where a base VM is already pre-created in vSphere by old vmware-iso builder and vsphere post-processor. ISO-builder and total feature parity is definitely in our plans, but I'm driven by real business cases. Even having issues and inconveniences, old vmware builder still allows us creating new VMs. This new plugin gives a solution we never had before - fast incremental builds and hierarchies of images.

My nearest priorities are:

• Windows guest support
• Moving plugin builds and tests to a public CI server
• Additional VM customization options

@marema31
Marc, I terribly sorry to say all that. Open source is intended to be a community, and I really try to collaborate where it is possible. But in this case our priorities and design decisions are so different, so I don't see a way how it can be merged.

@mwhooker and @rickard-von-essen
I would ask you to freeze the situation as it is now: do not merge #4882, and let our plugin stay in an external repository for few more months. Staying is this unofficial state allows us release often, get real user feedback sooner, experiment, and break backward compatibility until design is settled.
After that, we'll be proud to open a PR and been merged into Packer core.

In the meantime I welcome everyone to join us, and contribute.

Thank you.

[mwhooker]

@mkuzmin Thanks for your reasoned summary of the state of things. I will keep that in mind for the future. I don't have the bandwidth to evaluate the two plugins right now, but what you say rings true. I think by default we'll take your advice not to merge for now, and see how things shake out. I would love to be updated when you think your plugin is in a state where it can be merged in.

@marema31 I would be happy to get your thoughts as well. I noticed in the other ticket that you're not working on this any more as part of your job, so I'm not sure how much time you still want to dedicate to it.

If either of you would like to discuss more in private, please feel free to email [email protected]

[marema31]

@mkuzmin : You are right on the process I've taken to initialize the builder, it was also the way advised by @rickard-von-essen before on this thread. Since I'm not a seasonned developper, it was the only means for me to provides something that respect the coding philosophy of Packer. I've used the govc solution to quickly validate the architecture of the plugin, the idea was to only have to replace the driver using govc by a driver using the govmomi library (also used by govc). Since PR existed to integrate the govmomi (for post-processors), I was waiting their integration to don't have to go with this integration.

@mwhooker, effectively I'm no more using packer full time since I've changed of work. I'm now working for a vSphere-based cloud product team of a cloud provider company, and I'm currently converting them to use this useful tool. I was expecting to have time to rework more on this plugin by the end of the year.

To be honnest, I've initiated this plugin because it was something that was missing to facilate the adoption of Packer in my job and to try to give back to the community since I had never contributed on such large codebase. It was not simple for me due to my no knowledge of Golang but the code architecture allow me to have relatively quickly something that do the job. I made the PR to have a first quick review, some thoughts about my proposal and how to make it better and even works with other people on it. By the time, I've delivered it, I had nearly no responses, due to the fact the team was focused on 1.0 and I was waiting my turn. I've also done others PR that was interresting for my job, some were integrated because of their smallest size, some are still opens and wait for review, it's the normal rule and I respect it, the Packer team could not be reviewing and integrating every proposal in the minute they were done.

Long story short, I'm trying to contribute with my capacity, but if a better solution is proposed, I'd prefer that the maintainers select it. I will continue to follow the product and propose contribution if I can in my personnal time (therefore slower than before).

[ardrigh]

Is there anyone working on a possible builder solution for this?

As much as I would love to contribute one, I don't know Go code well enough to write one myself currently.

It would be helpful to have a vsphere-iso type builder using the vSphere API, and matching the vmware-iso builder functionality, which we could then use the vsphere-template post-processor on.

For our use case, we want to automate the rebuild of templates and build new templates from a Kickstart file. Then our "golden images" are two documented text files.

And then utilise the Terraform vsphere provider to provision machines from the templates Packer has built.

That workflow would simplify the maintenance of VMware images by a large factor, which I imagine would be great news to people having to do this type of work.

[marema31]

I have proposed a first draft of PR for this (#4591). I was waiting for comments on it. @mkuzmin is willing to provides another solution. I suppose packer could have a such solution in a near future.

Just my 2cts, I would love that this proposal ( #5367 (comment) ) could be integrated also in this builder.

[mkuzmin]

We have a huge progress at https://github.com/jetbrains-infra/packer-builder-vsphere
Now the plugin can create new VMs from scratch, works with clusters, supports boot_command on vSphere 6.5, can even automate macOS templates (works on MacStadium).

Also, a suite of integration tests is running on a public CI server.

[mkuzmin]

@mwhooker Hi Matthew

I think now we can start a discussion how to merge this builder into Packer repo.
I can open an empty pull request, discuss all the questions, and then push actual code there.

[marema31]

I will be happy to see this feature merged in packer.
good job 👍

[mwhooker]

Hi @mkuzmin, that's awesome! I think the best way to do this would be to open a new PR with this as an added plugin. Like you say, you can do review there. We'll also want to talk about if/how to deprecate the current builder (or parts of it).

[marema31]

Just my 2cts, be advised that since this plugin use Vsphere API that are available only on licensed ESXi (or in x days trial period), the old plugin may be always interresting for some of the current user.

[samsanthana]

Any update when this feature will be merged into packer ?

[SwampDragons]

@samsanthana I think we're still waiting on marema31 to open a PR

[marema31]

Hello,

Sorry @SwampDragons but I will not open the PR.

I had opened a PR for my proof-of-concept plugin to know if I was in the right tracks and asking for help (based on vmware packer plugin code as advised by mainteners and using a command line to interact with vSphere API to prototype) and it was never reviewed because mainteners was busy working on the 1.0.

After months mkuzmin commented my PR to explain it was not a good idea to merge it because he was part of a team that were making a better solution (the jetbrain packer plugins for vSphere) and they will create a PR when it will be more finished and tested, I then stop the work on this plugin. One year later, the plugin exist and work well with newer vSphere versions ( we use it at work), on previous post of this issue (on may 23rd), @mkuzmin was willling to be begin the merge, and @mwhooker was proposing to open a new PR, but as far I know the PR is not opened for the moment. Since their solution is effectively more accurate than mine, I've asked Packer maintainers to close my PR.

[SwampDragons]

Thank you for the explanation, and sorry for pinging you. All of this conversation happened before I joined Packer, so I probably read wrong or messed up names when coming up to speed on the issues.

@mkuzmin Are you still interested in opening a PR?

[garethmorris78]

Any updates @mkuzmin on raising a PR ? If this is a no-go then @SwampDragons can HashiCorp not push ahead with packer vSphere api integration ?

[SwampDragons]

I'll start roadmap planning assuming that we have to implement our own vSphere builder if we don't manage to get a PR from JetBrains in the next month or so.

[marema31]

Hello @SwampDragons , if I can help somehow on this process, just tell

[mkuzmin]

I honestly do not understand all this rush for PRs and official status.
Making a good and consistent product is much more important than having a thousand features.

3 years ago Terraform merged immature vSphere provider too early, to boost community activity. We suffer from this decision till today - the tool is overcomplicated and still full of bugs.

Anyway, I suppose I don't have this choice anymore, so here is #7168.
There is a huge amount of work to do this right, so please be prepared this will take 1-2 months.

[SwampDragons]

@mkuzmin if you're not ready, that's 100% fine! I am in no rush to actually merge; I just haven't actually heard from you in a while so I wasn't sure if you were even interested in still making a PR eventually.

The thing about "official" status is probably important to people because we (HashiCorp engineers) don't put our engineering time into maintaining plugins that aren't part of the Packer repo; I can understand that some people (or their companies) may not want to use a third party plugin.

The other problem I think we're facing is that with every release of esxi, VMWare seems bent on making it harder and harder to not use the api bindings, and I'm concerned that we will eventually have to EOL the other VMWare builders. As that time comes closer having some kind of roadmap (even if we're not ready to merge) is probably important so people can figure out what they want to do.

Let's plan to leave that PR open and work on it slowly together until you feel completely comfortable with its state. I don't want you to feel pressured or rushed to merge something you're not ready for.

[rismoney]

3 years ago Terraform merged immature vSphere provider too early, to boost community activity. We suffer from this decision till today - the tool is overcomplicated and still full of bugs.

no rush - community wants this right. The terraform module initial approach was faulty, through 0.4. I think that is acceptable, though not preferable. It's part of learning the space.

In early stages of product dev, a pivot is fine pre 1.0. The part that I don't really agree with, in hashicorp's dev cycles, is the main TF project. The .12 release, imho, has had a very tumultuous development cycle. Based on changelog, scope creep is a real problem. I'd recommend they move to clearly defined public PRs, instead of these waves of diffs coming from inside branches. With 220+ PRs and over 1000 issues, I believe they need some refinement in how they manage things.

[SwampDragons]

What if we do a special build where we submodule the vSphere builder so that folks don't need to deal with independent plugins? Is that the major stumbling block for people using the vSphere builder?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.