Apple Silicon (arm64) - what should I do with VirtualBox-based examples?

[geerlingguy]

I'm considering maybe changing everything to libvirt, since I believe I could make the examples cross-platform that way (Linux, macOS M1/Intel, and Windows) like I could recommending VirtualBox.

Also, I have an M1 Mac now.

Requirements:

• Be able to run arbitrary Ubuntu (latest LTS) and maybe Rocky Linux 8 virtualized instances
• Be able to assign an IP address to that instance (ideally ... some examples don't need it)
• Be able to run multiple instances at the same time, and have them be able to communicate with each other via hostname or IP address (ideally).

I also saw UTM, but it seems like it might not be as focused on the programmatic Vagrant-based use case (infrastructure as code) I use here.

Another option is to go to the cloud, but I don't like telling people to do things that could cost them money.

Update: Here are some of the solutions I've been trying out on my Mac:

• UTM
• Tart
• Lima
• OrbStack

[geerlingguy]

Throwing a few notes in here:

• UTM can set up a bridged network (Bridge Networking Support utmapp/UTM#358), but other networking support could be... weird (see Configure shared networking IP address utmapp/UTM#3294).
• There's an open but unworked feature request for Vagrant support for UTM (Provider for UTM (VM for M1 processors) hashicorp/vagrant#12518) — and there's also no direct support for QEMU, and the libvirt API compatibility layer with QEMU doesn't work on Apple Silicon yet either.
• Slightly-related: Trellis and Apple Silicon roots/trellis#1253 — Trellis seems to be recommending Parallels, but I'd like to have a solution that doesn't require any paid software.

Basically, this is probably fairly niche since many infrastructure automation folks run Linux on their workstation (so all this is a non-issue), but I still wonder—for those who do need to emulate 'real' hosts on their M1 Macs... what are they doing currently? I am using Docker for everything else at this point, but there are times when I want to control four isolated VMs accessible to each other only via IP.

[o0-o]

A (the?) UTM dev created an issue to track Vagrant integration the other day so it seems like there is some willingness to make that happen. Is anyone seriously working on libvirt for this? In my experience, it's been a second class citizen as both a Vagrant provider and macOS application (correct me if I'm wrong).

[danielporto]

that's actually very interesting use case. I'm thinking about having a macmini to deploy several services and woudl like to have a way to control vms remotely. so far, there is no tool, web interface that allow me to do that. and I must open a remote desktop into the "server" to boot up a vm.
I could replace the OS for linux, sure, but what if I want a M1 mac mini? Linux support is not quite there yet.

[zmingxie]

Related vagrant-libvirt is currently having installation issues with M1 mac because of confusing ruby native packages: vagrant-libvirt/vagrant-libvirt#1205

[trinitronx]

Related vagrant-libvirt is currently having installation issues with M1 mac because of confusing ruby native packages: vagrant-libvirt/vagrant-libvirt#1205

Yes, this is still correct as of today when using officially released vagrant and plugin gems.

It's definitely possible to build all the pieces from source and get a native arm64 Vagrant + vagrant-libvirt stack working on an M1 macBook. It's just a lot of extra work at the moment.

[kamidev]

Thanks a lot @trinitronx! Great to know this is already possible.

[johalun]

There's also this one https://github.com/ppggff/vagrant-qemu that I've played around with a bit for running x86_64 Linux VMs on Apple Silicon.
I don't know how feature rich it is but it would be nice to have a pure qemu provider that is 100% cross platform.

[jmgilman]

So has anyone found a decent solution for this? The closest I've been able to get is Multipass, but it's not very customizable and heavily geared towards Ubuntu (for obvious reasons). The best solution I've come across is Parallels but, as noted, it's not free software.

[Nurmukhamed]

Hello.
For book examples I used terraform, because Vagrant is slow and old software.
You may look examples here - https://github.com/Nurmukhamed/AnsibleForDevops

[geerlingguy]

I recently became aware of Tart, which instead of QEMU uses Apple's native Virtualization framework. It seems to run very well on my M1/M2 Macs, and is built more for CI/scripted use than apps like UTM, Parallels, and Fusion.

There is currently an issue requesting Linux support, and I would be keen to test that—even if some manual tart commands would have to be run prior to running an Ansible playbook. I've already tested my macOS-oriented mac-dev-playbook with Tart, and it works great: geerlingguy/mac-dev-playbook#160 (comment)

There's also an issue requesting a Vagrant plugin for Tart.

[timharek]

Any updates on this? I just bought the book and was excitied to try out the testing with local VMs 😅

[johalun]

For emulation I think UTM does a good job. They recently got CLI support for simple control of VMs.

utmapp/UTM#4805

[cruzzoe]

Adding another reminder this would be great to have. Bought the book and attempting to run on m1 but hitting the first hurdle when trying to use vagrant.

I saw its recommended for us to use aws or other clouds however a lot of the steps in the book set everything up via vagrant. Am I missing something or do we need to carry out these steps manually?

Thanks Jeff for all your work /YouTube vids, its very much appreciated! Wishing you well.

[o0-o]

Adding another reminder this would be great to have. Bought the book and attempting to run on m1 but hitting the first hurdle when trying to use vagrant.

I saw its recommended for us to use aws or other clouds however a lot of the steps in the book set everything up via vagrant. Am I missing something or do we need to carry out these steps manually?

Thanks Jeff for all your work /YouTube vids, its very much appreciated! Wishing you well.

Have you tried VMWare Fusion? There is a free tier and you can use it as a Vagrant provider on your M1 Mac.

https://gist.github.com/sbailliez/2305d831ebcf56094fd432a8717bed93

[mahdihijazi]

Adding another reminder this would be great to have. Bought the book and attempting to run on m1 but hitting the first hurdle when trying to use vagrant.

I saw its recommended for us to use aws or other clouds however a lot of the steps in the book set everything up via vagrant. Am I missing something or do we need to carry out these steps manually?

Thanks Jeff for all your work /YouTube vids, its very much appreciated! Wishing you well.

I second this, I am stuck on moving with the examples. Would be nice if someone can provide a working hack until this gets resolved. Thanks!

[o0-o]

Unfortunately, there is no simple solution or "hack". Jeff will likely need to release a new edition of his book and either embrace tart, docker, VMWare or something else, but none are drop-in solutions for Virtual Box. The Vagrant boxes themselves are also x86.

The best solution in my opinion would be to use libvirt since it's open source and potentially cross-platform (assuming WSL2), but unfortunately, its network driver is broken on macOS and has been for quite a while.

Here is the relevant issue for libvirt: https://gitlab.com/libvirt/libvirt/-/issues/75

[geerlingguy]

The best hack IMO is to get an account on Linode or DigitalOcean and spin up VMs to test there—at least that would universally work... The second best is to use some other local virtualization. Tart is what I've tested the most on my Mac, and it seems like it could be the solution, but nothing is really tightly integrated with Vagrant nowadays so I can have it auto assign an IP address and optionally run Ansible playbooks directly as part of vagrant provision :(

[e-minguez]

UTM added some preliminary scripting support like https://github.com/suse-edge/misc/blob/main/slemicro/create_vm.sh#L85-L132 but there is still room for improvement (this one for example utmapp/UTM#3294)

[FreshPrinceMayo]

The best hack IMO is to get an account on Linode or DigitalOcean and spin up VMs to test there—at least that would universally work... The second best is to use some other local virtualization. Tart is what I've tested the most on my Mac, and it seems like it could be the solution, but nothing is really tightly integrated with Vagrant nowadays so I can have it auto assign an IP address and optionally run Ansible playbooks directly as part of vagrant provision :(

There appears to be an issue with the digital ocean plugin on Vagrant 2.3.6

Issue: devopsgroup-io/vagrant-digitalocean#298
Pending pull-request fix: devopsgroup-io/vagrant-digitalocean#300

[gclawes]

Seems like the developer preview w/ support for Apple Silicon has been removed from VirtualBox downloads: https://www.virtualbox.org/wiki/Downloads

https://youtube.com/watch?v=obo1ksQleZo

[billsidea]

I was also stuck with the Vagrant/VirtualBox examples because I have an M1 Mac. For now, I just bought a cheap NUC for $89 and loaded Debian 12 on it and it works great with the Vagrant/VirtualBox examples in this book. It came with Ubuntu installed but I was having weird issues with .local references so I just installed Debian 12 and all is well.

I realize spending $89 is not for everyone, but I actually like having another device to play with.

[Edit: I bought the one with the A9-9400 AMD processor and 8GB RAM and it had a $10 coupon.]

[geerlingguy]

Going to test:

• UTM
• Tart
• Lima

My main criteria for each one, basically can I:

• Create a VM with a command / something repeatable
• Have it reachable on an IP address so I can ssh user@ip from the Mac locally
• Easily get Linux VMs up and running on it (Debian or Ubuntu at first, but maybe others)

It'd be even more cool if there were any tool that were cross platform between Windows and Mac (and Linux) without requiring command line use, but that seems not to be the case.

[geerlingguy]

UTM

Installation / setup (assumes you have an Ubuntu Server for Arm install ISO):

1. brew install --cask utm

2. sudo ln -sf /Applications/UTM.app/Contents/MacOS/utmctl /usr/local/bin/utmctl

3. Open UTM, create a virtual machine via the GUI

4. Select the Ubuntu server for arm64 ISO when it asks for an ISO, then boot from it and run through Ubuntu Server's install wizard.

5. Make sure you install SSH and configure your SSH keys or password authentication during setup.

6. If you want certain utmctl options to work (like ip-address, install and start qemu-guest-agent:

   sudo apt install -y qemu-guest-agent
   sudo systemctl enable qemu-guest-agent
   sudo systemctl start qemu-guest-agent
   

Connecting via SSH:

First, get your VM's IP address:

# From your Mac, assuming you have `qemu-guest-agent` on the VM:
utmctl ip-address [VM name here]

# Or inside the VM, run this:
ip a

Then connect from macOS:

ssh username@[ip address]

My VM had an IP address 192.168.64.3.

You can use utmctl to semi-headlessly start and stop a VM:

utmctl list  # show a list of all available VMs
utmctl start [VM name here]  # start VM
utmctl stop [VM name here]  # stop VM

Tart

Installation / setup:

1. brew install cirruslabs/cli/tart
2. tart clone ghcr.io/cirruslabs/ubuntu:20.04 ubuntu
3. Wait for the download to complete, then tart run ubuntu

Alternatively, you can install your own version of Linux if you have an arm64 ISO (like Ubuntu Server for arm64):

1. tart create --linux ubuntu
2. tart run --disk ~/Downloads/ubuntu-22.04.3-live-server-arm64.iso ubuntu
3. Install Ubuntu using the install wizard via Tart's GUI.

Note: I was unable to get the install wizard to complete on a slow hotel WiFi connection. Will try again later. There is also a Packer plugin for building Tart VMs.

Connecting via SSH:

# Run the following on the Tart ubuntu VM
sudo apt update
sudo apt install -y openssh-server
sudo ufw allow ssh

Then connect from macOS:

ssh TODO

Note: I could not get Ubuntu Server 22.04 to install (at least not with the unstable hotel Internet where I was), so I'll try again later. Also, I opened an issue: Consider hosting official Linux images?.

Lima

Installation / setup:

1. brew install lima
2. limactl create --name=test --cpus=2 --memory=4 template://ubuntu --tty=false
3. limactl start test

Connecting via SSH:

# Get the SSH connection info.
limactl show-ssh test

Then connect from macOS using at minimum the -o Port=[port number] returned from the above output:

ssh -p 64984 127.0.0.1

I am trying to see if there's some way of forcing an IP address for a Lima VM.

[zyga]

vmware fusion has pretty good management APIs and performance.

[geerlingguy]

@zyga - While true, I don't want to tell everyone to buy a licensed bit of software for testing, when so many free alternatives exist :)

[tmeckel]

@geerlingguy because I'm thinking about buying a M2 MacBook: did you try out running a x64 based Ubuntu with UTM?

[johalun]

@geerlingguy because I'm thinking about buying a M2 MacBook: did you try out running a x64 based Ubuntu with UTM?

I have on M1 Pro and x86 emulation is pretty painfully slow, even with all the bells and whistles like virtio storage, forced multithreading, etc. Not something you wanna spend your days coding and compiling in. I ended up using an old x86 laptop as headless Ubuntu server that I ssh'd into instead.

Edit: maybe there's a way to utilize Rosetta2 for speedups? I know there is for individual binaries, but a whole VM?

[WMRamadan]

I have tried Multipass by Canonical on Linux(Ubuntu), macOS(Intel) and macOS(Apple Silicon), it's pretty good and easy to use!

@geerlingguy Meets all your criteria expect that you can only run Ubuntu distro's on it.

1 - You can create a VM with a command.
2 - Have it reachable on an IP address so you can ssh user@ip locally
3 - Easily get Linux VMs up and running on it
4 - It's Cross platform and available for Linux, Mac and Windows

[gnattu]

Edit: maybe there's a way to utilize Rosetta2 for speedups? I know there is for individual binaries, but a whole VM?

@johalun Theoretically, you can use rosetta2 for the whole userland and only using the virtualized kernel in arm64, and this would bring you an experience almost like you are using an x86-64 VM because most of time you don't play with the kernel directly. You can try Orbstack which allows creating machines like this.

Just a couple of notes that the difference between Orbstack and other traditional VMs:

• OrbStack is using its own modified kernel to provided better performance and unique features, you cannot use your own kernel and bring your own iso at the moment, but it does come with lots of distros.
• All OrbStack Machines are technically running on one VM sharing the same kernel and the memory limit. So they are not real "VMs" but more likely containers. For development purpose this does not change the user experience much, but only to remember that your memory allocated to VM does not return even if you shutdown all your OrbStack machines(due to a macOS bug), so if you run low on memory, you should stop the whole OrbStack engine.

@geerlingguy All others are quite good. And I highly recommend OrbStack for developing purpose.

It does come with a CLI interface, creating a "VM" is as simple as orb create ubuntu my-machine It has automatic ip and domain management, so after creation, you can just access the machine with machine-name.orb.local, and in this example, it would be my-machine.orb.local. No more ip address guessing/extracting needed. Only two commands and you will be sitting in your machine, no GUI install process needed. And you don't have to use ssh, you can just use the command orb to attach to the default machine's shell or use orb -m my-machine to specify one.
If you want to connect to host inside the machine, then you should use host.orb.internal.

Highly recommend for this use case if you don't mind you are not using a real "VM" and cannot customize your kernel.

Edit: Oh forget to mention. There is one quirk for OrbStack. The default config is to enable Rosetta for the OrbStack engine, and that will reduce performance for everything, including native arm64 programs. So if you don't need to use x86-64 programs, I highly recommend you to disable the use Rosetta option for better performance, especially if you will do SIMD-intensive works.

[e-minguez]

UTM cli is pretty limited for now, however it supports Apple Script (that can be embedded into a script easily) where you can do much more. See https://github.com/suse-edge/misc/blob/main/slemicro/create_vm.sh#L202-L220 for example where we create a custom VM with custom CPU/memory/disk/whatever. HTH.

[bparanj]

I found this very useful: https://github.com/spurin/diveintoansible-lab

I was able to setup multiple Ubuntu and CentOS images using it. Just run docker compose up and the environment is ready for running playbooks. It also a nice UI you can access locally at port 1000.

[bibekme]

I've been using Multipass to create VMs and update the playbooks as needed. It's been great so far, but I'd say some manual work is still required.

[curtisspendlove]

@zyga - While true, I don't want to tell everyone to buy a licensed bit of software for testing, when so many free alternatives exist :)

I'm curious if the recent "personal license" free announcement changes this a bit...

I'll be investigating a bit this weekend to see if I can get local VMs back as part of my dev workflow.

It would be fantastic.

https://blogs.vmware.com/teamfusion/2024/05/fusion-pro-now-available-free-for-personal-use.html

[geerlingguy]

Good news... It seems that UTM now has plugins for both Vagrant and Packer! I need to see if IP addressing and Ansible stuff can be tied in, but if so—that could be an avenue to keep things mostly the same, and more easily manage VMs on Macs for testing!

hashicorp/vagrant#12518 (comment)

[naveenrajm7]

Hello @geerlingguy,

Thank you for noticing the Vagrant UTM Plugin! I’m the developer of the plugin and wanted to let you know that since the plugin passes ssh_info to Vagrant, you can use any provisioners that Vagrant supports, including Ansible. Regarding IP addressing, currently, there is no straightforward way of forcing UTM to pick the IP address we want.

If you give it a try, I’d love to hear your feedback.

[o0-o]

The generic images have also begun supporting a qemu plugin that was introduced at some point and many have arm64 variations now as well.

I am not sure how well supported qemu is in macOS. Most googling brings up results pertaining to running macOS in a qemu VM and not the other way around.

If qemu is viable on macOS, that would be the most cross-compatible Vagrant plugin.

[curtisspendlove]

We seem to slowly be "getting there" in regard to having a clean environment to run Ansible in again.

I'm mostly happy with the items mentioned here in the thread, including UTM and even the "developer" licensed VMWare Fusion Pro.

I'm also wanting to play around with this project a bit in the near future:

https://github.com/quickemu-project/quickemu

BUT, honestly...plain old Docker has been pretty good as well.

I've currently been developing a couple Ruby on Rails apps that use Terraform and Docker. You can basically run a Docker Dev Environment locally and use common CI/CD tooling to push a docker-compose cluster to a Terraform(ed) environment (I'm using kamal).

It's hard to decide what the best, current alternative is still.

I was somewhat inspired by the Rails core team embracing Docker in Rails 7.x, including default Dockerfile and Dev Containers.

https://edgeguides.rubyonrails.org/getting_started_with_devcontainer.html

But that will obviously work in slightly more "dockerized" environments and be less useful in environments that are traditionally more tricky to dockerize.