Qemu 6.2.0->7.0.0 causes colima to hard-crash M1 Macbook Pro

[peterldowns]

Describe the bug

The latest update to qemu (6.2.0 -> 7.0.0) in this commit https://github.com/NixOS/nixpkgs/commits/9d733f3d4975fb3c516a2bf2bf1bc6f50781a553 causes an M1 Macbook Pro to hard-crash (the entire computer restarts) when attempting to start a VM with colima.

Steps To Reproduce

Using an M1 Macbook Pro:

Attempt to start colima

nix-shell -p colima \
-I nixpkgs=https://github.com/NixOS/nixpkgs/archive/9d733f3d4975fb3c516a2bf2bf1bc6f50781a553.tar.gz \
--run 'colima start'

This also reproduces on the main branch of this repository, but does not reproduce on the parent commit to the 6.2.0->7.0.0 upgrade. The following command, referencing the commit from right before the upgrade, will work fine:

nix-shell -p colima \
-I nixpkgs=https://github.com/NixOS/nixpkgs/archive/b161e729ab3f7902645fa4308b87ec97515e2f25.tar.gz \
--run 'colima start'

Expected behavior

Colima should be able to start a VM without any issue.

Screenshots

No screenshot of the failure, as the computer locks up and hard-crashes.

Additional context

When the computer restarts, it prompts submission of a bug report to Apple due to the hard crash. Here is the included crash log that would be sent with that report (all generated by the computer automatically):

panic(cpu 2 caller 0xfffffe001d6e1814): vm_fault() KERN_FAILURE from guest fault on state 0xfffffe6032b30000 @sleh.c:3117 Debugger message: panic Memory ID: 0x6 OS release type: User OS version: 21E258 Kernel version: Darwin Kernel Version 21.4.0: Fri Mar 18 00:46:32 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T6000 Fileset Kernelcache UUID: 0631AF68D2B8D6FEA30E36D7895D4DB4 Kernel UUID: C342869F-FFB9-3CCE-A5A3-EA711C1E87F6 iBoot version: iBoot-7459.101.3 secure boot?: YES Paniclog version: 13 KernelCache slide: 0x0000000015cb4000 KernelCache base: 0xfffffe001ccb8000 Kernel slide: 0x0000000016464000 Kernel text base: 0xfffffe001d468000 Kernel text exec slide: 0x000000001654c000 Kernel text exec base: 0xfffffe001d550000 mach_absolute_time: 0x124be03d8d Epoch Time: sec usec Boot : 0x626ff3e3 0x000adbd7 Sleep : 0x00000000 0x00000000 Wake : 0x00000000 0x00000000 Calendar: 0x627000a6 0x000c0708

Zone info:
Foreign : 0xfffffe0025c10000 - 0xfffffe0025c20000
Native : 0xfffffe10000b8000 - 0xfffffe30000b8000
Readonly: 0xfffffe14ccd84000 - 0xfffffe166671c000
Metadata: 0xfffffe6dce89c000 - 0xfffffe6dda808000
Bitmaps : 0xfffffe6dda808000 - 0xfffffe6df220c000

CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 8 PVH locks held: None
CORE 9 PVH locks held: None
CORE 0: PC=0xfffffe001d5df418, LR=0xfffffe001d5df418, FP=0xfffffe6037f73f00
CORE 1: PC=0xfffffe001d5df418, LR=0xfffffe001d5df418, FP=0xfffffe6115603f00
CORE 2 is the one that panicked. Check the full backtrace for details.
CORE 3: PC=0xfffffe001d5df418, LR=0xfffffe001d5df418, FP=0xfffffe6115693f00
CORE 4: PC=0xfffffe001d5df418, LR=0xfffffe001d5df418, FP=0xfffffe6037c23f00
CORE 5: PC=0xfffffe001d5df418, LR=0xfffffe001d5df418, FP=0xfffffe6115663f00
CORE 6: PC=0xfffffe001d5df41c, LR=0xfffffe001d5df418, FP=0xfffffe61149ebf00
CORE 7: PC=0xfffffe001d5df418, LR=0xfffffe001d5df418, FP=0xfffffe61149bbf00
CORE 8: PC=0xfffffe001d5df41c, LR=0xfffffe001d5df418, FP=0xfffffe603290bf00
CORE 9: PC=0xfffffe001d5df41c, LR=0xfffffe001d5df418, FP=0xfffffe6114963f00
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xfffffe1ffefeed48: 18345 pages, 8 threads: pid 1696: qemu-system-aarc
Panicked thread: 0xfffffe24ce994140, backtrace: 0xfffffe6032a6b310, tid: 209694
lr: 0xfffffe001d5a9560 fp: 0xfffffe6032a6b380
lr: 0xfffffe001d5a9228 fp: 0xfffffe6032a6b3f0
lr: 0xfffffe001d6edecc fp: 0xfffffe6032a6b410
lr: 0xfffffe001d6e005c fp: 0xfffffe6032a6b480
lr: 0xfffffe001d6dda98 fp: 0xfffffe6032a6b540
lr: 0xfffffe001d5577f8 fp: 0xfffffe6032a6b550
lr: 0xfffffe001d5a8eac fp: 0xfffffe6032a6b8f0
lr: 0xfffffe001d5a8eac fp: 0xfffffe6032a6b960
lr: 0xfffffe001ddd2acc fp: 0xfffffe6032a6b980
lr: 0xfffffe001d6e1814 fp: 0xfffffe6032a6b9e0
lr: 0xfffffe001d6e13b0 fp: 0xfffffe6032a6ba50
lr: 0xfffffe001d562828 fp: 0xfffffe6032a6ba60
lr: 0xfffffe001d6f61c0 fp: 0xfffffe6032a6be50
lr: 0xfffffe001d6ddea8 fp: 0xfffffe6032a6bf10
lr: 0xfffffe001d5577f8 fp: 0xfffffe6032a6bf20

last started kext at 1353290446: com.apple.driver.driverkit.serial 6.0.0 (addr 0xfffffe001d343320, size 3416)
loaded kexts:
com.apple.filesystems.autofs 3.0
com.apple.UVCService 1
com.apple.driver.AppleTopCaseHIDEventDriver 5440.11
com.apple.driver.AppleBiometricServices 1
com.apple.driver.CoreKDL 1
com.apple.driver.usb.AppleUSBHostBillboardDevice 1.0
com.apple.driver.DiskImages.ReadWriteDiskImage 493.0.0
com.apple.driver.DiskImages.UDIFDiskImage 493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore 493.0.0
com.apple.driver.SEPHibernation 1
com.apple.driver.BCMWLANFirmware4387.Hashstore 1
com.apple.driver.AppleUSBDeviceNCM 5.0.0
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.driver.AppleFileSystemDriver 3.0.1
com.apple.nke.l2tp 1.9
com.apple.filesystems.tmpfs 1
com.apple.filesystems.lifs 1
com.apple.filesystems.apfs 1934.101.3
com.apple.IOTextEncryptionFamily 1.0.0
com.apple.filesystems.hfs.kext 583.100.10
com.apple.security.BootPolicy 1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.driver.AppleCS42L84Audio 540.34
com.apple.driver.AppleSmartIO2 1
com.apple.driver.ApplePMP 1
com.apple.driver.AppleSN012776Amp 540.34
com.apple.AppleEmbeddedSimpleSPINORFlasher 1
com.apple.driver.AppleT6000SOCTuner 1
com.apple.driver.AppleT6000CLPCv3 1
com.apple.driver.AppleSmartBatteryManager 161.0.0
com.apple.driver.AppleALSColorSensor 1.0.0d1
com.apple.driver.AppleAOPVoiceTrigger 140.1
com.apple.driver.AppleSPMIPMU 1.0.1
com.apple.driver.ApplePMPFirmware 1
com.apple.driver.AppleDPDisplayTCON 1
com.apple.driver.AppleAVD 560
com.apple.AGXG13X 190.17.3
com.apple.driver.AppleS5L8960XNCO 1
com.apple.driver.AppleSDXC 3.2.1
com.apple.driver.AppleEventLogHandler 1
com.apple.driver.AppleT6001PMGR 1
com.apple.driver.AppleAVE2 540.44.5
com.apple.driver.AppleMCDP29XXUpdateSupport 1
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AppleSerialShim 1
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AppleProResHW 175.4.0
com.apple.driver.AppleJPEGDriver 4.8.1
com.apple.driver.AppleMobileDispT600X-DCP 140.0
com.apple.driver.AppleM68Buttons 1.0.0d1
com.apple.driver.AppleS8000AES 1
com.apple.driver.AppleS8000DWI 1.0.0d1
com.apple.driver.AppleInterruptControllerV2 1.0.0d1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleBluetoothModule 1
com.apple.driver.AppleBCMWLANBusInterfacePCIe 1
com.apple.driver.AppleS5L8920XPWM 1.0.0d1
com.apple.driver.AudioDMAController-T600x 140.4
com.apple.driver.AppleT6000DART 1
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleS5L8940XI2C 1.0.0d2
com.apple.driver.AppleT6000 1
com.apple.iokit.IOUserEthernet 1.0.1
com.apple.driver.usb.AppleUSBUserHCI 1
com.apple.iokit.IOKitRegistryCompatibility 1
com.apple.iokit.EndpointSecurity 1
com.apple.driver.AppleDiskImages2 126.100.13
com.apple.AppleSystemPolicy 2.0.0
com.apple.nke.applicationfirewall 402
com.apple.kec.InvalidateHmac 1
com.apple.kec.AppleEncryptedArchive 1
com.apple.driver.driverkit.serial 6.0.0
com.apple.kext.triggers 1.0
com.apple.iokit.IOAVBFamily 1040.6
com.apple.plugin.IOgPTPPlugin 1040.3
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.driver.AppleHIDKeyboard 228.1
com.apple.driver.AppleActuatorDriver 5440.11
com.apple.driver.AppleMultitouchDriver 5440.11
com.apple.driver.AppleHSBluetoothDriver 5440.11
com.apple.driver.IOBluetoothHIDDriver 9.0.0
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily 1
com.apple.driver.usb.cdc.acm 5.0.0
com.apple.driver.usb.serial 6.0.0
com.apple.driver.usb.cdc.ecm 5.0.0
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.usb.cdc 5.0.0
com.apple.driver.AppleUSBAudio 415.11
com.apple.iokit.IOAudioFamily 340.2
com.apple.vecLib.kext 1.2.0
com.apple.driver.usb.AppleUSBXHCIPCI 1.2
com.apple.driver.AppleThunderboltPCIUpAdapter 4.1.1
com.apple.driver.AppleThunderboltDPOutAdapter 8.5.1
com.apple.driver.DiskImages.KernelBacked 493.0.0
com.apple.driver.AppleXsanScheme 3
com.apple.driver.AppleTrustedAccessory 1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.AppleSEPHDCPManager 1.0.1
com.apple.driver.usb.networking 5.0.0
com.apple.driver.AppleThunderboltDPInAdapter 8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily 8.5.1
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.nke.ppp 1.9
com.apple.driver.AppleHIDTransportSPI 5440.6
com.apple.driver.AppleHIDTransport 5440.6
com.apple.driver.AppleInputDeviceSupport 5440.6
com.apple.driver.AppleConvergedIPCOLYBTControl 1
com.apple.driver.AppleConvergedPCI 1
com.apple.driver.AppleBluetoothDebug 1
com.apple.driver.AppleBSDKextStarter 3
com.apple.filesystems.hfs.encodings.kext 1
com.apple.driver.AppleCSEmbeddedAudio 540.34
com.apple.driver.AppleDCPDPTXProxy 1.0.0
com.apple.driver.DCPDPFamilyProxy 1
com.apple.driver.AppleDiagnosticDataAccessReadOnly 1.0.0
com.apple.driver.AppleBTM 1.0.1
com.apple.driver.ApplePassthroughPPM 3.0
com.apple.driver.AppleAOPAudio 140.3
com.apple.driver.AppleEmbeddedAudio 540.34
com.apple.iokit.AppleARMIISAudio 140.2
com.apple.driver.AppleSPU 1
com.apple.AGXFirmwareKextG13XRTBuddy 190.17.3
com.apple.AGXFirmwareKextRTBuddy64 190.17.3
com.apple.iokit.IONVMeFamily 2.1.0
com.apple.driver.AppleNANDConfigAccess 1.0.0
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleStockholmControl 1.0.0
com.apple.iokit.IOGPUFamily 35.29
com.apple.driver.ApplePMGR 1
com.apple.driver.AppleHPM 3.4.4
com.apple.driver.AppleT6000TypeCPhy 1
com.apple.driver.AppleT8103TypeCPhy 1
com.apple.driver.DCPAVFamilyProxy 1
com.apple.driver.AppleSART 1
com.apple.driver.AppleUSBXDCIARM 1.0
com.apple.driver.AppleUSBXDCI 1.0
com.apple.iokit.IOUSBDeviceFamily 2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI 1
com.apple.driver.usb.AppleUSBXHCI 1.2
com.apple.driver.AppleEmbeddedUSBHost 1
com.apple.driver.usb.AppleUSBHub 1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice 1.2
com.apple.driver.AppleSPMI 1.0.1
com.apple.iokit.IOMobileGraphicsFamily-DCP 343.0.0
com.apple.driver.AppleDCP 1
com.apple.driver.AppleFirmwareKit 1
com.apple.iokit.IOMobileGraphicsFamily 343.0.0
com.apple.driver.usb.AppleUSBHostPacketFilter 1.0
com.apple.driver.AppleARMWatchdogTimer 1
com.apple.driver.AppleDisplayCrossbar 1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleTypeCPhy 1
com.apple.driver.AppleThunderboltNHI 7.2.81
com.apple.driver.AppleT6000PCIeC 1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.driver.ApplePIODMA 1
com.apple.driver.AppleT600xPCIe 1
com.apple.driver.AppleMultiFunctionManager 1
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore 1.0.0
com.apple.iokit.IO80211Family 1200.12.2b1
com.apple.driver.IOImageLoader 1.0.0
com.apple.driver.AppleOLYHAL 1
com.apple.driver.corecapture 1.0.4
com.apple.driver.AppleEmbeddedPCIE 1
com.apple.driver.AppleMCA2-T600x 640.5
com.apple.driver.AppleEmbeddedAudioLibs 140.1
com.apple.driver.AppleFirmwareUpdateKext 1
com.apple.driver.AppleH13CameraInterface 6.44.0
com.apple.driver.AppleH10PearlCameraInterface 17.0.6
com.apple.driver.AppleGPIOICController 1.0.2
com.apple.driver.AppleFireStormErrorHandler 1
com.apple.driver.AppleMobileApNonce 1
com.apple.iokit.IOTimeSyncFamily 1040.3
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily 594
com.apple.iokit.IOBluetoothSerialManager 9.0.0
com.apple.iokit.IOBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerUARTTransport 9.0.0
com.apple.iokit.IOBluetoothHostControllerTransport 9.0.0
com.apple.driver.IOBluetoothHostControllerPCIeTransport 9.0.0
com.apple.iokit.IOBluetoothFamily 9.0.0
com.apple.driver.FairPlayIOKit 68.16.0
com.apple.iokit.CSRBluetoothHostControllerUSBTransport 9.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 9.0.0
com.apple.driver.AppleSSE 1.0
com.apple.driver.AppleSEPKeyStore 2
com.apple.driver.AppleUSBTDM 533.100.11
com.apple.iokit.IOUSBMassStorageDriver 210.101.2
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOSCSIBlockCommandsDevice 456.100.7
com.apple.iokit.IOSCSIArchitectureModelFamily 456.100.7
com.apple.driver.AppleIPAppender 1.0
com.apple.driver.AppleFDEKeyStore 28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost 1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties 1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC 3.1.9
com.apple.driver.RTBuddy 1.0.0
com.apple.driver.AppleEmbeddedTempSensor 1.0.0
com.apple.driver.AppleARMPMU 1.0
com.apple.iokit.IOAccessoryManager 1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient 1.0.1b8
com.apple.iokit.IONetworkingFamily 3.4
com.apple.iokit.IOSerialFamily 11
com.apple.driver.AppleSEPManager 1.0.1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor 1
com.apple.driver.AppleBiometricSensor 2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.driver.AppleANELoadBalancer 5.64.2
com.apple.driver.AppleH11ANEInterface 5.64.2
com.apple.AUC 1.0
com.apple.iokit.IOAVFamily 1.0.0
com.apple.iokit.IOHDCPFamily 1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family 1.0
com.apple.driver.AppleIISController 140.1
com.apple.driver.AppleAudioClockLibs 140.1
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.iokit.IOSurface 302.14
com.apple.driver.IODARTFamily 1
com.apple.security.quarantine 4
com.apple.security.sandbox 300.0
com.apple.kext.AppleMatch 1.0.0d1
com.apple.driver.AppleMobileFileIntegrity 1.0.5
com.apple.security.AppleImage4 4.2.0
com.apple.kext.CoreTrust 1
com.apple.iokit.IOCryptoAcceleratorFamily 1.0.1
com.apple.driver.AppleARMPlatform 1.0.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOSlowAdaptiveClockingFamily 1.0.0
com.apple.iokit.IOReportFamily 47
com.apple.kec.pthread 1
com.apple.kec.Libm 1
com.apple.kec.corecrypto 12.0

** Stackshot Succeeded ** Bytes Traced 408913 (Uncompressed 1070448) **

Notify maintainers

@edolstra @alyssais

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-darwin"`
 - host os: `Darwin 21.4.0, macOS 12.3.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.7.0`
 - channels(pld): `"darwin"`
 - channels(root): `"nixpkgs-22.05pre360843.3eb07eeafb5"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs
output here

[patryk4815]

https://gitlab.com/qemu-project/qemu/-/issues/903#note_911000975
Please check in macOS 12.4 Beta1 if still crashing.

[peterldowns]

Thank you for linking that issue, I hadn't thought to search the qemu issue tracker 🤦 . I unfortunately cannot upgrade my company-managed computer to the beta version of macOS, hopefully they'll release 12.4 with that fix soon.

A question that is maybe better suited for the nix maintainers: for my own information / in the meantime, how does Nix normally deal with platform specific issues like this? Is there any recommended workaround or tactic that I can use to update our company's shared darwin-configuration.nix to use a colima derivation that links against the older version of qemu? Right now we're simply pinning nixpkgs-unstable to the last-known-good commit hash for qemu. I haven't contributed to nixpkgs before, would it be worth the time or even possible to update the colima/lima derivations to depend on qemu 6.2.x for darwin users?

If you don't have time to respond to these questions I understand, thank you for all of the work you do to maintain Nix, really appreciate it.

[patryk4815]

you can create overlay something like:

    pkgsOld = import (fetchTarball https://github.com/NixOS/nixpkgs/archive/9d733f3d4975fb3c516a2bf2bf1bc6f50781a553.tar.gz) { };
    pkgs = import (fetchTarball https://github.com/NixOS/nixpkgs/archive/master.tar.gz) {
        overlays = [
            (self: super: rec {
                qemu = pkgsOld.qemu;
            })
        ];
    };

[patryk4815]

@peterldowns Looks like you can try use limactl lima-vm/lima#795 they fixed it somehow :)

[peterldowns]

I was just about to write back in -- looks like this is fixed in lima v0.10.0, which is already released on nixpkgs master! I have tested and resolved my issue by no longer installing colima from unstable.

Unstable:

• colima 0.3.4
• lima 0.9.2

Master:

• colima 0.3.4
• lima 0.10.0

Previous darwin configuration:

let
  unstable = import
    (builtins.fetchTarball "channel:nixos-unstable")
    { config = config.nixpkgs.config; };
in
# ...
unstable.colima
# ...

Fixed by updating to:

# ... no need to import unstable anymore
pkgs.colima
# ...

I wouldn't have expected unstable to be on a lower version the master, but hey, easy fix :) Thanks for the help!

[patryk4815]

🤔 I think we could just upgrade https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/applications/virtualization/lima/default.nix#L11 and it should solve your issue yes

[peterldowns]

Yes, I believe all that is needed is for unstable to take the latest v0.10.0 version of lima, as has already been done on the master branch, since using the master branch works correctly for me. I am not sure how to make that change myself but if any maintainers read this, please feel free to close this issue.

[zowoq]

I wouldn't have expected unstable to be on a lower version the master

We do have a proper doc with diagrams as to how the various branches and channels work somewhere in the repo but I couldn't find it ...

TLDR: changes go to master first and then they get pushed to each channel automatically.

0.10.0 is already in nixpkgs-unstable and nixos-unstable-small, it'll be in nixos-unstable soon.

https://nixpk.gs/pr-tracker.html?pr=170608

[peterldowns]

Thank you, that's very useful to know, and I'll keep that pr-tracker handy for the future :)