Floppy gets overwritten with multiple parallel runs

[agowa]

Overview of the Issue

When two or more packer instances are executed simultaneously for different projects against the same vsphere server the floppy file gets overwritten.

Reproduction Steps

1. Create a simple packer project for a deployment that uses a floppy
2. Copy this project and rename the vm in the copy to get two independent projects
3. Start deploy both projects simultaneously
4. Now the build may fail, disk gets removed while mid-deployment. Also when looking into vsphere logs it can be observed that the floppy files are used by something else (aka. that they're locked)

Description:
07/30/2021, 3:36:55 PM Warning message on Win server 2016 standard template Core (english) - latest on <SERVER> in <DC> : Cannot open floppy image packer-tmp-created-floppy.flp: 4 (Interrupted system call)
Event Type Description:
A warning message listing a collection of observations has been reported by the virtual machine
Possible Causes:
The event contains details on why this warning was issued
Related events:
There are no related events.

as well as

Description:
07/30/2021, 3:36:55 PM Warning message on Win server 2016 standard template Core (english) - latest on <SERVER> in <DC> : Could not connect to floppy "packer-tmp-created-floppy.flp". It is being used by another virtual machine or some other program. It will start disconnected.
Event Type Description:
A warning message listing a collection of observations has been reported by the virtual machine
Possible Causes:
The event contains details on why this warning was issued
Related events:
There are no related events.

Plugin and Packer version

Packer v1.7.4

Simplified Packer Buildfile

Operating system and Environment details

OS, Architecture, and any other information you can provide about the
environment.
GitLab CI pipeline

image: ubuntu:latest

stages:
  - build

before_script:
  - rm -f /etc/apt/apt.conf.d/docker-clean  # Re-enable apt cache
  - apt update
  - apt install -y wget ca-certificates gzip tar openssl lsb-release gnupg
  - wget -q -O - https://apt.releases.hashicorp.com/gpg | apt-key add -
  - echo "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" >> /etc/apt/sources.list
  - apt update
  - apt install packer

build_vsphere_2019_standard_core_images: &build_vsphere_2019_standard_core_images
  stage: build
  script:
    - packer init -upgrade win_2019_standard_core.pkr.hcl
    - packer build -force -only='vsphere-iso.*' -timestamp-ui win_2019_standard_core.pkr.hcl
  only:
    - master
  environment:
    name: production

build_vsphere_2019_standard_gui_images: &build_vsphere_2019_standard_gui_images
  stage: build
  script:
    - packer init -upgrade win_2019_standard_gui.pkr.hcl
    - packer build -force -only='vsphere-iso.*' -timestamp-ui win_2019_standard_gui.pkr.hcl
  only:
    - master
  environment:
    name: production

Log Fragments and crash.log files

Packer doesn't crash, building the VM fails (race conditions, see above).

[tenthirtyam]

Out of curiosity, do you also observe the issue occur if your Packer template has > 1 source and build or only if running parallel executions?

For example, does the issue occur if you have the following in a template:

Snip....

build {
    sources = [
      "source.vsphere-iso.windows-server-standard-dexp",
      "source.vsphere-iso.windows-server-standard-core",
      "source.vsphere-iso.windows-server-datacenter-dexp",
      "source.vsphere-iso.windows-server-datacenter-core"
      ]

And then you run build all four at the same time...

packer build -force -var-file="vsphere.pkrvars.hcl" .

Ryan

[agowa]

@tenthirtyam The above was just an example, as we have multiple teams building different images for different projects and operating systems. Combining all of them into a single git repository is just not practical as they all have different settings.

[tenthirtyam]

@agowa338 I assume then that this is has not been observed with a single execution with multiple source/builds in a single template, but only observed during parallel executions of Packer to the same vSphere endpoint?

Ryan

[agowa]

yes, only when multiple parallel executions of Packer to the same vSphere endpoint run.

[tenthirtyam]

I'm unable to reproduce this in my environments.

packer {
  required_version = ">= 1.7.7"
  required_plugins {
    vsphere = {
      version = ">= v1.0.2"
      source  = "github.com/hashicorp/vsphere"
    }
  }
}

Host: Ubuntu Server 20.04 LTS
Method: Ubuntu server using floppy_content (new in v1.7.7)
Naming: vmname+ local.buildstamp

The build were started 2s apart:

Online and Ansible engaged....

Builds completed w/o error.

Note: skipping import to content library (new in 1.7.7) for testing purposes.

Ryan

[tenthirtyam]

The changes proposed in #399 may allow for this to be avoided by allowing for isolated remote caches.

[agowa]

The simplest solution to avoid this overwriting issue is probably to add a runtime generated uuid or a sha256 hash (that dependent on the virtual floppies content) to the filename.

Would adding something like this (just to be extra safe) work with your proposal? @tenthirtyam

[tenthirtyam]

There's another change I've already prepared for similar that will ensure additional coverage.

[tenthirtyam]

I plan to implement this with:

import

github.com/hashicorp/packer-plugin-sdk/uuid

where uuid.TimeOrderedUUID is appended in the file name. This is exactly how the sdk generates the names for ISOs it creates as packer-<uuid.TimeOrderedUUID>.iso.

Here we can generate a similar name like packer-<uuid.TimeOrderedUUID>.flp.

Should be in this week.

[tenthirtyam]

Looking at this a little more, the SDK is actually using its own tmp package by using "packer-*.iso".

https://github.com/hashicorp/packer-plugin-sdk/blob/main/tmp/tmp.go

Either way, this can be updated to generate a unique name in the same format but with the .flp extension.

[agowa]

But wouldn't TimeOrderedUUID still pose a significant risk of collisions? E.g. when there are multiple CI jobs getting triggered simultaneously?

Also if the unique id is content dependent we can also detect if it already exists and overwrite (avoid cluttering the datastore in case the final deletion failed in former runs).

[tenthirtyam]

PR #403