[Bug] UEFI mismatch results in boot loop

[m-ildefons]

During installation sometimes seeder fails to re-boot.

The installation process should be:

1. Boot installer ISO image via PXE
2. Create filesystems, bootloader, , preload images, copy files
3. Re-boot via PXE, chainload bootloader from hard-disk <---- this fails, resulting in a boot-loop
4. Finalize setup

This fails sometimes during stage 3 with an ominous UEFI mismatch error in seeder-boots logs and the re-boot failing due to a nonexistent file on the tftp-server

The server in question is set up to boot in BIOS mode in the order:

• PXE boot from network
• Boot from CD/DVD
• Boot from HDD/SSD

A CD/DVD is not attached at the time of re-booting.
This behavior is inconsistent and occurs with Harvester settings that have worked in the past. The behavior occurs with various versions of the Harvester installer: v1.2.1, v1.3.0-rc2, v1.3.0-rc3

seeder-boots logs (relevant excerpt):

[...]
{"level":"info","ts":1708947585.588214,"caller":"[email protected]/handler.go:105","msg":"","service":"github.com/tinkerbell/boots","pkg":"dhcp","pkg":"dhcp","event":"recv","mac":"8c:dc:d4:b5:f2:39","via":"0.0.0.0","iface":"mgmt-br","xid":"\"d4:b5:f2:39\"","type":"DHCPREQUEST"}
{"level":"info","ts":1708947585.58826,"caller":"boots/dhcp.go:88","msg":"parsed option82/circuitid","service":"github.com/tinkerbell/boots","pkg":"main","mac":"8c:dc:d4:b5:f2:39","circuitID":""}
{"level":"info","ts":1708947585.5883825,"caller":"job/dhcp.go:80","msg":"uefi mismatch, using dhcp","service":"github.com/tinkerbell/boots","mac":"8c:dc:d4:b5:f2:39","hardware.id":"","instance.id":"","dhcp":false,"job":true}
{"level":"info","ts":1708947585.5899632,"caller":"[email protected]/handler.go:61","msg":"","service":"github.com/tinkerbell/boots","pkg":"dhcp","pkg":"dhcp","event":"send","mac":"8c:dc:d4:b5:f2:39","dst":"255.255.255.255","iface":"mgmt-br","xid":"\"d4:b5:f2:39\"","type":"DHCPACK","address":"172.19.108.40","next_server":"172.19.104.28","filename":"undionly.kpxe"}
{"level":"info","ts":1708947588.9981601,"caller":"[email protected]/handler.go:105","msg":"","service":"github.com/tinkerbell/boots","pkg":"dhcp","pkg":"dhcp","event":"recv","mac":"8c:dc:d4:b5:f2:39","via":"0.0.0.0","iface":"mgmt-br","xid":"\"a4:b2:5c:3f\"","type":"DHCPDISCOVER","secs":4}
{"level":"info","ts":1708947588.9982717,"caller":"boots/dhcp.go:88","msg":"parsed option82/circuitid","service":"github.com/tinkerbell/boots","pkg":"main","mac":"8c:dc:d4:b5:f2:39","circuitID":""}
{"level":"info","ts":1708947588.9984965,"caller":"job/dhcp.go:80","msg":"uefi mismatch, using dhcp","service":"github.com/tinkerbell/boots","mac":"8c:dc:d4:b5:f2:39","hardware.id":"","instance.id":"","dhcp":false,"job":true}
{"level":"info","ts":1708947588.9985657,"caller":"job/dhcp.go:164","msg":"","service":"github.com/tinkerbell/boots","mac":"8c:dc:d4:b5:f2:39","hardware.id":"","instance.id":"","instance.state":"","os_slug":"","os_distro":"harvester","os_version":"v1.2.1"}
{"level":"info","ts":1708947589.0012038,"caller":"[email protected]/handler.go:61","msg":"","service":"github.com/tinkerbell/boots","pkg":"dhcp","pkg":"dhcp","event":"send","mac":"8c:dc:d4:b5:f2:39","dst":"255.255.255.255","iface":"mgmt-br","xid":"\"a4:b2:5c:3f\"","type":"DHCPOFFER","address":"172.19.108.40","next_server":"172.19.104.28","filename":"nonexistent"}
{"level":"info","ts":1708947589.0032737,"caller":"[email protected]/handler.go:105","msg":"","service":"github.com/tinkerbell/boots","pkg":"dhcp","pkg":"dhcp","event":"recv","mac":"8c:dc:d4:b5:f2:39","via":"0.0.0.0","iface":"mgmt-br","xid":"\"a4:b2:5c:3f\"","type":"DHCPREQUEST","secs":10}
{"level":"info","ts":1708947589.0033216,"caller":"boots/dhcp.go:88","msg":"parsed option82/circuitid","service":"github.com/tinkerbell/boots","pkg":"main","mac":"8c:dc:d4:b5:f2:39","circuitID":""}
{"level":"info","ts":1708947589.0034354,"caller":"job/dhcp.go:80","msg":"uefi mismatch, using dhcp","service":"github.com/tinkerbell/boots","mac":"8c:dc:d4:b5:f2:39","hardware.id":"","instance.id":"","dhcp":false,"job":true}
{"level":"info","ts":1708947589.0034606,"caller":"job/dhcp.go:164","msg":"","service":"github.com/tinkerbell/boots","mac":"8c:dc:d4:b5:f2:39","hardware.id":"","instance.id":"","instance.state":"","os_slug":"","os_distro":"harvester","os_version":"v1.2.1"}
{"level":"info","ts":1708947589.0050242,"caller":"[email protected]/handler.go:61","msg":"","service":"github.com/tinkerbell/boots","pkg":"dhcp","pkg":"dhcp","event":"send","mac":"8c:dc:d4:b5:f2:39","dst":"255.255.255.255","iface":"mgmt-br","xid":"\"a4:b2:5c:3f\"","type":"DHCPACK","address":"172.19.108.40","next_server":"172.19.104.28","filename":"nonexistent"}
[...]

Screenshot of the HTML5 console of the affected server: