v0.5.9-rc2 breaks tar2files (ldd) targets on crossbuilds

[jschintag]

When upgrading to bazeldnf v0.5.9-rc2 cross building for different arches fails to find included header files.
I specifically tested this when building kubevirt with kubevirt/kubevirt#10490:

INFO: From ImageLayer cmd/libguestfs/version-container-layer.tar:
Duplicate file in archive: ./etc/passwd, picking first occurrence
ERROR: /root/go/src/kubevirt.io/kubevirt/vendor/libvirt.org/go/libvirt/BUILD.bazel:3:11: GoCompilePkg vendor/libvirt.org/go/libvirt/go_default_library.a failed: (Exit 1): builder failed: error executing command bazel-out/k8-opt-exec-2B5CBBC6/bin/external/go_sdk/builder_reset/builder compilepkg -sdk external/go_sdk -installsuffix linux_s390x -tags selinux,selinux -src vendor/libvirt.org/go/libvirt/callbacks.go ... (remaining 214 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
In file included from vendor/libvirt.org/go/libvirt/connect_helper.h:30,
                 from kubevirt/vendor/libvirt.org/go/libvirt/connect.go:42:
vendor/libvirt.org/go/libvirt/libvirt_generated.h:35:10: fatal error: libvirt/libvirt.h: No such file or directory
   35 | #include <libvirt/libvirt.h>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
compilepkg: error running subcommand external/go_sdk/pkg/tool/linux_amd64/cgo: exit status 2

This only happens when crosscompiling, most likely because the libraries are otherwise natively installed.

I could verify that the referenced PR works fine with bazeldnf v0.5.8, so the error is introduced somewhere afterwards.

[jschintag]

Same error also occurs when building kubevirt for aarch64 on x86 from the main branch, using bazeldnf v0.5.9-rc2

[rmohr]

My immediate suspicion would be 5b26a65. @andreabolognani FYI.

[rmohr]

That's the only actual logic change we have in 0.5.9 (and was the reason for not staying on 0.5.8).

[andreabolognani]

I need to finish some stuff before the weekend so I won't have time to dig into this until next week, but from a quick look I don't immediately see an obvious connection between my change and the failure you're experiencing. The list of changes is pretty damning though... :)

If someone else has the time to do it before I get a chance, it would be interesting to compare how v0.5.9-rc1 and v0.5.9-rc2 behave, both when used on KubeVirt's main branch and the s390x branch, when cross-building for aarch64.

[rmohr]

If someone else has the time to do it before I get a chance, it would be interesting to compare how v0.5.9-rc1 and v0.5.9-rc2 behave, both when used on KubeVirt's main branch and the s390x branch, when cross-building for aarch64.

Not sure either if I will find the time, but hope so. A nice scoped problem to solve :)

[jschintag]

If someone else has the time to do it before I get a chance, it would be interesting to compare how v0.5.9-rc1 and v0.5.9-rc2 behave, both when used on KubeVirt's main branch and the s390x branch, when cross-building for aarch64.

I can try out rc-1 today, though it might take a while since i can't use the cache as it might have cached a successful build before (happened already once...)

[jschintag]

rc-1 also has the same error. Though i also don't want to loose 5b26a65, since i was the one running into the problem in the first place.
Since i'm new in the bazeldnf/bazel world, how would the resulting filepath look like?
Since the import this time is called libvirt/libvirt.h and not libvirt.h, my theory would be that the resulting filepath relative from the import path is not <bazeldnf-folder>/libvirt/libvirt.h, but maybe something more like the full path <bazeldnf-folder>/usr/include/libvirt/libvirt.h?
As i said, i'm not that familiar with this, this is just my uneducated guess.

[andreabolognani]

i can't use the cache as it might have cached a successful build before (happened already once...)

Yeah, it's always a good idea to clean everything up when testing this kind of stuff. I've been bitten by that before.

I'm also looking into it now, but can you please give a summary of what works and what doesn't based on your own testing? That'll speed up things a bit.

[jschintag]

Works:

• v0.5.8
  Does not work:
• v0.5.9-rc1
• v0.5.9-rc2

I used the kubevirt main branch and cross-compiling aarch64 for all tests.

[andreabolognani]

Okay. I assume native compilation worked though?

[jschintag]

Yes, native worked. But that is most likely because the builder image has libvirt-dev installed. So it does not need the includes from bazeldnf

[andreabolognani]

Right. Ideally that wouldn't be present but IIUC KubeVirt wants to also be buildable without Bazel. Pretty silly IMO but I don't make the rules :)

[andreabolognani]

I have figured out the issue. More details here.

[jschintag]

Since v0.5.9 is now released, i think we can close this.
The only question is if we should document the needed changes somewhere? Though i have looked through the README.md and there is no description of how the entry in BUILD.bazel is created. Is it auto-generated?

@rmohr I can add this change to the docs if you point me to where it should be added.

[rmohr]

The only question is if we should document the needed changes somewhere? Though i have looked through the README.md and there is no description of how the entry in BUILD.bazel is created. Is it auto-generated?

Not sure wich entries specifically you refer to. Some are auto-update (which contain the version). The rest is just written into the README.md.

[jschintag]

The only question is if we should document the needed changes somewhere? Though i have looked through the README.md and there is no description of how the entry in BUILD.bazel is created. Is it auto-generated?

Not sure wich entries specifically you refer to. Some are auto-update (which contain the version). The rest is just written into the README.md.

In kubevirt/kubevirt#11106, the changes to BUILD.bazel