Add FIPS env var to build crossbuild images for FIPS compliance

[michel-laterman]

Create images that use the microsoft/go toolchain instead of the upstream go one so that crossbuild images can be used in order to make FIPS compliant binaries.

FIPS images can be created by setting FIPS=true, i.e.:

FIPS=true make -C go -f Makefile.debian11 build

Images will have their tag extension end in -fips

[mergify]

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d is the label to automatically backport to the 1./d branch. /d is the digit
  NOTE: backport-skip has been added to this pull request.

[michel-laterman]

It looks like the builds are inconsistent; specifically the pcap installation

For the last buildkite execution debian11 failed when installing pcap with fips set to false

> [linux/arm64->amd64 final 4/4] RUN cd /libpcap/libpcap-1.8.1         && CC=aarch64-linux-gnu-gcc ./configure --enable-usb=no --enable-bluetooth=no --enable-dbus=no --host=aarch64-unknown-linux-gnu --with-pcap=linux         && make:
--
  | 3.680 checking build system type... aarch64-unknown-linux-gnu
  | 4.955 checking host system type... aarch64-unknown-linux-gnu
  | 5.357 checking target system type... aarch64-unknown-linux-gnu
  | 5.364 checking for aarch64-unknown-linux-gnu-gcc... aarch64-linux-gnu-gcc
  | 6.018 checking whether the C compiler works... no
  | 6.392 configure: error: in `/libpcap/libpcap-1.8.1':
  | 6.394 configure: error: C compiler cannot create executables
  | 6.394 See `config.log' for more details

yet when fips was set to true the debian11 build succeeded

There is also inconsistent behaviour when compared across builds, i.e the previous commit (where I fixed a typo in the label) https://buildkite.com/elastic/golang-crossbuild/builds/723
has debian10 fips:false as a failure, but this one succeeded with the most recent commit.

There is also an issue with the main branch around pcap, as well as other PRs (like #494)

[michel-laterman]

The most recent attempt had a segfault when compiling pcap for debian 9 fipst true:

202.1 aarch64-linux-gnu-gcc -fvisibility=hidden -fpic -I.  -DBUILDING_PCAP -DHAVE_CONFIG_H  -D_U_="__attribute__((unused))" -g -O2    -c bpf_filter.c
--
  | 202.3 aarch64-linux-gnu-gcc: internal compiler error: Segmentation fault (program cc1)
  | 202.3 Please submit a full bug report,

This image was correctly built previously
Could the buildkite runners be under-resourcing the docker builds?

[v1v]

Could the buildkite runners be under-resourcing the docker builds?

I've been told you can configure the machineType settings to use a bigger VM. So far the current one is n1-standard-4.

how did I know the type?

1. Go to https://buildkite.com/elastic/golang-crossbuild/builds/728
2. Click on Timeline
3. Look for Agent Targeting Rules.

How can you change the machine type?

1. Go to https://github.com/elastic/golang-crossbuild/blob/main/.buildkite/pipeline.yml
2. Search for:

        agents:
          provider: "gcp"

3. And use

        agents:
          provider: "gcp"
          machineType: "<CHANGEME>"

And <CHANGEME> should be one valid value, for instance pick one Machine types from https://cloud.google.com/compute/docs/general-purpose-machines

[michel-laterman]

I tried to use n4-standard-4 but it looks like we can't find any runners: The job, :linux: Staging / Ubuntu X86_64 - Makefile - fips: false, has been canceled as it failed to get an agent after 5 tries.

I've updated the x86_64 builders to n1-standard-8 (from the default n1-standard-4) and so far only the debian12 fips:true job has failed (during pcap installation); i'm going to rerun a couple times to see if it's just a flakey build.

[michel-laterman]

buildkite test this

[michel-laterman]

I don't think increasing instance sizes helps

[michel-laterman]

I've tried running FIPS="false" buildkite/scripts/build.sh Makefile.debian11 on an n2-standard-8 buildkite VM and it completed successfully; however this has failed in the latest build.

[jlind23]

@michel-laterman what our next options are then?

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b FIPS-build upstream/FIPS-build
git merge upstream/main
git push upstream FIPS-build

[michel-laterman]

I've squashed my changes and rebased onto #502, i'll let this run (as a sanity check) but will likely need to disable ppc builds on debian12 in order to get this working

[michel-laterman]

Rebased on main now that #504 has been merged

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 80c95b4

History

• 💚 Build #654 succeeded fbd89f8
• 💚 Build #653 succeeded a112df9
• 💚 Build #642 succeeded 292a823
• 💚 Build #640 succeeded 81c0b05
• 💚 Build #637 succeeded 233d201
• 💚 Build #636 succeeded 233d201

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 80c95b4

History

• 💚 Build #654 succeeded fbd89f8
• 💚 Build #653 succeeded a112df9
• 💚 Build #642 succeeded 292a823
• 💚 Build #640 succeeded 81c0b05
• 💚 Build #637 succeeded 233d201
• 💚 Build #636 succeeded 233d201

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 80c95b4

Failed CI Steps

• :linux: Staging / Ubuntu X86_64 - Makefile.debian10 - fips: true
• :linux: Staging / Ubuntu X86_64 - Makefile.debian10 - fips: true
• :linux: Staging / Ubuntu X86_64 - Makefile.debian11 - fips: true
• :linux: Staging / Ubuntu X86_64 - Makefile.debian12 - fips: true

History

• 💔 Build #750 failed fbd89f8
• 💔 Build #749 failed a112df9
• 💔 Build #738 failed 292a823
• 💔 Build #736 failed 81c0b05

[simitt]

LGTM