[CI] Add ARM packaging #24041

v1v · 2021-02-15T15:36:37Z

What does this PR do?

Enable ARM packaging

Why is it important?

Every PR will validate their packaging

Issues

This is a follow up from #23854 to support ARM docker images.

$ docker pull docker.elastic.co/observability-ci/auditbeat-oss:pr-24041-arm64
$ docker inspect docker.elastic.co/observability-ci/auditbeat-oss:pr-24041-arm64 | grep -i archi
        "Architecture": "arm64",
$ docker pull docker.elastic.co/observability-ci/auditbeat-oss:pr-24041-amd64
$ docker inspect docker.elastic.co/observability-ci/auditbeat-oss:pr-24041-amd64 | grep -i archi
        "Architecture": "amd64",

kvch · 2021-02-15T16:23:57Z

Jenkinsfile

+* This method runs the packaging for ARM
+*/
+def packagingArm(Map args = [:]) {
+  PACKAGES = "docker"


We can probably drop this. This environment variable is needed when beats are released in the RM to make sure we do not produce extra artifacts.

Thanks @kvch , I just changed it.

OTOH, the existing packaging automation it's using that particular flag:

beats/.ci/packaging.groovy

Line 227 in 125c29d

PACKAGES = "docker"

Should it be removed too?

Eventually this PR should be the only way to build all the packages, therefore the .ci/packaging.groovy will be removed, but for the time being we are having both packaging processes, even though if they are duplicated. Since we don't wanna change the current process yet until it has been fully tested.

@kvch , it seems the PACKAGES="docker" is needed otherwise:

[2021-02-16T13:12:36.923Z] standard_init_linux.go:211: exec user process caused "exec format error" [2021-02-16T13:12:36.923Z] standard_init_linux.go:211: exec user process caused "exec format error" [2021-02-16T13:13:48.030Z] package ran for 5m23.440051467s [2021-02-16T13:13:48.031Z] Error: failed building packetbeat-oss type=deb for platform=linux/arm64: failed while running FPM in docker: running "docker run -e EXEC_UID=1152 -e EXEC_GID=1152 --rm -w /app -v /var/lib/jenkins/workspace/Beats_beats_PR-24041/src/github.com/elastic/beats/packetbeat:/app docker.elastic.co/beats-dev/fpm:1.11.0 fpm --force --input-type tar --output-type deb --name packetbeat --architecture arm64 --version 8.0.0 --vendor Elastic --license ASL-2.0 --description Packetbeat analyzes network traffic and sends the data to Elasticsearch. --url https://www.elastic.co/products/beats/packetbeat --after-install build/package/packetbeat-oss-linux-arm64.deb/scripts/systemd-daemon-reload.sh --config-files /etc/packetbeat/packetbeat.yml -p build/distributions/packetbeat-oss-8.0.0-SNAPSHOT-arm64.deb build/distributions/tmp-deb-packetbeat-8.0.0-SNAPSHOT-linux-arm64-8205217749.tar.gz" failed with exit code 1

See https://beats-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/Beats/pipelines/beats/pipelines/PR-24041/runs/2/steps/10864/log/?start=0

I'm just reverting my last commit and see if I see the same behaviour

I'm afraid the PACKAGES=docker is required otherwise the above error will happen.

@kvch , any clue?

Indeed. Building these artifacts on ARM needs a bit more work.

elasticmachine · 2021-02-15T17:27:51Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #24041 updated
Start Time: 2021-02-23T14:38:46.475+0000
Duration: 49 min 14 sec
Commit: 22caa89

Test stats 🧪

Test	Results
Failed	0
Passed	46710
Skipped	4939
Total	51649

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	46710
Skipped	4939
Total	51649

v1v · 2021-02-16T09:37:51Z

Requires -> #24034

…-arm * upstream/master: [Metricbeat][Kubernetes] Extend state_node with more conditions (elastic#23905) [CI] googleStorageUploadExt step (elastic#24048) Check fields are documented for aws metricsets (elastic#23887) Update go-concert to 0.1.0 (elastic#23770) [Libbeat][New Processor] XML Decode (elastic#23678) Fix: bad substitution of API key (elastic#24036) [Filebeat] Add Pensando DFW Module (elastic#21063) [Filebeat] Check if processor is supported by ES version (elastic#23763) Syslog system tests: be more forgiving (elastic#24021)

This reverts commit 7b7ce07.

…-arm * upstream/master: [CI] install docker-compose with retry (elastic#24069) Add nodes to filebeat-kubernetes.yaml ClusterRole - fixes elastic#24051 (elastic#24052) updating manifest files for filebeat threatintel module (elastic#24074) Add Zeek Signatures (elastic#23772) Update Beats to ECS 1.8.0 (elastic#23465) Support running Docker logging plugin on ARM64 (elastic#24034) Fix ec2 metricset fields.yml and add integration test (elastic#23726) Only build targz and zip versions of Beats if PACKAGES is set in agent (elastic#24060) [Filebeat] Add field definitions for known Netflow/IPFIX vendor fields (elastic#23773) [Elastic Agent] Enroll with Fleet Server (elastic#23865) [Filebeat] Convert logstash logEvent.action objects to strings (elastic#23944) [Ingest Management] Fix reloading of log level for services (elastic#24055) Add Agent standalone k8s manifest (elastic#23679)

Jenkinsfile

elasticmachine · 2021-02-19T11:25:52Z

Pinging @elastic/ingest-management (Team:Ingest Management)

…-arm * upstream/master: (24 commits) Add example input autodsicover config (elastic#24157) Empty configuration options generate `<no value>` string for azure-eventhub input (elastic#24156) Remove OSS reference for kibana and elasticsearch (elastic#24164) Skip flaky TestActions on MacOSx (elastic#23966) [Filebeat][AWS] Fix vpcflow pipeline exception: Cannot invoke "Object.getClass()" because "receiver" is null (elastic#24167) [Elastic Agent] Fix docker entrypoint for elastic-agent. (elastic#24155) [PACKAGING] Push docker images with the architecture in the version (elastic#24121) [Agent] Add agent standalone manifests for system module & Pod's log collection (elastic#23938) indicator type url is in upper case (elastic#24152) [Filebeat] Document netflow internal_networks and set default (elastic#24110) [Filebeat] Adding fixes to the TI module (elastic#24133) [Enhancement] Add RotateOnStartup feature flag for file output (elastic#19347) [Ingest Manager] Fix: Successfully installed and enrolled agent running standalone (elastic#24128) Set Elastic licence type for APM server Beats update job (elastic#24122) Add logrotation section on Running Filebeat on k8s (elastic#24120) [CI] Run if manual UI (elastic#24116) [CI] enable x-pack/heartbeat in the CI (elastic#23873) chore: comment out the E2E (elastic#24109) chore: add-backport-next (elastic#24098) Adjust the position of the architecture name in Dockerlogbeat tarball (elastic#24095) ...

…ture/packaging-arm * 'feature/packaging-arm' of github.com:v1v/beats: Update Jenkinsfile

…dows-7 * upstream/master: [CI] Add ARM packaging (elastic#24041) Add example input autodsicover config (elastic#24157) Empty configuration options generate `<no value>` string for azure-eventhub input (elastic#24156)

[CI] Add ARM packaging

a219585

v1v requested review from kvch and a team February 15, 2021 15:36

v1v requested review from a team as code owners February 15, 2021 15:36

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 15, 2021

v1v self-assigned this Feb 15, 2021

v1v marked this pull request as draft February 15, 2021 15:36

v1v added automation Team:Automation Label for the Observability productivity team labels Feb 15, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 15, 2021

mdelapenya approved these changes Feb 15, 2021

View reviewed changes

kvch approved these changes Feb 15, 2021

View reviewed changes

kvch reviewed Feb 15, 2021

View reviewed changes

cachedout approved these changes Feb 16, 2021

View reviewed changes

v1v added 5 commits February 16, 2021 12:41

As suggested in the code review

7b7ce07

Revert "As suggested in the code review"

716ccca

This reverts commit 7b7ce07.

Caused by elastic#24034

e4f9497

v1v commented Feb 18, 2021

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

Update Jenkinsfile

d3df415

v1v marked this pull request as ready for review February 18, 2021 16:08

botelastic bot added the Team:Ingest Management label Feb 19, 2021

v1v added 3 commits February 23, 2021 14:35

Add docker arch arm64

4bbb59a

Merge branch 'feature/packaging-arm' of github.com:v1v/beats into fea…

22caa89

…ture/packaging-arm * 'feature/packaging-arm' of github.com:v1v/beats: Update Jenkinsfile

v1v merged commit 9e6b676 into elastic:master Feb 23, 2021

v1v added a commit to v1v/beats that referenced this pull request Feb 23, 2021

[CI] Add ARM packaging (elastic#24041)

1d7d55a

v1v mentioned this pull request Feb 23, 2021

Cherry-pick to 7.x: [CI] Add ARM packaging (#24041) #24187

Merged

v1v added a commit to v1v/beats that referenced this pull request Feb 23, 2021

[CI] Add ARM packaging (elastic#24041)

1a9f524

v1v mentioned this pull request Feb 23, 2021

Cherry-pick to 7.12: [CI] Add ARM packaging (#24041) #24188

Merged

v1v added a commit that referenced this pull request Feb 24, 2021

[CI] Add ARM packaging (#24041) (#24187)

3e7530c

v1v added a commit that referenced this pull request Feb 24, 2021

[CI] Add ARM packaging (#24041) (#24188)

519fe0d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Add ARM packaging #24041

[CI] Add ARM packaging #24041

v1v commented Feb 15, 2021 •

edited

Loading

kvch Feb 15, 2021

v1v Feb 16, 2021

v1v Feb 16, 2021

v1v Feb 16, 2021

kvch Feb 19, 2021

elasticmachine commented Feb 15, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

v1v commented Feb 16, 2021

elasticmachine commented Feb 19, 2021

[CI] Add ARM packaging #24041

[CI] Add ARM packaging #24041

Conversation

v1v commented Feb 15, 2021 • edited Loading

What does this PR do?

Why is it important?

Issues

kvch Feb 15, 2021

Choose a reason for hiding this comment

v1v Feb 16, 2021

Choose a reason for hiding this comment

v1v Feb 16, 2021

Choose a reason for hiding this comment

v1v Feb 16, 2021

Choose a reason for hiding this comment

kvch Feb 19, 2021

Choose a reason for hiding this comment

elasticmachine commented Feb 15, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

v1v commented Feb 16, 2021

elasticmachine commented Feb 19, 2021

v1v commented Feb 15, 2021 •

edited

Loading

elasticmachine commented Feb 15, 2021 •

edited by jenkins-beats-ci bot

Loading