Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Add ARM packaging #24041

Merged
merged 10 commits into from
Feb 23, 2021
20 changes: 18 additions & 2 deletions Jenkinsfile
Original file line number Diff line number Diff line change
@@ -261,7 +261,21 @@ def k8sTest(Map args = [:]) {
}

/**
* This method runs the packaging
* This method runs the packaging for ARM
*/
def packagingArm(Map args = [:]) {
PACKAGES = "docker"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably drop this. This environment variable is needed when beats are released in the RM to make sure we do not produce extra artifacts.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kvch , I just changed it.

OTOH, the existing packaging automation it's using that particular flag:

Should it be removed too?

Eventually this PR should be the only way to build all the packages, therefore the .ci/packaging.groovy will be removed, but for the time being we are having both packaging processes, even though if they are duplicated. Since we don't wanna change the current process yet until it has been fully tested.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kvch , it seems the PACKAGES="docker" is needed otherwise:

[2021-02-16T13:12:36.923Z] standard_init_linux.go:211: exec user process caused "exec format error"
[2021-02-16T13:12:36.923Z] standard_init_linux.go:211: exec user process caused "exec format error"
[2021-02-16T13:13:48.030Z] package ran for 5m23.440051467s
[2021-02-16T13:13:48.031Z] Error: failed building packetbeat-oss type=deb for platform=linux/arm64: failed while running FPM in docker: running "docker run -e EXEC_UID=1152 -e EXEC_GID=1152 --rm -w /app -v /var/lib/jenkins/workspace/Beats_beats_PR-24041/src/github.com/elastic/beats/packetbeat:/app docker.elastic.co/beats-dev/fpm:1.11.0 fpm --force --input-type tar --output-type deb --name packetbeat --architecture arm64 --version 8.0.0 --vendor Elastic --license ASL-2.0 --description Packetbeat analyzes network traffic and sends the data to Elasticsearch. --url https://www.elastic.co/products/beats/packetbeat --after-install build/package/packetbeat-oss-linux-arm64.deb/scripts/systemd-daemon-reload.sh --config-files /etc/packetbeat/packetbeat.yml -p build/distributions/packetbeat-oss-8.0.0-SNAPSHOT-arm64.deb build/distributions/tmp-deb-packetbeat-8.0.0-SNAPSHOT-linux-arm64-8205217749.tar.gz" failed with exit code 1

See https://beats-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/Beats/pipelines/beats/pipelines/PR-24041/runs/2/steps/10864/log/?start=0

I'm just reverting my last commit and see if I see the same behaviour

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid the PACKAGES=docker is required otherwise the above error will happen.

@kvch , any clue?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed. Building these artifacts on ARM needs a bit more work.

v1v marked this conversation as resolved.
Show resolved Hide resolved
def PLATFORMS = [ 'linux/arm64' ].join(' ')
withEnv([
"PLATFORMS=${PLATFORMS}",
"PACKAGES=docker"
]) {
target(args)
}
}

/**
* This method runs the packaging for Linux
*/
def packagingLinux(Map args = [:]) {
def PLATFORMS = [ '+all',
@@ -285,7 +299,6 @@ def packagingLinux(Map args = [:]) {
}
}


/**
* Upload the packages to their snapshot or pull request buckets
* @param beatsFolder beats folder
@@ -920,6 +933,9 @@ class RunCommand extends co.elastic.beats.BeatsFunction {
if(args?.content?.containsKey('mage')) {
steps.target(context: args.context, command: args.content.mage, directory: args.project, label: args.label, withModule: withModule, isMage: true, id: args.id)
}
if(args?.content?.containsKey('packaging-arm')) {
steps.packagingArm(context: args.context, command: args.content.get('packaging-arm'), directory: args.project, label: args.label, isMage: true, id: args.id, e2e: args.content.get('e2e'), package: true)
}
if(args?.content?.containsKey('packaging-linux')) {
steps.packagingLinux(context: args.context, command: args.content.get('packaging-linux'), directory: args.project, label: args.label, isMage: true, id: args.id, e2e: args.content.get('e2e'), package: true)
}
6 changes: 6 additions & 0 deletions auditbeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -87,3 +87,9 @@ stages:
packaging-linux: "mage package"
e2e:
enabled: false
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"
6 changes: 6 additions & 0 deletions filebeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -85,3 +85,9 @@ stages:
packaging-linux: "mage package"
e2e:
enabled: false
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"
6 changes: 6 additions & 0 deletions heartbeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -84,3 +84,9 @@ stages:
packaging-linux: "mage package"
e2e:
enabled: false
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"
6 changes: 6 additions & 0 deletions journalbeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -37,3 +37,9 @@ stages:
packaging-linux: "mage package"
e2e:
enabled: false
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"
6 changes: 6 additions & 0 deletions metricbeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -79,3 +79,9 @@ stages:
packaging-linux: "mage package"
e2e:
enabled: false ## e2e is enabled only for x-pack beats
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"
6 changes: 6 additions & 0 deletions packetbeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -85,3 +85,9 @@ stages:
packaging-linux: "mage package"
e2e:
enabled: false
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"
6 changes: 6 additions & 0 deletions x-pack/auditbeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -85,3 +85,9 @@ stages:
packaging-linux: "mage package"
e2e:
enabled: false
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"
6 changes: 6 additions & 0 deletions x-pack/elastic-agent/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -89,3 +89,9 @@ stages:
packaging-linux: "mage package"
e2e:
enabled: false
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"
8 changes: 7 additions & 1 deletion x-pack/filebeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -91,4 +91,10 @@ stages:
packaging-linux:
packaging-linux: "mage package"
e2e:
enabled: false
enabled: false
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"
6 changes: 6 additions & 0 deletions x-pack/heartbeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -17,3 +17,9 @@ stages:
packaging-linux: "mage package"
e2e:
enabled: false
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"
6 changes: 6 additions & 0 deletions x-pack/metricbeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -84,3 +84,9 @@ stages:
e2e:
enabled: false
entrypoint: 'metricbeat-test.sh'
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"
6 changes: 6 additions & 0 deletions x-pack/packetbeat/Jenkinsfile.yml
Original file line number Diff line number Diff line change
@@ -86,3 +86,9 @@ stages:
packaging-linux: "mage package"
e2e:
enabled: false
packaging-arm:
packaging-arm: "mage package"
e2e:
enabled: false
platforms: ## override default label in this specific stage.
- "arm"