feat: add a mage command to package system tests

[mdelapenya]

What does this PR do?

This PR adds a mage command to package the system tests results. This command is invoked in the CI in the same conditions that in the past.

The command is composed by:

• a mage packageSystemTests command, plugged into filebeat and metricbeat modules (both OSS and xpack flavours).
• a Tar method under the commons package, so that it's possible to compress a source directory using TAR+GZIP algorithms, and writing it to the build dir in the root directory of the Beats repository.

Besides that, we are removing a python script that detected the path to the Python system-tests (used in the pipeline to store the folder name in a variable), and we are updating the Jenkins pipeline to call the new build system command instead of the previous script. To keep existing behaviour, that we could consider it wrong and would possible discuss about it in a follow-up issue, we are passing a new argument to the groovy method in the pipeline, representing the directory in which the Beat live (i.e. filebeat, x-pack/filebeat), so that it's possible to run the mage command in the right context.

Why is it important?

In the past, the collection of the system tests results was done at the pipeline level, so developers were not able to reproduce them. With this new command now it's possible to collect them directly from the build system.

For debugging purpose, it will be possible to debug a CI worker, capturing it and executing this command, to verify if the files to be uploaded are there.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

$ cd filebeat
$ mage pythonIntegTest
$ mage packageSystemTests
>> skipping symlink: build/system-tests/run/test_harvester.Test.test_ignore_symlink/log/symlink.log
>> skipping symlink: build/system-tests/run/test_harvester.Test.test_symlink_and_file/log/symlink.log
>> skipping symlink: build/system-tests/run/test_harvester.Test.test_symlink_rotated/log/symlink.log
>> skipping symlink: build/system-tests/run/test_harvester.Test.test_symlinks_enabled/log/symlink.log
>> skipping symlink: build/system-tests/run/test_harvester.Test.test_truncate/log/symlink.log
>> skipping symlink: build/system-tests/run/test_input.Test.test_skip_symlinks/log/test.log

# checking file system contents
$ find build/system-tests/run -name "*" | wc -l
    3940
    
# checking TAR contents
$ tar -tf ../build/system-tests-filebeat.tar.gz | wc -l
    3934

Expected behaviour: a TAR file in the root's build directory named as the Beat will exist (i.e. ./build/system-tests-filebeat.tar.gz)

Related issues

• Relates [Build System] Add a command to package system tests files into a TAR file #26208

Use cases

Screenshots

Logs

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-08-20T14:40:07.538+0000

• Duration: 144 min 37 sec

• Commit: 6d165b5

Test stats 🧪

Test	Results
Failed	0
Passed	53527
Skipped	5322
Total	58849

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	53527
Skipped	5322
Total	58849

[andrewkroh]

Good idea

[mdelapenya]

Waiting on @andrewkroh for final review 🙏

[andrewkroh]

I think it's missing an error check, but otherwise LGTM.

[mdelapenya]

@jsoriano I'm gonna merge this one, but would like your help with the backports labels first: 7.x, 7.15 and 7.14?

[jsoriano]

@jsoriano I'm gonna merge this one, but would like your help with the backports labels first: 7.x, 7.15 and 7.14?

I would say to backport this to 7.x and 7.15.