[7.17] Fix Python systems tests with forked docker-compose package

[cmacknz]

This is a manual backport of the fix for the Python systems tests contained in #38199, which uses a fork of the docker-compose Python package as a work around.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-python-system-tests upstream/fix-python-system-tests
git merge upstream/7.17
git push upstream fix-python-system-tests

[elasticmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 135 min 10 sec

Pipeline error

This error is likely related to the pipeline itself. Click here
and then you will see the error (either incorrect syntax or an invalid configuration).

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[cmacknz]

The test failed but not with the errors this change fixes, force merging.

[pkoutsovasilis]

The test failed but not with the errors this change fixes, force merging.

maybe we forgot to backport something here (not related to the docker fix), I will take a look.

[cmacknz]

#38664 is one of them

[cmacknz]

The other was https://beats-ci.elastic.co/job/Beats/job/beats/job/PR-38743/3/testReport/junit/metricbeat.module.prometheus.test_prometheus/TestRemoteWrite/Build_Test___metricbeat_pythonIntegTest___test_remote_write/

        if network_mode == 'host' and port_bindings:
>           raise host_config_incompatible_error(
                'network_mode', 'host', 'port_bindings'
            )
E           docker.errors.InvalidArgument: "host" network_mode is incompatible with port_bindings

/opt/venv/lib/python3.11/site-packages/docker/types/containers.py:337: InvalidArgument

That I'm pretty sure I haven't seen before.