[7.17](backport #39270) Ensure ordered DRA artifacts

[mergify]

Proposed commit message

As things are now we allow parallel builds on the
packaging pipeline, which would result in out of order artifacts (depending on which one takes longer to finish).

This commit leverages a queue to ensure a FIFO queue for DRA artifacts using the design pattern in:
https://buildkite.com/blog/concurrency-gates

Related issues

• https://github.com/elastic/ingest-dev/issues/3095

Testing/Screenshots

Using the latest approach, I triggered two DRA main snapshot builds in succession using the env vars:

DRA_BRANCH="main"
RUN_SNAPSHOT="true"
DRY_RUN="true"

and was able to see the first one triggered proceed:

whereas the second one was paused (as desired) waiting for the snapshot concurrency group:

It would have been nice to trigger a staging build at the same time (which should allow one job at a time, since it's a different concurrency gate) but it's not easy given the conditionals.

---

This is an automatic backport of pull request #39270 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of d275f27 has failed:

On branch mergify/bp/7.17/pr-39270
Your branch is up to date with 'origin/7.17'.

You are currently cherry-picking commit d275f2768c.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   .buildkite/packaging.pipeline.yml

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[botelastic]

This pull request doesn't have a Team:<team> label.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 13 min 57 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[dliappis]

Closing because it's already included in #39197 (#39270 has been cherry picked into it)