[FEATURE] Enable secure external execution of tests

[miguelgfierro]

Description

Since we removed the pull_request_target on GitHub (see #1840), external contributors won't trigger the tests. We need to think in ways of enabling this solution.

Expected behavior with the suggested feature

Other Comments

[miguelgfierro]

Ideas:

From Jianjie:
I found a framework on building these bots: probot
We could possibly set something up to listen on the github webhooks: pull_request_review_comment
And trigger a workflow: actions-create-workflow-dispatch

From Miguel:
Will it work if some of the core devs manually trigger the external action?

[miguelgfierro]

The current state is that when an external person does a PR, the admins need to press a button before the tests run, see https://github.com/recommenders-team/recommenders/actions/runs/6026551815/attempts/1

however, when this is pressed, then there is an authentication error with AzureML:

Error: Az CLI Login failed. Please check the credentials and make sure az is installed on the runner. For more information refer https://aka.ms/create-secrets-for-GitHub-workflows

See https://github.com/recommenders-team/recommenders/actions/runs/6026551815/job/16352596352?pr=1984

[miguelgfierro]

Follow the idea of feathr. They use a label that can be only added by an admin, after it is done, the triggers are done also for people that come from a fork.

See https://github.com/feathr-ai/feathr/blob/main/.github/workflows/pull_request_push_test.yml#L68

Blair Chen is going to send us more info about it.

[miguelgfierro]

hey @blrchen, one question, if seems that when we add the line if: github.event_name == 'pull_request' || (github.event_name == 'pull_request_target' && contains(github.event.pull_request.labels.*.name, 'safe to test')) in every PR, no matter whether it is from people inside the team or a person doing a fork, we need to add the label before the tests are executed.

Ideally what we would like is that the tests are executed automatically if one of the maintainers do a PR and that we need to add the label if the PR comes from a fork (an outsider). Do you know whether this is possible?

FYI @SimonYansenZhao @loomlike @anargyri

[blrchen]

We are a small team with only a few PR activities each week, so manually adding labels is manageable.

If you prefer automation, you might consider enabling a PR bot to assign labels automatically if a PR is submitted by someone from approved list.

[miguelgfierro]

thanks @blrchen. Do you have some code example of a similar bot?

[blrchen]

Hi @miguelgfierro ,

Unfortunately I don't have any samples. I've noticed that many repositories setup workflows to manage custom labels for issues and PRs. This could potentially be a solution.

[miguelgfierro]

Things I've tried:

   if: contains(github.event.pull_request.labels.*.name, 'safe to test') || github.event.pull_request.head.repo.fork == 'false'`

   if: github.event.pull_request.head.repo.fork == 'false' || (github.event_name == 'pull_request_target' && contains(github.event.pull_request.labels.*.name, 'safe to test'))´

   if: (github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == 'false') || (github.event_name == 'pull_request_target' && contains(github.event.pull_request.labels.*.name, 'safe to test'))

   if: (github.event_name == 'pull_request_target' && github.event.pull_request.head.repo.fork == 'false') || (github.event_name == 'pull_request_target' && contains(github.event.pull_request.labels.*.name, 'safe to test'))