course-automation: Automatic verification for mandatory parts of course-automation final

[landeholt]

Assignment Proposal

Title

Automatic verification for mandatory parts of course-automation final

Names and KTH ID

• John Landeholt ([email protected])
• John Landeholt ([email protected])

Deadline

Deadline to complete task 1: April 5, 17h Stockholm time

Category

Course Automation

Description

In order to alleviate redundant work for this course TAs I propose to automate verification of the following mandatory parts for "course-automation":

• timeliness: the automation is done before the first task deadline (in order to be useful for the course)
  The first task deadline should be a set as a variable that can be changed for each year. If the final pull request to this repo is after first task, the pull request should become invalid.
• repo: the code for the task is available in a public repo
  Check through the readme.md for a github-url and check whether the repo is public or not

The automation should perform the following :

• Run for each new pull request that is the course automation task
• Check if it is a final submission (i.e has a repo-link)
  • if final: check if it is before first task deadline and if repo is public
  • if not final: check if it is before first task
• Give feedback accordingly to the pull request

[dd2482-bot]

Missing student registration :asim

[dd2482-bot]

Missing student registration :asim

[dd2482-bot]

Missing student registration :asim

[dd2482-bot]

Student paulinev already registered for this task

[dd2482-bot]

Student paulinev already registered for this task

[dd2482-bot]

Student paulinev already registered for this task

[dd2482-bot]

Student paulinev already registered for this task

[landeholt]

Student paulinev already registered for this task

No and no

[khaes-kth]

Thanks @landeholt for this proposal

it is relevant and interesting.

By "if not final: check if it is before first task", do you mean check if the selected deadline is before the first task or if the PR creation date is before that?

[khaes-kth]

Student paulinev already registered for this task

No and no

Don't worry about this message. We're fixing it.

[landeholt]

Thanks @landeholt for this proposal

it is relevant and interesting.

By "if not final: check if it is before first task", do you mean check if the selected deadline is before the first task or if the PR creation date is before that?

I will have to check whether a PR is a proposal or a final one.

The reason for why is that it cannot be asked to have a repo already at the proposal stage. I was thinking that I could still invalidate proposals after the deadline.

[khaes-kth]

I see. My question is what do you check if you identify the PR as a proposal? Do you check the deadline suggested in the readme or do you check the creation date of the PR?

[landeholt]

I see. My question is what do you check if you identify the PR as a proposal? Do you check the deadline suggested in the readme or do you check the creation date of the PR?

Ah, now I follow. I am not sure for the time being. I am investigating the template, and would like to propose a change to explicitly state whether it is a final submission or a proposal. This would also make it much easier to automate the labeling.

When regarded to course-automation, you have these following possible scenarios:

1. the PR was created before deadline but doesn't have a repository link in it.
   This could indicate that it is a proposal and should therefore be accepted and could further be labeled as proposal
2. the PR was created before deadline and has a repository link in it
   This could indicate that it is a final submission and should therefore be accepted and could futher be labeled as final submission
3. the PR was created after deadline
   This should not be accepted

[khaes-kth]

I think this is an acceptable proposal. My understanding is that when a PR is created, your GHAction does the following:
1- Checks if it is a course automation by reading the text under "Category".
2- Checks if it is a proposal or final submission by reading the title, description, and readme and looking for a repo link.
3- If it is a proposal, it checks a) the selected deadline (in the readme) is task one and the PR is created before the deadline.
4- If it is a final submission, it checks that the PR is created before the selected deadline.
5- If 3 and 4 conditions are not met, the build fails.

[khaes-kth]

Student paulinev already registered for this task

No and no

Don't worry about this message. We're fixing it.

To fix this, can you please pull the latest version from upstream (KTH/devops-course 2022 branch)?

[landeholt]

I think this is an acceptable proposal. My understanding is that when a PR is created, your GHAction does the following: 1- Checks if it is a course automation by reading the text under "Category". 2- Checks if it is a proposal or final submission by reading the title, description, and readme and looking for a repo link. 3- If it is a proposal, it checks a) the selected deadline (in the readme) is task one and the PR is created before the deadline. 4- If it is a final submission, it checks that the PR is created before the selected deadline. 5- If 3 and 4 conditions are not met, the build fails.

I am right now evaluating what would be best practice. Should I read the PR body, or the readme from contributions/course-automation/*/*.md?

Will GHAction trigger on PR body updates?

I know that it does it on commits, hence might be better to read from file instead.

Regarding if it is a proposal or final, I've setup a switch case:

repo_urls = get_repo_urls(body)
if len(repo_urls) == 0 and is_final:
    raise MissingRepoError("No repository url found in provided pull request. Please provide one, or clearly state in your pull request that it is only a proposal.")

elif len(repo_urls) == 0 and not stage:
    # when stage (proposal or submission) is not explictly defined
    raise UnclearPullRequest("Cannot evaluate whether PR is final submission or proposal. Please state it explicitly in your PR")
elif len(repo_urls) == 1 and not is_final and stage:
    # explicitly stated that it is a proposal but has a repo-link (edge case for when someone already has a repo before being accepted)
    return
elif len(repo_urls) == 1 and not stage:
    # when repo-link exists, but not explicitly stated that it is a final submission
    is_final = True
    payload["__result__"]["stage"] = "final_submission"
    
elif len(repo_urls) > 1:
    raise AmbiguousRepoError("More than one repo was provided in the pull request")

Regarding checking deadline, I will be using the created_at attribute for the pull_request payload.

[khaes-kth]

Should I read the PR body, or the readme from contributions/course-automation/*/*.md?

I think it makes sense to focus on the README file.