ci: Implement workflow for full suite IT (cron + manual) #1353

pawelpasterz · 2020-11-25T15:14:52Z

What it actually does?

Basically, the workflow runs integration tests (IT) and reacts to results. If there are tests failed an issue is created with the IT_Failed label.

and a comment is added with build scan URL + list of commits from the last run (and associated PRs)

If an issue with the IT_Failed label is already created only comment is added to the existing ticket. Once the IT run is succeeded issue is closed automatically.

Workflow is implemented to listen to 2 triggers:

scheduled (at 00:00 every workday)
manual

Workflow is divided into 3 main sections

build and run IT
prepare message (specific for 1) results)
create issue or close issue or add comment *or skip

More detailed workflow step-by-step

Run IT
1. checkout the code, build and run integrationTests task
Process results
1. check if the issue is already opened
2. get time when the workflow was started last time, skip cancelled and in_progress (sets output last-run-date)
3. create either a success or failure message
4. for failure message:
  1. get a list of commits since the last run
  2. for each commit get associated PR
  3. append the results table
5. make comment body
6. decide what to do next:
  1. if IT status is success and no issue opened -- do nothing
  2. if IT status is success and there is an issue opened -- add a comment and close the issue
  3. it IT status is failure and there is no issue opened -- create an issue and add comment
  4. if IT status is failure and there is issue opened -- add comment

Test Plan

How do we know the code works?

Note:
flank-scrips has Flank/flank repo hardcoded (and some other variables). I will implement a feature to make life easier while testing scripts in the near future...
For now, before the build, please replace:

flank/flank-scripts/src/main/kotlin/flank/scripts/github/GithubApi.kt

Line 35 in 9d1ba8d

private const val URL_BASE = "https://api.github.com/repos/Flank/flank"

with your forked repo
flank/flank-scripts/src/main/kotlin/flank/scripts/integration/IssueList.kt

Line 10 in 8214ad6

"creator" to "github-actions[bot]",

github-actions[bot] with your repo's username

(PAT with full access needed)

build flank locally ./gradlew build
run java -jar ./flank-scripts/build/libs/flank-scripts.jar integration processResults --result success --url https://any.url --github-token $(echo $GITHUB_TOKEN) --run-id 123456

no issue created

** Parameters:
      result: SUCCESS
      url:    https://any.url
      runID:  123456
** No workflow run found for full_suite_integration_tests.yml
** No opened issue

run java -jar ./flank-scripts/build/libs/flank-scripts.jar integration processResults --result failure --url https://any.url --github-token $(echo $GITHUB_TOKEN) --run-id 123456
new issue was created
run java -jar ./flank-scripts/build/libs/flank-scripts.jar integration processResults --result success --url https://any.url --github-token $(echo $GITHUB_TOKEN) --run-id 123456
a comment was added
run java -jar ./flank-scripts/build/libs/flank-scripts.jar integration processResults --result success --url https://any.url --github-token $(echo $GITHUB_TOKEN) --run-id 123456
issue was closed

Example

Example issue pawelpasterz#25
Contains all steps (create, add a comment, and close)

Checklist

github-actions · 2020-11-25T15:23:47Z

Timestamp: 2020-12-14 05:18:44
Buildscan url for ubuntu-workflow run 419964851
https://gradle.com/s/dzqnhrweigzwq

.github/workflows/full_suite_integration_tests_cron.yml

pawelpasterz · 2020-11-26T13:17:59Z

Some changes to be implemented:

job should gather a list of commits since last IT run
find PR for listed commits
build scan URL + PRs list should be separated comment
if the issue already created, add a comment instead of creating a new one

cc
@piotradamczyk5 @jan-gogo

jan-goral · 2020-11-26T15:48:04Z

Some changes to be implemented:

* job should gather a list of commits since last IT run

* find PR for listed commits

* build scan URL + PRs list should be separated comment

* if the issue already created, add a comment instead of creating a new one

cc
@piotradamczyk5 @jan-gogo

#1351 (comment)

codecov-io · 2020-12-01T13:09:25Z

Codecov Report

Merging #1353 (a256556) into master (8f49608) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1353      +/-   ##
============================================
- Coverage     77.61%   77.58%   -0.03%     
  Complexity      717      717              
============================================
  Files           248      248              
  Lines          4788     4788              
  Branches        915      915              
============================================
- Hits           3716     3715       -1     
  Misses          572      572              
- Partials        500      501       +1

Sloox

If this is blocked shall we move it to draft?

.github/workflows/full_suite_integration_tests_cron.yml

piotradamczyk5 · 2020-12-02T12:09:17Z

Once the IT run is succeeded issue is closed automatically.

I do not agree with this condition, because later on we will have running it on custom branches and the issue could be closed without resolution

flank-scripts/src/main/kotlin/flank/scripts/integration/CommitList.kt

flank-scripts/src/main/kotlin/flank/scripts/integration/Extensions.kt

flank-scripts/src/main/kotlin/flank/scripts/integration/IssueList.kt

piotradamczyk5

Please resolve comments and update flank-scripts/readme.md also

flank-scripts/build.gradle.kts

jan-goral

Great work! Only a few minor formatting style-related suggestions.
I have to perform manual tests.

flank-scripts/src/main/kotlin/flank/scripts/integration/PrepareMessage.kt

.github/workflows/full_suite_integration_tests.yml

flank-scripts/src/main/kotlin/flank/scripts/integration/CommitList.kt

flank-scripts/src/main/kotlin/flank/scripts/integration/Extensions.kt

flank-scripts/src/main/kotlin/flank/scripts/integration/IssueList.kt

flank-scripts/src/main/kotlin/flank/scripts/integration/Extensions.kt

jan-goral

Tested step by step, everything works fine

MacBook-Pro-jan-gogo:flank-jan-gogo jan-gogo$ java -jar ./flank-scripts/build/libs/flank-scripts.jar integration processResults --result success --url https://any.url --github-token $(echo $GITHUB_TOKEN) --run-id 123456
** Parameters:
     result: SUCCESS
     url:    https://any.url
     runID:  123456
** No workflow run found for full_suite_integration_tests.yml
** No opened issue
MacBook-Pro-jan-gogo:flank-jan-gogo jan-gogo$ java -jar ./flank-scripts/build/libs/flank-scripts.jar integration processResults --result failure --url https://any.url --github-token $(echo $GITHUB_TOKEN) --run-id 123456
** Parameters:
     result: FAILURE
     url:    https://any.url
     runID:  123456
** No workflow run found for full_suite_integration_tests.yml
** No opened issue
** Creating new issue
{
    "title": "Full Suite integration tests failed on master",
    "body": "### Integration Test failed on master",
    "labels": [
        "IT_Failed",
        "bug"
    ]
}
Error while doing GitHub request, because of Issues are disabled for this repo, more info at https://docs.github.com/v3/issues/
MacBook-Pro-jan-gogo:flank-jan-gogo jan-gogo$ java -jar ./flank-scripts/build/libs/flank-scripts.jar integration processResults --result failure --url https://any.url --github-token $(echo $GITHUB_TOKEN) --run-id 123456
** Parameters:
     result: FAILURE
     url:    https://any.url
     runID:  123456
** No workflow run found for full_suite_integration_tests.yml
** No opened issue
** Creating new issue
{
    "title": "Full Suite integration tests failed on master",
    "body": "### Integration Test failed on master",
    "labels": [
        "IT_Failed",
        "bug"
    ]
}
** Issue created:
     url:    https://github.com/jan-gogo/flank/issues/3
     number: 3 
** Comment posted
{
    "body": "### Full suite IT run :x: FAILED :x:\n**Timestamp:** 2020-12-11 14:04:23\n**Job run:** [123456](https://github.com/Flank/flank/actions/runs/123456\n**Build scan URL:** https://any.url\nNo new commits\n"
}
MacBook-Pro-jan-gogo:flank-jan-gogo jan-gogo$ java -jar ./flank-scripts/build/libs/flank-scripts.jar integration processResults --result success --url https://any.url --github-token $(echo $GITHUB_TOKEN) --run-id 123456
** Parameters:
     result: SUCCESS
     url:    https://any.url
     runID:  123456
** No workflow run found for full_suite_integration_tests.yml
** Issue found: https://github.com/jan-gogo/flank/issues/3
** Comment posted
{
    "body": "### Full suite IT run :white_check_mark: SUCCEEDED :white_check_mark:\n**Timestamp:** 2020-12-11 14:04:48\n**Job run:** [123456](https://github.com/Flank/flank/actions/runs/123456\n**Build scan URL:** https://any.url\n**Closing issue**"
}
** Closing issue
MacBook-Pro-jan-gogo:flank-jan-gogo jan-gogo$ java -jar ./flank-scripts/build/libs/flank-scripts.jar integration processResults --result success --url https://any.url --github-token $(echo $GITHUB_TOKEN) --run-id 123456
** Parameters:
     result: SUCCESS
     url:    https://any.url
     runID:  123456
** No workflow run found for full_suite_integration_tests.yml
** No opened issue
MacBook-Pro-jan-gogo:flank-jan-gogo jan-gogo$

piotradamczyk5 · 2020-12-11T16:56:44Z

@Mergifyio rebase

mergify · 2020-12-11T16:57:01Z

Command rebase: action_required

Pull request must be rebased manually.
GitHub App like Mergify are not allowed to rebase pull request where .github/workflows is changed.

…pened

pawelpasterz force-pushed the 1351-add-it-cron-job branch from 06f2e57 to 797fa84 Compare November 25, 2020 15:17

pawelpasterz force-pushed the 1351-add-it-cron-job branch from 797fa84 to f575ed7 Compare November 25, 2020 15:32

pawelpasterz self-assigned this Nov 26, 2020

pawelpasterz marked this pull request as ready for review November 26, 2020 11:22

pawelpasterz requested review from adamfilipow92, zuziaka, bootstraponline, jan-goral, piotradamczyk5 and Sloox November 26, 2020 11:22

piotradamczyk5 reviewed Nov 26, 2020

View reviewed changes

.github/workflows/full_suite_integration_tests_cron.yml Outdated Show resolved Hide resolved

piotradamczyk5 reviewed Nov 26, 2020

View reviewed changes

.github/workflows/full_suite_integration_tests_cron.yml Outdated Show resolved Hide resolved

pawelpasterz marked this pull request as draft November 26, 2020 13:14

pawelpasterz force-pushed the 1351-add-it-cron-job branch from be678af to e073cbb Compare December 1, 2020 06:20

pawelpasterz changed the title ~~ci: Implement cron job for full suite IT~~ ci: Implement workflow for full suite IT (cron + manual) Dec 1, 2020

pawelpasterz force-pushed the 1351-add-it-cron-job branch 2 times, most recently from bf90c3b to c49b850 Compare December 1, 2020 14:30

pawelpasterz marked this pull request as ready for review December 1, 2020 14:31

pawelpasterz requested a review from piotradamczyk5 December 1, 2020 14:32

pawelpasterz force-pushed the 1351-add-it-cron-job branch 2 times, most recently from 8257dcc to 8a4808a Compare December 2, 2020 08:02

Sloox reviewed Dec 2, 2020

View reviewed changes

piotradamczyk5 reviewed Dec 2, 2020

View reviewed changes

.github/workflows/full_suite_integration_tests_cron.yml Outdated Show resolved Hide resolved

piotradamczyk5 reviewed Dec 2, 2020

View reviewed changes

.github/workflows/full_suite_integration_tests_cron.yml Outdated Show resolved Hide resolved

piotradamczyk5 reviewed Dec 2, 2020

View reviewed changes

.github/workflows/full_suite_integration_tests_cron.yml Outdated Show resolved Hide resolved

piotradamczyk5 reviewed Dec 11, 2020

View reviewed changes

flank-scripts/src/main/kotlin/flank/scripts/integration/CommitList.kt Show resolved Hide resolved

piotradamczyk5 reviewed Dec 11, 2020

View reviewed changes

flank-scripts/src/main/kotlin/flank/scripts/integration/CommitList.kt Show resolved Hide resolved

piotradamczyk5 reviewed Dec 11, 2020

View reviewed changes

flank-scripts/src/main/kotlin/flank/scripts/integration/Extensions.kt Outdated Show resolved Hide resolved

piotradamczyk5 reviewed Dec 11, 2020

View reviewed changes

flank-scripts/src/main/kotlin/flank/scripts/integration/IssueList.kt Outdated Show resolved Hide resolved

piotradamczyk5 suggested changes Dec 11, 2020

View reviewed changes

piotradamczyk5 reviewed Dec 11, 2020

View reviewed changes

flank-scripts/build.gradle.kts Outdated Show resolved Hide resolved

pawelpasterz requested a review from piotradamczyk5 December 11, 2020 09:53

jan-goral reviewed Dec 11, 2020

View reviewed changes

jan-goral approved these changes Dec 11, 2020

View reviewed changes

piotradamczyk5 approved these changes Dec 11, 2020

View reviewed changes

pawelpasterz added 14 commits December 14, 2020 06:13

Implement cron job for full suite IT

08a6739

Implement feture to skip issue creation if there is at least alredy o…

adb62be

…pened

Refactor

2777aa9

Use github context to get repository

95d3c8e

Update GH API

8cc1b92

Add integration logic

b06788a

Rewrite to flank scripts

c615b6a

Refactor and update tests

e7dc4df

Change workflow yml

d0eaafa

Change formatter

079e9ad

Address CR comments

5f131dd

Update flank-scripts readme file

7502808

Remove flow

4ee87fd

Linting

a256556

pawelpasterz force-pushed the 1351-add-it-cron-job branch from 73130da to a256556 Compare December 14, 2020 05:13

pawelpasterz merged commit 44800ce into master Dec 14, 2020

pawelpasterz deleted the 1351-add-it-cron-job branch December 14, 2020 06:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: Implement workflow for full suite IT (cron + manual) #1353

ci: Implement workflow for full suite IT (cron + manual) #1353

pawelpasterz commented Nov 25, 2020 •

edited

Loading

github-actions bot commented Nov 25, 2020 •

edited

Loading

pawelpasterz commented Nov 26, 2020 •

edited

Loading

jan-goral commented Nov 26, 2020

codecov-io commented Dec 1, 2020 •

edited

Loading

Sloox left a comment

piotradamczyk5 commented Dec 2, 2020

piotradamczyk5 left a comment

jan-goral left a comment

jan-goral left a comment

piotradamczyk5 commented Dec 11, 2020

mergify bot commented Dec 11, 2020

ci: Implement workflow for full suite IT (cron + manual) #1353

ci: Implement workflow for full suite IT (cron + manual) #1353

Conversation

pawelpasterz commented Nov 25, 2020 • edited Loading

What it actually does?

Workflow is implemented to listen to 2 triggers:

Workflow is divided into 3 main sections

More detailed workflow step-by-step

Run IT

Process results

Test Plan

Example

Checklist

github-actions bot commented Nov 25, 2020 • edited Loading

pawelpasterz commented Nov 26, 2020 • edited Loading

jan-goral commented Nov 26, 2020

codecov-io commented Dec 1, 2020 • edited Loading

Codecov Report

Sloox left a comment

Choose a reason for hiding this comment

piotradamczyk5 commented Dec 2, 2020

piotradamczyk5 left a comment

Choose a reason for hiding this comment

jan-goral left a comment

Choose a reason for hiding this comment

jan-goral left a comment

Choose a reason for hiding this comment

piotradamczyk5 commented Dec 11, 2020

mergify bot commented Dec 11, 2020

pawelpasterz commented Nov 25, 2020 •

edited

Loading

github-actions bot commented Nov 25, 2020 •

edited

Loading

pawelpasterz commented Nov 26, 2020 •

edited

Loading

codecov-io commented Dec 1, 2020 •

edited

Loading