Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Reporter for Container CI #213

Merged
merged 56 commits into from
Jul 26, 2024
Merged

Test Reporter for Container CI #213

merged 56 commits into from
Jul 26, 2024

Conversation

cholocate
Copy link
Contributor

Description

The following changes in the .yaml files worked with the previous forked repository, and will likely require some additional changes as we migrate to the new ai-containers format.

Changes Made

  • provide upload-artifact actions in action.yml to store test-runner logs persistently across each matrix element that was picked up in group-diff
  • provide merge-logs action to download artifacts after the test-runner completed and uses jq to produce combined JSON file with all tests and its status, then transfer result to output.txt

Validation

  • Expected format should be similar to below:

image

Copy link

github-actions bot commented Jul 3, 2024

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

PackageVersionScoreDetails
actions/actions/download-artifact fa0a91b85d4f404e444e00e005971372dc801d16 🟢 6.6
Details
CheckScoreReason
Code-Review🟢 10all changesets reviewed
Maintained🟢 45 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 4
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Branch-Protection⚠️ -1internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Packaging⚠️ -1packaging workflow not detected
Binary-Artifacts🟢 10no binaries found in the repo
Fuzzing⚠️ 0project is not fuzzed
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies⚠️ 1dependency not pinned by hash detected -- score normalized to 1
Security-Policy🟢 9security policy file detected
SAST🟢 10SAST tool is run on all commits
Vulnerabilities🟢 100 existing vulnerabilities detected
actions/marocchino/sticky-pull-request-comment 331f8f5b4215f0445d3c07b4967662a32a2d3e31 🟢 5.4
Details
CheckScoreReason
Maintained🟢 1030 commit(s) and 2 issue activity found in the last 90 days -- score normalized to 10
Code-Review⚠️ 2Found 1/5 approved changesets -- score normalized to 2
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Packaging⚠️ -1packaging workflow not detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Binary-Artifacts🟢 10no binaries found in the repo
Token-Permissions🟢 9detected GitHub workflow tokens with excessive permissions
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Fuzzing⚠️ 0project is not fuzzed
Security-Policy⚠️ 0security policy file not detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
Vulnerabilities🟢 100 existing vulnerabilities detected

Scanned Manifest Files

.github/workflows/integration-test.yaml

@tylertitsworth tylertitsworth added WIP Work in Progress github_actions Pull requests that update GitHub Actions code labels Jul 3, 2024
@github-advanced-security
Copy link

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

@tylertitsworth tylertitsworth changed the title Combining Test Runner Logs and output single table for all tests ran. Test Reporter for Container CI Jul 8, 2024
Comment on lines 91 to 95
- name: Display structure of downloaded files
run: |
ls .
cat combined.json
cat output.txt
Copy link
Contributor

@tylertitsworth tylertitsworth Jul 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be removed assuming you are done debugging

)
json_summary_path = f"test-runner-summary-{unique_identifier}.json"

with open(json_summary_path, "w") as file:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
with open(json_summary_path, "w") as file:
with open(json_summary_path, "w", encoding="utf-8") as file:

This will fix your lint error.

Copy link

Test-Group Test Status
python-tests-logs import-core PASS
python-tests-logs perf-full PASS
python-tests-logs perf-core PASS
python-tests-logs perf-stock PASS

@cholocate
Copy link
Contributor Author

Resolved case where tests.yaml is not present when group-diff triggered the container. Expected behavior: Skip job. Validated (please see the following PR run)
Also validated regular case where test-runner is supposed to run. Please see this PR run

Challenges and notes to self:

Make sure to always use [[ ]] when running bash conditional if statements, and also have matching quotes.

@tylertitsworth tylertitsworth added Review and removed WIP Work in Progress labels Jul 26, 2024
Copy link
Contributor

@tylertitsworth tylertitsworth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tylertitsworth tylertitsworth merged commit 14a2008 into main Jul 26, 2024
7 of 13 checks passed
@tylertitsworth tylertitsworth deleted the leminh/test-runners branch July 26, 2024 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
github_actions Pull requests that update GitHub Actions code Review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants