Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use result files for deployment #263

Merged
merged 10 commits into from
Mar 27, 2024
Merged

use result files for deployment #263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
b6a6c6c
update job_metadata.py for improving deployment
truib Mar 14, 2024
9d37c4e
use result file for determining status of a job
truib Mar 14, 2024
0753f29
removing blank line
truib Mar 14, 2024
ecc4a81
adjusts tests and use refactored functions
truib Mar 15, 2024
2a92aa9
added some logging to check_job_status
truib Mar 15, 2024
bea1abb
add missing ')'
truib Mar 15, 2024
2e8e0fd
add a bit more logging + fix status comparison
truib Mar 21, 2024
e705494
replacing 'tarball' with 'artefact' where it makes most sense
truib Mar 26, 2024
34ead4d
clean up unused constants
truib Mar 26, 2024
6beaa22
removing unused config settings
truib Mar 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 18 additions & 53 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,9 @@ You can exit the virtual environment simply by running `deactivate`.

### <a name="step4.1"></a>Step 4.1: Installing tools to access S3 bucket

The [`scripts/eessi-upload-to-staging`](https://github.com/EESSI/eessi-bot-software-layer/blob/main/scripts/eessi-upload-to-staging) script uploads a tarball and an associated metadata file to an S3 bucket.
The
[`scripts/eessi-upload-to-staging`](https://github.com/EESSI/eessi-bot-software-layer/blob/main/scripts/eessi-upload-to-staging)
script uploads an artefact and an associated metadata file to an S3 bucket.

It needs two tools for this:
* the `aws` command to actually upload the files;
Expand Down Expand Up @@ -444,14 +446,17 @@ information about the result of the command that was run (can be empty).

The `[deploycfg]` section defines settings for uploading built artefacts (tarballs).
```
tarball_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging
artefact_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging
```
`tarball_upload_script` provides the location for the script used for uploading built software packages to an S3 bucket.
`artefact_upload_script` provides the location for the script used for uploading built software packages to an S3 bucket.

```
endpoint_url = URL_TO_S3_SERVER
```
`endpoint_url` provides an endpoint (URL) to a server hosting an S3 bucket. The server could be hosted by a commercial cloud provider like AWS or Azure, or running in a private environment, for example, using Minio. The bot uploads tarballs to the bucket which will be periodically scanned by the ingestion procedure at the Stratum 0 server.
`endpoint_url` provides an endpoint (URL) to a server hosting an S3 bucket. The
server could be hosted by a commercial cloud provider like AWS or Azure, or
running in a private environment, for example, using Minio. The bot uploads
artefacts to the bucket which will be periodically scanned by the ingestion procedure at the Stratum 0 server.


```ini
Expand All @@ -466,7 +471,7 @@ bucket_name = {
}
```

`bucket_name` is the name of the bucket used for uploading of tarballs.
`bucket_name` is the name of the bucket used for uploading of artefacts.
The bucket must be available on the default server (`https://${bucket_name}.s3.amazonaws.com`), or the one provided via `endpoint_url`.

`bucket_name` can be specified as a string value to use the same bucket for all target repos, or it can be mapping from target repo id to bucket name.
Expand All @@ -481,7 +486,7 @@ The `upload_policy` defines what policy is used for uploading built artefacts to
|`upload_policy` value|Policy|
|:--------|:--------------------------------|
|`all`|Upload all artefacts (mulitple uploads of the same artefact possible).|
|`latest`|For each build target (prefix in tarball name `eessi-VERSION-{software,init,compat}-OS-ARCH)` only upload the latest built artefact.|
|`latest`|For each build target (prefix in artefact name `eessi-VERSION-{software,init,compat}-OS-ARCH)` only upload the latest built artefact.|
|`once`|Only once upload any built artefact for the build target.|
|`none`|Do not upload any built artefacts.|

Expand All @@ -496,30 +501,30 @@ deployment), or a space delimited list of GitHub accounts.
no_deploy_permission_comment = Label `bot:deploy` has been set by user `{deploy_labeler}`, but this person does not have permission to trigger deployments
```
This defines a message that is added to the status table in a PR comment
corresponding to a job whose tarball should have been uploaded (e.g., after
corresponding to a job whose artefact should have been uploaded (e.g., after
setting the `bot:deploy` label).


```
metadata_prefix = LOCATION_WHERE_METADATA_FILE_GETS_DEPOSITED
tarball_prefix = LOCATION_WHERE_TARBALL_GETS_DEPOSITED
artefact_prefix = LOCATION_WHERE_TARBALL_GETS_DEPOSITED
```

These two settings are used to define where (which directory) in the S3 bucket
(see `bucket_name` above) the metadata file and the tarball will be stored. The
(see `bucket_name` above) the metadata file and the artefact will be stored. The
value `LOCATION...` can be a string value to always use the same 'prefix'
regardless of the target CVMFS repository, or can be a mapping of a target
repository id (see also `repo_target_map` below) to a prefix.

The prefix itself can use some (environment) variables that are set within
the upload script (see `tarball_upload_script` above). Currently those are:
the upload script (see `artefact_upload_script` above). Currently those are:
* `'${github_repository}'` (which would be expanded to the full name of the GitHub
repository, e.g., `EESSI/software-layer`),
* `'${legacy_aws_path}'` (which expands to the legacy/old prefix being used for
storing tarballs/metadata files, the old prefix is
storing artefacts/metadata files, the old prefix is
`EESSI_VERSION/TARBALL_TYPE/OS_TYPE/CPU_ARCHITECTURE/TIMESTAMP/`), _and_
* `'${pull_request_number}'` (which would be expanded to the number of the pull
request from which the tarball originates).
request from which the artefact originates).
Note, it's important to single-quote (`'`) the variables as shown above, because
they may likely not be defined when the bot calls the upload script.

Expand All @@ -529,7 +534,7 @@ The list of supported variables can be shown by running
**Examples:**
```
metadata_prefix = {"eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"}
tarball_prefix = {
artefact_prefix = {
"eessi-pilot-2023.06": "",
"eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"
}
Expand Down Expand Up @@ -656,46 +661,6 @@ running_job = job `{job_id}` is running
#### `[finished_job_comments]` section

The `[finished_job_comments]` section sets templates for messages about finished jobs.
```
success = :grin: SUCCESS tarball `{tarball_name}` ({tarball_size} GiB) in job dir
```
`success` specifies the message for a successful job that produced a tarball.

```
failure = :cry: FAILURE
```
`failure` specifies the message for a failed job.

```
no_slurm_out = No slurm output `{slurm_out}` in job dir
```
`no_slurm_out` specifies the message for missing Slurm output file.

```
slurm_out = Found slurm output `{slurm_out}` in job dir
```
`slurm_out` specifies the message for found Slurm output file.

```
missing_modules = Slurm output lacks message "No missing modules!".
```
`missing_modules` is used to signal the lack of a message that all modules were built.

```
no_tarball_message = Slurm output lacks message about created tarball.
```
`no_tarball_message` is used to signal the lack of a message about a created tarball.

```
no_matching_tarball = No tarball matching `{tarball_pattern}` found in job dir.
```
`no_matching_tarball` is used to signal a missing tarball.

```
multiple_tarballs = Found {num_tarballs} tarballs in job dir - only 1 matching `{tarball_pattern}` expected.
```
`multiple_tarballs` is used to report that multiple tarballs have been found.

```
job_result_unknown_fmt = <details><summary>:shrug: UNKNOWN _(click triangle for details)_</summary><ul><li>Job results file `{filename}` does not exist in job directory, or parsing it failed.</li><li>No artefacts were found/reported.</li></ul></details>
```
Expand Down
22 changes: 7 additions & 15 deletions app.cfg.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ no_build_permission_comment = Label `bot:build` has been set by user `{build_lab

[deploycfg]
# script for uploading built software packages
tarball_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging
artefact_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging

# URL to S3/minio bucket
# if attribute is set, bucket_base will be constructed as follows
Expand Down Expand Up @@ -160,11 +160,11 @@ upload_policy = once
# value can be a space delimited list of GH accounts
deploy_permission =

# template for comment when user who set a label has no permission to trigger deploying tarballs
# template for comment when user who set a label has no permission to trigger deploying artefacts
no_deploy_permission_comment = Label `bot:deploy` has been set by user `{deploy_labeler}`, but this person does not have permission to trigger deployments

# settings for where (directory) in the S3 bucket to store the metadata file and
# the tarball
# the artefact
# - Can be a string value to always use the same 'prefix' regardless of the target
# CVMFS repository, or can be a mapping of a target repository id (see also
# repo_target_map) to a prefix.
Expand All @@ -173,17 +173,17 @@ no_deploy_permission_comment = Label `bot:deploy` has been set by user `{deploy_
# * 'github_repository' (which would be expanded to the full name of the GitHub
# repository, e.g., 'EESSI/software-layer'),
# * 'legacy_aws_path' (which expands to the legacy/old prefix being used for
# storing tarballs/metadata files) and
# storing artefacts/metadata files) and
# * 'pull_request_number' (which would be expanded to the number of the pull
# request from which the tarball originates).
# request from which the artefact originates).
# - The list of supported variables can be shown by running
# `scripts/eessi-upload-to-staging --list-variables`.
# - Examples:
# metadata_prefix = {"eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"}
# tarball_prefix = {"eessi-pilot-2023.06": "", "eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"}
# artefact_prefix = {"eessi-pilot-2023.06": "", "eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"}
# If left empty, the old/legacy prefix is being used.
metadata_prefix =
tarball_prefix =
artefact_prefix =


[architecturetargets]
Expand Down Expand Up @@ -247,14 +247,6 @@ running_job = job `{job_id}` is running


[finished_job_comments]
success = :grin: SUCCESS tarball `{tarball_name}` ({tarball_size} GiB) in job dir
failure = :cry: FAILURE
no_slurm_out = No slurm output `{slurm_out}` in job dir
slurm_out = Found slurm output `{slurm_out}` in job dir
missing_modules = Slurm output lacks message "No missing modules!".
no_tarball_message = Slurm output lacks message about created tarball.
no_matching_tarball = No tarball matching `{tarball_pattern}` found in job dir.
multiple_tarballs = Found {num_tarballs} tarballs in job dir - only 1 matching `{tarball_pattern}` expected.
job_result_unknown_fmt = <details><summary>:shrug: UNKNOWN _(click triangle for detailed information)_</summary><ul><li>Job results file `{filename}` does not exist in job directory, or parsing it failed.</li><li>No artefacts were found/reported.</li></ul></details>
job_test_unknown_fmt = <details><summary>:shrug: UNKNOWN _(click triangle for detailed information)_</summary><ul><li>Job test file `{filename}` does not exist in job directory, or parsing it failed.</li></ul></details>

Expand Down
76 changes: 19 additions & 57 deletions eessi_bot_job_manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,35 +43,23 @@

# Local application imports (anything from EESSI/eessi-bot-software-layer)
from connections import github
from tools import config, run_cmd
from tools import config, job_metadata, run_cmd
from tools.args import job_manager_parse
from tools.job_metadata import read_job_metadata_from_file, read_metadata_file
from tools.pr_comments import get_submitted_job_comment, update_comment


laraPPr marked this conversation as resolved.
Show resolved Hide resolved
AWAITS_LAUNCH = "awaits_launch"
FAILURE = "failure"
FINISHED_JOB_COMMENTS = "finished_job_comments"
JOB_RESULT_COMMENT_DESCRIPTION = "comment_description"
JOB_RESULT_UNKNOWN_FMT = "job_result_unknown_fmt"
JOB_TEST_COMMENT_DESCRIPTION = "comment_description"
JOB_TEST_UNKNOWN_FMT = "job_test_unknown_fmt"
MISSING_MODULES = "missing_modules"
MULTIPLE_TARBALLS = "multiple_tarballs"
NEW_JOB_COMMENTS = "new_job_comments"
NO_MATCHING_TARBALL = "no_matching_tarball"
NO_SLURM_OUT = "no_slurm_out"
NO_TARBALL_MESSAGE = "no_tarball_message"
RUNNING_JOB = "running_job"
RUNNING_JOB_COMMENTS = "running_job_comments"
SLURM_OUT = "slurm_out"
SUCCESS = "success"

REQUIRED_CONFIG = {
FINISHED_JOB_COMMENTS: [FAILURE, JOB_RESULT_UNKNOWN_FMT, MISSING_MODULES,
MULTIPLE_TARBALLS, NO_MATCHING_TARBALL,
NO_SLURM_OUT, NO_TARBALL_MESSAGE, SLURM_OUT,
SUCCESS],
FINISHED_JOB_COMMENTS: [JOB_RESULT_UNKNOWN_FMT, JOB_TEST_UNKNOWN_FMT],
NEW_JOB_COMMENTS: [AWAITS_LAUNCH],
RUNNING_JOB_COMMENTS: [RUNNING_JOB]
}
Expand Down Expand Up @@ -254,42 +242,6 @@ def determine_finished_jobs(self, known_jobs, current_jobs):

return finished_jobs

def read_job_result(self, job_result_file_path):
"""
Read job result file and return the contents of the 'RESULT' section.

Args:
job_result_file_path (string): path to job result file

Returns:
(ConfigParser): instance of ConfigParser corresponding to the
'RESULT' section or None
"""
# reuse function from module tools.job_metadata to read metadata file
result = read_metadata_file(job_result_file_path, self.logfile)
if result and "RESULT" in result:
return result["RESULT"]
else:
return None

def read_job_test(self, job_test_file_path):
"""
Read job test file and return the contents of the 'TEST' section.

Args:
job_test_file_path (string): path to job test file

Returns:
(ConfigParser): instance of ConfigParser corresponding to the
'TEST' section or None
"""
# reuse function from module tools.job_metadata to read metadata file
test = read_metadata_file(job_test_file_path, self.logfile)
if test and "TEST" in test:
return test["TEST"]
else:
return None

def process_new_job(self, new_job):
"""
Process a new job by verifying that it is a bot job and if so
Expand Down Expand Up @@ -335,7 +287,9 @@ def process_new_job(self, new_job):

# assuming that a bot job's working directory contains a metadata
# file, its existence is used to check if the job belongs to the bot
metadata_pr = read_job_metadata_from_file(job_metadata_path, self.logfile)
metadata_pr = job_metadata.get_section_from_file(job_metadata_path,
job_metadata.JOB_PR_SECTION,
self.logfile)

if metadata_pr is None:
log(f"No metadata file found at {job_metadata_path} for job {job_id}, so skipping it",
Expand Down Expand Up @@ -431,7 +385,9 @@ def process_running_jobs(self, running_job):
job_metadata_path = os.path.join(job_dir, metadata_file)

# check if metadata file exist
metadata_pr = read_job_metadata_from_file(job_metadata_path, self.logfile)
metadata_pr = job_metadata.get_section_from_file(job_metadata_path,
job_metadata.JOB_PR_SECTION,
self.logfile)
if metadata_pr is None:
raise Exception("Unable to find metadata file")

Expand Down Expand Up @@ -525,11 +481,13 @@ def process_finished_job(self, finished_job):
# check if _bot_jobJOBID.result exits
job_result_file = f"_bot_job{job_id}.result"
job_result_file_path = os.path.join(new_symlink, job_result_file)
job_results = self.read_job_result(job_result_file_path)
job_results = job_metadata.get_section_from_file(job_result_file_path,
job_metadata.JOB_RESULT_SECTION,
self.logfile)

job_result_unknown_fmt = finished_job_comments_cfg[JOB_RESULT_UNKNOWN_FMT]
# set fallback comment_description in case no result file was found
# (self.read_job_result returned None)
# (job_metadata.get_section_from_file returned None)
comment_description = job_result_unknown_fmt.format(filename=job_result_file)
if job_results:
# get preformatted comment_description or use previously set default for unknown
Expand All @@ -552,11 +510,13 @@ def process_finished_job(self, finished_job):
# --> bot/test.sh and bot/check-test.sh scripts are run in job script used by bot for 'build' action
job_test_file = f"_bot_job{job_id}.test"
job_test_file_path = os.path.join(new_symlink, job_test_file)
job_tests = self.read_job_test(job_test_file_path)
job_tests = job_metadata.get_section_from_file(job_test_file_path,
job_metadata.JOB_TEST_SECTION,
self.logfile)

job_test_unknown_fmt = finished_job_comments_cfg[JOB_TEST_UNKNOWN_FMT]
# set fallback comment_description in case no test file was found
# (self.read_job_result returned None)
# (job_metadata.get_section_from_file returned None)
comment_description = job_test_unknown_fmt.format(filename=job_test_file)
if job_tests:
# get preformatted comment_description or use previously set default for unknown
Expand All @@ -576,7 +536,9 @@ def process_finished_job(self, finished_job):
# obtain id of PR comment to be updated (from file '_bot_jobID.metadata')
metadata_file = f"_bot_job{job_id}.metadata"
job_metadata_path = os.path.join(new_symlink, metadata_file)
metadata_pr = read_job_metadata_from_file(job_metadata_path, self.logfile)
metadata_pr = job_metadata.get_section_from_file(job_metadata_path,
job_metadata.JOB_PR_SECTION,
self.logfile)
if metadata_pr is None:
raise Exception("Unable to find metadata file ... skip updating PR comment")

Expand Down
Loading
Loading