Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Further automate tests #516

Open
forsyth2 opened this issue Sep 28, 2023 · 6 comments · May be fixed by #520
Open

Further automate tests #516

forsyth2 opened this issue Sep 28, 2023 · 6 comments · May be fixed by #520
Labels
Testing Files in `tests` modified

Comments

@forsyth2
Copy link
Collaborator

#440 provides step-by-step directions for testing zppy for release, but can we go further than that? I.e., can we automate all those steps?

@forsyth2 forsyth2 added the Testing Files in `tests` modified label Sep 28, 2023
@forsyth2
Copy link
Collaborator Author

Resources suggested by @tomvothecoder:

CI/CD runners on HPC machines instead of on cloud:

@forsyth2
Copy link
Collaborator Author

forsyth2 commented Oct 5, 2023

Next steps

  • Create a bash script to automate these steps
  • Determine if the script can be run via GitHub Actions

@forsyth2 forsyth2 linked a pull request Oct 5, 2023 that will close this issue
@chengzhuzhang
Copy link
Collaborator

Thank you for getting to it. I think this will benefit in the long run, we can try to get this in for next unified. In terms of time-line, let's try prioritize those user-facing features (i.e. land support, zstash update, cdat migration) the land support needs lots of interaction with science team to finalize.

@forsyth2
Copy link
Collaborator Author

forsyth2 commented Oct 5, 2023

Yes, that timeline sounds good to me. I just wanted to get an idea of what issues we would face here.

@forsyth2
Copy link
Collaborator Author

#628 added 20 unit tests. Integration tests are really where we need further automation. Further action items:

  • Address the TODO items in Automate zppy tests #520. If we can get those steps automated, then we can effectively automate the entire process described at https://docs.e3sm.org/zppy/_build/html/main/dev_guide/release_testing.html
  • Once Automate zppy tests #520 is merged, we'll have a process we can run automatically every week. Ideally, we could set up some sort of cron process to run the Weekly tests weekly (rather than manually running them weekly).
  • Once the weekly tests are running automatically, it would further help to run some of the min-case cfg files automatically as well. Specifically, it would be great to have some sort of "if file was changed in this PR or recent commit history, then run min-case-X cfg" setup.
  • Look into mocking external dependencies rather than relying on real data paths, which can be finicky. Hackathon ideas for zppy #629 describes this further

@forsyth2
Copy link
Collaborator Author

@tomvothecoder noted --y on mamba may allow automatic bypass of the y/n user prompts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Testing Files in `tests` modified
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants