-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implementing conda lock #1164
Merged
Merged
Implementing conda lock #1164
Changes from 32 commits
Commits
Show all changes
77 commits
Select commit
Hold shift + click to select a range
075d1af
Update version number and release date for version 2.3.0
b948b21
added conda lock file
valeriupredoi 4ff602d
added test GH wkflow
valeriupredoi e559cc9
explicitly create the env
valeriupredoi 765d7c2
see what pip is being used
valeriupredoi 1b39e61
detect if env actually activates
valeriupredoi 917ca5c
try create the empty env before the actual deps get in
valeriupredoi 959a504
ga workflow to create and veryfy lock file
valeriupredoi 8b71b8d
run action every 10 days at 4am
valeriupredoi a307b66
Merge branch 'main' into condalock
valeriupredoi 83911e7
fixed syntax in cron
valeriupredoi 53cca74
added osx lock file
valeriupredoi 43b67e3
added documentation
valeriupredoi a47de98
deactivate osx lock file creation and verification
valeriupredoi 3aa2c7b
Merge branch 'main' into condalock
valeriupredoi f0175e6
prettyfied the docs for conda lock
valeriupredoi 5b6629a
reactivate conda lock file creation in actions
bvreede e5b86ab
Merge branch 'condalock' of github.com:ESMValGroup/ESMValCore into co…
bvreede d3da6ea
try a dict approach
valeriupredoi 6cc3236
try with if
valeriupredoi 9be4e86
reindent
valeriupredoi 90a5d0b
try conditional at step level
valeriupredoi 64c5260
set env variable
valeriupredoi b067050
set env upstream and delete linux file setter
valeriupredoi 9119556
try again with modern syntax gahh
valeriupredoi 171a703
try using it as sys variable
valeriupredoi 6b64ab0
export only inside step
valeriupredoi 867be9a
indent, GA are very silly in places
valeriupredoi 9e9c3ce
seriously running out of ideas wo rewriting the whole thing
valeriupredoi 32f0231
put it in one step
valeriupredoi cb79514
sod it, separate job it is then
valeriupredoi d40418f
Merge branch 'main' into condalock
valeriupredoi a1fa826
suggestion by Barbara
valeriupredoi 83aff2e
Merge branch 'main' into condalock
valeriupredoi 80dc847
renamed file
valeriupredoi 5b18479
renamed file
valeriupredoi 3c3fb80
olde file
valeriupredoi d99a833
olde file
valeriupredoi 96e071e
regenerated file
valeriupredoi 90f5624
regenerated file
valeriupredoi 65e6abf
use mamba
valeriupredoi 77c8a39
use mamba
valeriupredoi 7d9fb60
set use mamba true
valeriupredoi bd1c90b
remove pin on esmpy
valeriupredoi b3dba41
remove pin on esmpy
valeriupredoi 46ef0fe
use miniconda proper
valeriupredoi 276693c
try with lock file directly
valeriupredoi 5309a64
back to mamma conda
valeriupredoi 5cf0d7c
back to mamma conda
valeriupredoi 49b56ce
remove py36
valeriupredoi 8ae965e
remove py36 and try with update on OSX
valeriupredoi 5bf264c
turned of osx support for now
valeriupredoi 69dcde5
test automated PR
valeriupredoi 42d58ec
twear auto pr
valeriupredoi 4965cc9
try github token
valeriupredoi a04cd14
added link to GA source
valeriupredoi c21137a
cleanup auto PR job
valeriupredoi b49124f
cleanup auto PR job
valeriupredoi 5668bdc
set run priority and correct branch name
valeriupredoi 2e00eb9
changed PR branch name
valeriupredoi a43bd89
Merge branch 'main' into condalock
valeriupredoi e7c315b
removed osx condalock file
valeriupredoi d0f4758
overhauled the doc
valeriupredoi 667c9b3
read pin
valeriupredoi b1a2f28
readd pin on esmpy
valeriupredoi 2699dc6
re-add HPC instructions and add mention why conda lock
valeriupredoi ae0fbd6
Merge branch 'main' into condalock
valeriupredoi 9d7fa8d
removed OSX bits for good
valeriupredoi 8f9428e
fix issue with file not being reused between jobs
valeriupredoi a1c7d77
removed OX stuffs
valeriupredoi e27a20a
all steps in one flow
valeriupredoi b78491c
remove obsolete comment and add python 3.10
valeriupredoi 69b41b0
quotes around floats for python versions how silly
valeriupredoi c43a79c
Update .github/workflows/create-condalock-file.yml
valeriupredoi 87a0322
Update doc/quickstart/install.rst
valeriupredoi 682c684
Merge branch 'main' into condalock
valeriupredoi b3333cf
remove feature branch for GA test
valeriupredoi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| name: Conda lock file creation | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| - condalock | ||
| schedule: | ||
| - cron: '0 4 */10 * *' | ||
|
|
||
| jobs: | ||
| create-lock-file: | ||
| name: Create conda lock file for latest Python | ||
| runs-on: 'ubuntu-latest' | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - uses: conda-incubator/setup-miniconda@v2 | ||
| with: | ||
| auto-update-conda: true | ||
| python-version: "3.9" | ||
| miniconda-version: "latest" | ||
| channels: conda-forge | ||
| - name: Show conda config | ||
| shell: bash -l {0} | ||
| run: | | ||
| conda info | ||
| conda list | ||
| conda config --show-sources | ||
| conda config --show | ||
| - name: Python info | ||
| shell: bash -l {0} | ||
| run: | | ||
| which python | ||
| python --version | ||
| - name: Install conda-lock | ||
| shell: bash -l {0} | ||
| run: conda install -y conda-lock | ||
| - name: Create conda lock file for linux-64 | ||
| shell: bash -l {0} | ||
| run: conda-lock lock --platform linux-64 -f environment.yml --mamba | ||
| - name: Create conda lock file for osx-64 | ||
| shell: bash -l {0} | ||
| run: conda-lock lock --platform osx-64 -f environment.yml --mamba | ||
| verify-linux: | ||
| name: verify / Linux / python-${{ matrix.python-version }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python-version: ["3.6", "3.7", "3.8", "3.9"] | ||
| runs-on: 'ubuntu-latest' | ||
| needs: create-lock-file | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - uses: conda-incubator/setup-miniconda@v2 | ||
| with: | ||
| activate-environment: esmvaltool-fromlock | ||
| python-version: ${{ matrix.python-version }} | ||
| miniconda-version: "latest" | ||
| channels: conda-forge | ||
| - shell: bash -l {0} | ||
| run: conda --version | ||
| - shell: bash -l {0} | ||
| run: which python | ||
| - shell: bash -l {0} | ||
| run: python -V | ||
| - shell: bash -l {0} | ||
| run: conda create --name esmvaltool-fromlock --file conda-linux-64.lock | ||
| - shell: bash -l {0} | ||
| run: conda install pip | ||
| - shell: bash -l {0} | ||
| run: which python | ||
| - shell: bash -l {0} | ||
| run: pip --version | ||
| - shell: bash -l {0} | ||
| run: pip install -e .[develop] | ||
| - shell: bash -l {0} | ||
| run: esmvaltool --help | ||
| - shell: bash -l {0} | ||
| run: esmvaltool version | ||
| - shell: bash -l {0} | ||
| run: pytest -n 2 -m "not installation and not sequential" | ||
| - shell: bash -l {0} | ||
| run: pytest -n 0 -m "sequential" | ||
| verify-osx: | ||
| name: verify / OSX / python-${{ matrix.python-version }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python-version: ["3.6", "3.7", "3.8", "3.9"] | ||
| runs-on: 'macos-latest' | ||
| needs: create-lock-file | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - uses: conda-incubator/setup-miniconda@v2 | ||
| with: | ||
| activate-environment: esmvaltool-fromlock | ||
| python-version: ${{ matrix.python-version }} | ||
| miniconda-version: "latest" | ||
| channels: conda-forge | ||
| - shell: bash -l {0} | ||
| run: conda --version | ||
| - shell: bash -l {0} | ||
| run: which python | ||
| - shell: bash -l {0} | ||
| run: python -V | ||
| - shell: bash -l {0} | ||
| run: conda create --name esmvaltool-fromlock --file conda-osx-64.lock | ||
| - shell: bash -l {0} | ||
| run: conda install pip | ||
| - shell: bash -l {0} | ||
| run: which python | ||
| - shell: bash -l {0} | ||
| run: pip --version | ||
| - shell: bash -l {0} | ||
| run: pip install -e .[develop] | ||
| - shell: bash -l {0} | ||
| run: esmvaltool --help | ||
| - shell: bash -l {0} | ||
| run: esmvaltool version | ||
| - shell: bash -l {0} | ||
| run: pytest -n 2 -m "not installation and not sequential" | ||
| - shell: bash -l {0} | ||
| run: pytest -n 0 -m "sequential" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| # Install esmvalcore from a conda lock file | ||
| # To build a conda lock file install conda-lock first then | ||
| # run conda-lock lock --platform linux-64 -f environment.yml --mamba | ||
| # (mamba activated for speed). Change platform for osx-64 or win-64. | ||
| # Env creation then happens as per normal use with | ||
| # conda create --name esmvaltool-fromlock --file conda-linux-64.lock | ||
| # note that pip and conda are NOT installed. | ||
|
|
||
| name: Conda-lock Install | ||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| - condalock | ||
| # run the test only if the PR is to main | ||
| # turn it on if required | ||
| #pull_request: | ||
| # branches: | ||
| # - main | ||
| schedule: | ||
| - cron: '0 0 * * *' | ||
|
|
||
| jobs: | ||
| linux: | ||
| runs-on: "ubuntu-latest" | ||
| strategy: | ||
| matrix: | ||
| python-version: [3.6, 3.7, 3.8, 3.9] | ||
| fail-fast: false | ||
| name: Linux Python ${{ matrix.python-version }} | ||
| steps: | ||
| - uses: actions/checkout@v2 | ||
| - uses: conda-incubator/setup-miniconda@v2 | ||
| with: | ||
| activate-environment: esmvaltool-fromlock | ||
| # environment-file: conda-linux-64.lock | ||
| python-version: ${{ matrix.python-version }} | ||
| miniconda-version: "latest" | ||
| channels: conda-forge | ||
| - shell: bash -l {0} | ||
| run: mkdir -p source_install_linux_artifacts_python_${{ matrix.python-version }} | ||
| - shell: bash -l {0} | ||
| run: conda --version 2>&1 | tee source_install_linux_artifacts_python_${{ matrix.python-version }}/conda_version.txt | ||
| - shell: bash -l {0} | ||
| run: which python | ||
| - shell: bash -l {0} | ||
| run: python -V 2>&1 | tee source_install_linux_artifacts_python_${{ matrix.python-version }}/python_version.txt | ||
| - shell: bash -l {0} | ||
| run: conda create --name esmvaltool-fromlock --file conda-linux-64.lock | ||
| - shell: bash -l {0} | ||
| run: conda install pip | ||
| - shell: bash -l {0} | ||
| run: which python | ||
| - shell: bash -l {0} | ||
| run: pip --version | ||
| - shell: bash -l {0} | ||
| run: pip install -e .[develop] | ||
| - shell: bash -l {0} | ||
| run: esmvaltool --help | ||
| - shell: bash -l {0} | ||
| run: esmvaltool version 2>&1 | tee source_install_linux_artifacts_python_${{ matrix.python-version }}/version.txt | ||
| - shell: bash -l {0} | ||
| run: pytest -n 2 -m "not installation and not sequential" | ||
| - shell: bash -l {0} | ||
| run: pytest -n 0 -m "sequential" | ||
| - name: Upload artifacts | ||
| if: ${{ always() }} # upload artifacts even if fail | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: Source_Install_Linux_python_${{ matrix.python-version }} | ||
| path: source_install_linux_artifacts_python_${{ matrix.python-version }} | ||
|
|
||
| # osx: | ||
| # runs-on: "macos-latest" | ||
| # strategy: | ||
| # matrix: | ||
| # python-version: [3.6, 3.7, 3.8, 3.9] | ||
| # fail-fast: false | ||
| # name: OSX Python ${{ matrix.python-version }} | ||
| # steps: | ||
| # - uses: actions/checkout@v2 | ||
| # - uses: conda-incubator/setup-miniconda@v2 | ||
| # with: | ||
| # activate-environment: esmvaltool | ||
| # environment-file: environment.yml | ||
| # python-version: ${{ matrix.python-version }} | ||
| # miniconda-version: "latest" | ||
| # channels: conda-forge | ||
| # - shell: bash -l {0} | ||
| # run: mkdir -p source_install_osx_artifacts_python_${{ matrix.python-version }} | ||
| # - shell: bash -l {0} | ||
| # run: conda --version 2>&1 | tee source_install_osx_artifacts_python_${{ matrix.python-version }}/conda_version.txt | ||
| # - shell: bash -l {0} | ||
| # run: python -V 2>&1 | tee source_install_osx_artifacts_python_${{ matrix.python-version }}/python_version.txt | ||
| # - shell: bash -l {0} | ||
| # run: pip install -e .[develop] 2>&1 | tee source_install_osx_artifacts_python_${{ matrix.python-version }}/install.txt | ||
| # - shell: bash -l {0} | ||
| # run: esmvaltool --help | ||
| # - shell: bash -l {0} | ||
| # run: esmvaltool version 2>&1 | tee source_install_osx_artifacts_python_${{ matrix.python-version }}/version.txt | ||
| # - name: Upload artifacts | ||
| # if: ${{ always() }} # upload artifacts even if fail | ||
| # uses: actions/upload-artifact@v2 | ||
| # with: | ||
| # name: Source_Install_OSX_python_${{ matrix.python-version }} | ||
| # path: source_install_osx_artifacts_python_${{ matrix.python-version }} | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is still commented out?