Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CI Setup #681

Merged
merged 22 commits into from
Jan 5, 2022
Merged

Improve CI Setup #681

merged 22 commits into from
Jan 5, 2022

Conversation

limdauto
Copy link
Collaborator

@limdauto limdauto commented Jan 5, 2022

Description

  • Use the newer cimg image as opposed to the legacy circleci.
  • Run npm install instead of npm ci to allow node_modules cache to take effect
  • Remove conda in linux build. Just standard python venv is enough.
  • Cache pip & python virtualenvs between builds.
  • Parallelise make pylint

Net result

  • Linux build: 11m25s -> 7m35s (35%)
  • Windows build: 11m47s -> 9m37s (18%)

Before:

Screenshot 2022-01-05 at 16 32 19

After:

Screenshot 2022-01-05 at 16 31 47

Further improvement

Maybe we can run windows test on main only.

Development notes

QA notes

Checklist

  • Read the contributing guidelines
  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added new entries to the RELEASE.md file
  • Added tests to cover my changes

Copy link
Member

@tynandebold tynandebold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. The results speak for themselves. Nice work.

Just one comment about the tools/ci.js file, but I'll approve straight away.

@@ -51,7 +43,7 @@ commands:
steps:
- run:
name: Install Node dependencies
command: node tools/ci.js
command: npm install
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can delete the tools/ci.js file now. Doesn't seem to be used anywhere else.

@limdauto limdauto merged commit 72b47ff into main Jan 5, 2022
@limdauto limdauto deleted the fix/ci-python-cache branch January 5, 2022 17:12
Copy link
Contributor

@studioswong studioswong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice one @limdauto - thanks for setting this up!

Comment on lines -24 to -34
# Get rid of pyenv stuff
sudo rm -rf .pyenv/ /opt/circleci/.pyenv/
# Download and install miniconda
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh > miniconda.sh
bash miniconda.sh -b -p $HOME/miniconda
# Create an anaconda virtualenv for python ${CONDA_ENV_PY_VERSION} and make that the default python interpreter
echo ". /home/circleci/miniconda/etc/profile.d/conda.sh" >> $BASH_ENV
echo "conda deactivate; conda activate kedro-viz" >> $BASH_ENV
. /home/circleci/miniconda/etc/profile.d/conda.sh
conda create --name kedro-viz python=${CONDA_ENV_PY_VERSION} -y
source $BASH_ENV
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@limdauto this is more for my curiosity - I wonder why won't we need conda as the default python interpreter with the new setup? Is it becuase of the switch to use cimg?

rashidakanchwala pushed a commit that referenced this pull request Jan 10, 2022
Signed-off-by: Rashida Kanchwala <[email protected]>
rashidakanchwala pushed a commit that referenced this pull request Jan 11, 2022
Signed-off-by: Rashida Kanchwala <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants