Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: use custom docker images for services #92

Merged
merged 4 commits into from
Jun 27, 2023
Merged

feat: use custom docker images for services #92

merged 4 commits into from
Jun 27, 2023

Conversation

Ian2012
Copy link
Contributor

@Ian2012 Ian2012 commented Jun 26, 2023 •
edited
Loading

Description

This PR defines custom docker images for the aspects and superset services.

The scripts have been unified per service.

Automatically bump version and release based on conventional commits.

Important notice:

eduNEXT will be in charge of hosting and keeping the images up to date.

@openedx-webhooks openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Jun 26, 2023
@openedx-webhooks
Copy link

openedx-webhooks commented Jun 26, 2023 •
edited
Loading

Thanks for the pull request, @Ian2012! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

  • supporting documentation
  • Open edX discussion forum threads
  • timeline information ("this must be merged by XX date", and why that is)
  • partner information ("this is a course on edx.org")
  • any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

@Ian2012 Ian2012 marked this pull request as draft June 26, 2023 18:47
@Ian2012 Ian2012 requested review from bmtcril and pomegranited June 26, 2023 18:53
@Ian2012 Ian2012 force-pushed the cag/custom-docker-images branch from b389e9c to 6b5cc58 Compare June 26, 2023 20:36
@Ian2012 Ian2012 marked this pull request as ready for review June 26, 2023 21:22
@Ian2012 Ian2012 force-pushed the cag/custom-docker-images branch 2 times, most recently from d04c469 to 44df5e0 Compare June 26, 2023 22:32
@Ian2012 Ian2012 changed the title Custom docker images feat: use custom docker images for services Jun 26, 2023
@itsjeyd itsjeyd added the core contributor PR author is a Core Contributor (who may or may not have write access to this repo). label Jun 27, 2023
@itsjeyd
Copy link

itsjeyd commented Jun 27, 2023

Thanks for the changes @Ian2012, are they ready for review?

@itsjeyd itsjeyd added the waiting on author PR author needs to resolve review requests, answer questions, fix tests, etc. label Jun 27, 2023
@Ian2012 Ian2012 force-pushed the cag/custom-docker-images branch from 44df5e0 to 6a87aee Compare June 27, 2023 11:47
@Ian2012
Copy link
Contributor Author

Ian2012 commented Jun 27, 2023 •
edited
Loading

@bmtcril Have in mind that once this is branch is merged the workflow will create:

  • A tag v.0.1.0
  • A release in github, which will trigger the pipy publish CI workflow.
  • Commit the changelog automatically generated.

@Ian2012 Ian2012 force-pushed the cag/custom-docker-images branch 4 times, most recently from aea5bff to bf01304 Compare June 27, 2023 12:33
@Ian2012
Copy link
Contributor Author

Ian2012 commented Jun 27, 2023

@bmtcril this is ready for review now

@Ian2012 Ian2012 force-pushed the cag/custom-docker-images branch from bf01304 to d86f3a1 Compare June 27, 2023 13:05
bmtcril
bmtcril approved these changes Jun 27, 2023
View reviewed changes
Copy link
Contributor

@bmtcril bmtcril left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worked well for me, though I didn't try a fresh install.

Timing with the changes:
Main branch, cold: 5m35s
Main branch, cached: 5m13s
This branch, cold: 5m18s
This branch, cached: 4m5s

Pretty good improvement. I think the big bulk of the time is in Superset inits now. I don't have any ideas yet on how to speed that up. 🤔

tutoraspects/templates/aspects/apps/scripts/aspects/clickhouse-demo-xapi-data.sh
@@ -4,8 +4,6 @@ python3 -m venv virtualenv
. virtualenv/bin/activate

echo "Loading demo xAPI data..."
pip install git+https://github.com/openedx/[email protected]#egg=xapi-db-load==0.2
pip install pandas # clickhouse_connect is missing this
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to review this it seems to be working now without pandas: openedx/xapi-db-load#32 we can upgrade this to 0.3 after this PR.

tutoraspects/templates/aspects/build/aspects/Dockerfile Outdated
@@ -0,0 +1,5 @@
FROM python:3.8

RUN pip install dbt-core==1.4.0 dbt-clickhouse==1.4.1 pandas
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can get rid of pandas here now? Or maybe dbt-clickhouse pins an older version of clickhouse-connect. 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(dbt-clickhouse does not seem to be pinning, so I think removing it will be ok)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed and built

bmtcril
bmtcril reviewed Jun 27, 2023
View reviewed changes
.github/workflows/release.yaml Outdated
uses: stefanzweifel/[email protected]
with:
branch: ${{ github.ref }}
commit_message: "chore(release): preparing ${{ steps.tag_version.outputs.new_version }}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think actions will run on this, but the commit message doesn't match the conventional commits and should just start with "chore:".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@Ian2012 Ian2012 force-pushed the cag/custom-docker-images branch from d86f3a1 to beee179 Compare June 27, 2023 15:06
@Ian2012 Ian2012 force-pushed the cag/custom-docker-images branch from beee179 to d8ff56c Compare June 27, 2023 15:07
@bmtcril bmtcril merged commit 33c6dd3 into main Jun 27, 2023
@bmtcril bmtcril deleted the cag/custom-docker-images branch June 27, 2023 16:05
@openedx-webhooks
Copy link

@Ian2012 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

@pomegranited
Copy link
Contributor

@Ian2012 CC @bmtcril I did a fresh install of the latest main after this merged, and it worked beautifully. So good!

@itsjeyd itsjeyd removed the waiting on author PR author needs to resolve review requests, answer questions, fix tests, etc. label Aug 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bmtcril bmtcril bmtcril approved these changes

@pomegranited pomegranited pomegranited left review comments

Assignees
No one assigned
Labels
core contributor PR author is a Core Contributor (who may or may not have write access to this repo). open-source-contribution PR author is not from Axim or 2U
Projects
Contributions
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Use custom docker images and CI for pushing images
5 participants
@Ian2012 @openedx-webhooks @itsjeyd @pomegranited @bmtcril