Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(bootstrap.sh): bump the mariadb version from 11.1 to 11.2 #12

Closed
wants to merge 1 commit into from

Conversation

dkreth
Copy link

@dkreth dkreth commented Dec 18, 2024

version 11.1 no longer exists on the mirror server we're using, which causes the bootstrap.sh script to fail (on the 404), which causes the ./mwaa-local-env start command to fail. This can be resolved by using version 11.2 instead.

I chose 11.2 because it was the next minor version available and worked for me locally, but it may make sense to use the most recent minor version (11.7) instead -- I just haven't tested that yet.

Summary by CodeRabbit

  • Chores
    • Updated the installation process for MariaDB to version 11.2.2.
    • Added installation steps for dnf-plugins-core and building dependencies for Python 3.
    • Continued support for Python 3.11 and maintained the upgrade process for pip and necessary dependencies for Apache Airflow.

+ 11.1 no longer exists on the mirror server we're using, which causes the bootstrap.sh script to fail (on the 404), which causes the `./mwaa-local-env start` command to fail
@dkreth dkreth requested review from ronco and G00SE-EGG December 18, 2024 18:53
@dkreth dkreth self-assigned this Dec 18, 2024
Copy link

coderabbitai bot commented Dec 18, 2024

Walkthrough

The pull request modifies the docker/script/bootstrap.sh script, focusing on updating the MariaDB package installation process. The script now downloads MariaDB version 11.2.2 for both aarch64 and amd64 architectures, replacing the previous 11.1.2 version. A new section for installing dnf-plugins-core and Python build dependencies has been added, while maintaining the existing Python 3.11 installation process and Airflow dependency setup.

Changes

File Change Summary
docker/script/bootstrap.sh - Updated MariaDB package version from 11.1.2 to 11.2.2
- Added installation of dnf-plugins-core
- Included new section for Python build dependencies

Poem

🐰 A Bootstrap Script's Delight

Marching forward with MariaDB's might,
Version 11.2.2 takes flight!
Python dependencies now in view,
Dnf plugins joining the crew
Our Docker script, sleek and bright! 🚀


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
docker/script/bootstrap.sh (2)

56-62: Consider quoting $(uname -p) to prevent word splitting

While the current implementation works, it's best practice to quote command substitutions to prevent word splitting and potential issues with special characters.

Apply this change to all affected lines:

-  wget https://mirror.mariadb.org/yum/11.2/fedora38-aarch64/rpms/MariaDB-common-11.2.2-1.fc38.$(uname -p).rpm -P /mariadb_rpm
+  wget https://mirror.mariadb.org/yum/11.2/fedora38-aarch64/rpms/MariaDB-common-11.2.2-1.fc38."$(uname -p)".rpm -P /mariadb_rpm
🧰 Tools
🪛 Shellcheck (0.10.0)

[warning] 56-56: Quote this to prevent word splitting.

(SC2046)


[warning] 57-57: Quote this to prevent word splitting.

(SC2046)


[warning] 58-58: Quote this to prevent word splitting.

(SC2046)


[warning] 60-60: Quote this to prevent word splitting.

(SC2046)


[warning] 61-61: Quote this to prevent word splitting.

(SC2046)


[warning] 62-62: Quote this to prevent word splitting.

(SC2046)


56-62: Consider using variables for version numbers

To make future version updates easier to manage, consider extracting the version number into a variable at the top of the script.

#!/bin/sh

set -e
set -x

+ MARIADB_VERSION="11.2"
+ MARIADB_RELEASE="11.2.2-1.fc38"

# ... rest of the script ...

if [[ $(uname -p) == "aarch64" ]]; then
-  wget https://mirror.mariadb.org/yum/11.2/fedora38-aarch64/rpms/MariaDB-common-11.2.2-1.fc38.$(uname -p).rpm -P /mariadb_rpm
+  wget "https://mirror.mariadb.org/yum/${MARIADB_VERSION}/fedora38-aarch64/rpms/MariaDB-common-${MARIADB_RELEASE}.$(uname -p).rpm" -P /mariadb_rpm
🧰 Tools
🪛 Shellcheck (0.10.0)

[warning] 56-56: Quote this to prevent word splitting.

(SC2046)


[warning] 57-57: Quote this to prevent word splitting.

(SC2046)


[warning] 58-58: Quote this to prevent word splitting.

(SC2046)


[warning] 60-60: Quote this to prevent word splitting.

(SC2046)


[warning] 61-61: Quote this to prevent word splitting.

(SC2046)


[warning] 62-62: Quote this to prevent word splitting.

(SC2046)

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6029be1 and c33fa10.

📒 Files selected for processing (1)
  • docker/script/bootstrap.sh (1 hunks)
🧰 Additional context used
🪛 Shellcheck (0.10.0)
docker/script/bootstrap.sh

[warning] 56-56: Quote this to prevent word splitting.

(SC2046)


[warning] 57-57: Quote this to prevent word splitting.

(SC2046)


[warning] 58-58: Quote this to prevent word splitting.

(SC2046)


[warning] 60-60: Quote this to prevent word splitting.

(SC2046)


[warning] 61-61: Quote this to prevent word splitting.

(SC2046)


[warning] 62-62: Quote this to prevent word splitting.

(SC2046)

Comment on lines +56 to +62
wget https://mirror.mariadb.org/yum/11.2/fedora38-aarch64/rpms/MariaDB-common-11.2.2-1.fc38.$(uname -p).rpm -P /mariadb_rpm
wget https://mirror.mariadb.org/yum/11.2/fedora38-aarch64/rpms/MariaDB-shared-11.2.2-1.fc38.$(uname -p).rpm -P /mariadb_rpm
wget https://mirror.mariadb.org/yum/11.2/fedora38-aarch64/rpms/MariaDB-devel-11.2.2-1.fc38.$(uname -p).rpm -P /mariadb_rpm
else
wget https://mirror.mariadb.org/yum/11.1/fedora38-amd64/rpms/MariaDB-common-11.1.2-1.fc38.$(uname -p).rpm -P /mariadb_rpm
wget https://mirror.mariadb.org/yum/11.1/fedora38-amd64/rpms/MariaDB-shared-11.1.2-1.fc38.$(uname -p).rpm -P /mariadb_rpm
wget https://mirror.mariadb.org/yum/11.1/fedora38-amd64/rpms/MariaDB-devel-11.1.2-1.fc38.$(uname -p).rpm -P /mariadb_rpm
wget https://mirror.mariadb.org/yum/11.2/fedora38-amd64/rpms/MariaDB-common-11.2.2-1.fc38.$(uname -p).rpm -P /mariadb_rpm
wget https://mirror.mariadb.org/yum/11.2/fedora38-amd64/rpms/MariaDB-shared-11.2.2-1.fc38.$(uname -p).rpm -P /mariadb_rpm
wget https://mirror.mariadb.org/yum/11.2/fedora38-amd64/rpms/MariaDB-devel-11.2.2-1.fc38.$(uname -p).rpm -P /mariadb_rpm
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codebase verification

Consider upgrading to MariaDB 11.6.1 for better long-term maintenance

The verification shows that version 11.2.2 packages are being redirected (302 Found), while 11.6.1 is the latest available version in both architectures with direct downloads. For better maintainability, consider updating to version 11.6.1:

  • Replace 11.2.2 with 11.6.1 in the package URLs
  • Both aarch64 and amd64 architectures have consistent package availability
  • Version 11.7 directory exists but doesn't contain the required packages yet

Example update for aarch64:

https://mirror.mariadb.org/yum/11.6/fedora38-aarch64/rpms/MariaDB-common-11.6.1-1.fc38.$(uname -p).rpm
🔗 Analysis chain

Verify MariaDB version selection and availability

The PR mentions version 11.7 as a potential alternative. Let's verify the availability and stability of different versions.

While version 11.2.2 resolves the immediate 404 error, we should confirm if using the latest stable version (11.7) would be more beneficial for long-term maintenance.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check available MariaDB versions and their availability on the mirror
# Test both architectures to ensure consistent availability

echo "Checking aarch64 versions..."
curl -sI "https://mirror.mariadb.org/yum/11.2/fedora38-aarch64/rpms/" | grep "HTTP/"
curl -sI "https://mirror.mariadb.org/yum/11.7/fedora38-aarch64/rpms/" | grep "HTTP/"

echo "Checking amd64 versions..."
curl -sI "https://mirror.mariadb.org/yum/11.2/fedora38-amd64/rpms/" | grep "HTTP/"
curl -sI "https://mirror.mariadb.org/yum/11.7/fedora38-amd64/rpms/" | grep "HTTP/"

# Check if these are the latest stable versions
curl -s "https://downloads.mariadb.org/rest-api/mariadb/" | jq -r '.[] | select(.release_status=="stable") | .release_id'

Length of output: 721


Script:

#!/bin/bash
# Let's check the actual package availability and versions for both architectures
echo "Checking aarch64 packages..."
curl -s "https://mirror.mariadb.org/yum/11.7/fedora38-aarch64/rpms/" | grep -o 'MariaDB-\(common\|shared\|devel\)-[0-9.]\+-[0-9]\.fc38\.aarch64\.rpm' || true

echo -e "\nChecking amd64 packages..."
curl -s "https://mirror.mariadb.org/yum/11.7/fedora38-amd64/rpms/" | grep -o 'MariaDB-\(common\|shared\|devel\)-[0-9.]\+-[0-9]\.fc38\.x86_64\.rpm' || true

# Let's also check if the current 11.2.2 packages are still available
echo -e "\nVerifying current 11.2.2 packages..."
for arch in "aarch64" "amd64"; do
  for pkg in "common" "shared" "devel"; do
    curl -sI "https://mirror.mariadb.org/yum/11.2/fedora38-${arch}/rpms/MariaDB-${pkg}-11.2.2-1.fc38.$([ "$arch" = "amd64" ] && echo "x86_64" || echo "aarch64").rpm" | grep "HTTP/"
  done
done

Length of output: 3360

🧰 Tools
🪛 Shellcheck (0.10.0)

[warning] 56-56: Quote this to prevent word splitting.

(SC2046)


[warning] 57-57: Quote this to prevent word splitting.

(SC2046)


[warning] 58-58: Quote this to prevent word splitting.

(SC2046)


[warning] 60-60: Quote this to prevent word splitting.

(SC2046)


[warning] 61-61: Quote this to prevent word splitting.

(SC2046)


[warning] 62-62: Quote this to prevent word splitting.

(SC2046)

@ronco
Copy link

ronco commented Dec 18, 2024

This repo is a fork of an open source project (https://github.com/aws/aws-mwaa-local-runner). It looks like they fixed this issues 3 weeks ago: aws#414. I wonder if the better solve is to merge or rebase the latest from the upstream repo.

Copy link
Author

dkreth commented Dec 18, 2024

@ronco

  1. That's a good thought. I created a draft PR that brings in the changes from the upstream; I put them in this PR for an easy way to visualize the diff. But there are quite a few changes.
  2. Because there are so many changes, would it be safer (for now) to just cherry-pick the single commit we need? This PR does that.
  3. The most-recent README of the OSS upstream repo mentions that "Starting from Airflow version 2.9, MWAA has open-sourced the original Docker image used in [their] production deployments". Would that effectively eliminate the need for this aws-mwaa-local-runner repo, so long as we upgrade to MWAA 2.9? Or does that not mean what I think it means? Full quote below:

Starting from Airflow version 2.9, MWAA has open-sourced the original Docker image used in our production deployments. You can refer to our open-source image repository at https://github.com/aws/amazon-mwaa-docker-images to create a local environment identical to that of MWAA.
You can also continue to use the MWAA Local Runner for testing and packaging requirements for all Airflow versions supported on MWAA.

What do you think?

Copy link

ronco commented Dec 18, 2024

@dkreth, ah…. Good investigation here. As much as I dislike cherry-picking, I think that might be the way to go until we’re ready to upgrade our MWAA instance to 2.9+.

@dkreth
Copy link
Author

dkreth commented Jan 7, 2025

closed in favor of #14

@dkreth dkreth closed this Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants