Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate ETL Script Execution on Docker Build #118

Open
agennadi opened this issue Dec 10, 2024 · 0 comments
Open

Automate ETL Script Execution on Docker Build #118

agennadi opened this issue Dec 10, 2024 · 0 comments
Assignees
Labels

Comments

@agennadi
Copy link
Collaborator

Blocked by #23

Context

Currently, the project relies on several Python scripts to fetch data from external APIs, parse it, and insert it into the database. These scripts need to be executed automatically each time docker-compose up --build is run. Automating this process will ensure the database is seeded with the latest data without requiring manual intervention, improving the workflow and reducing potential errors.

Definition of Done

  1. All relevant Python scripts are executed automatically when docker-compose up --build is run.
  2. The solution is integrated into the existing Docker Compose setup without disrupting other services.

Engineering Details

  1. Scripts are located in the /backend/etl directory of the project and include: AddressDataHandler, LandslideDataHandler, LiquefactionDataHandler, SeismicDataHandler, SoftStoryPropertiesDataHandler, TsunamiDataHandler.
  2. Docker Compose configuration (docker-compose.yaml) should be updated to include a mechanism for executing the scripts. Options to consider:
    1. Modifications to the existing backend container using entrypoint or command.
    2. A startup.sh script that chains the execution of all Python scripts.
    3. A new container dedicated to running the scripts.
  3. Ensure the solution respects dependencies (e.g., scripts should run only after the database container is ready).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: In Review
Development

No branches or pull requests

1 participant