You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the project relies on several Python scripts to fetch data from external APIs, parse it, and insert it into the database. These scripts need to be executed automatically each time docker-compose up --build is run. Automating this process will ensure the database is seeded with the latest data without requiring manual intervention, improving the workflow and reducing potential errors.
Definition of Done
All relevant Python scripts are executed automatically when docker-compose up --build is run.
The solution is integrated into the existing Docker Compose setup without disrupting other services.
Engineering Details
Scripts are located in the /backend/etl directory of the project and include: AddressDataHandler, LandslideDataHandler, LiquefactionDataHandler, SeismicDataHandler, SoftStoryPropertiesDataHandler, TsunamiDataHandler.
Docker Compose configuration (docker-compose.yaml) should be updated to include a mechanism for executing the scripts. Options to consider:
Modifications to the existing backend container using entrypoint or command.
A startup.sh script that chains the execution of all Python scripts.
A new container dedicated to running the scripts.
Ensure the solution respects dependencies (e.g., scripts should run only after the database container is ready).
The text was updated successfully, but these errors were encountered:
Blocked by #23
Context
Currently, the project relies on several Python scripts to fetch data from external APIs, parse it, and insert it into the database. These scripts need to be executed automatically each time
docker-compose up --build
is run. Automating this process will ensure the database is seeded with the latest data without requiring manual intervention, improving the workflow and reducing potential errors.Definition of Done
docker-compose up --build
is run.Engineering Details
/backend/etl
directory of the project and include:AddressDataHandler, LandslideDataHandler, LiquefactionDataHandler, SeismicDataHandler, SoftStoryPropertiesDataHandler, TsunamiDataHandler
.docker-compose.yaml
) should be updated to include a mechanism for executing the scripts. Options to consider:startup.sh
script that chains the execution of all Python scripts.The text was updated successfully, but these errors were encountered: