Current Status:

• goal: adding Flask & Postgres to Docker, set up database
  • flask is the api
  • postgres holds all the station data in 2 tables: stations and station_changes
• currently:
  • making requests to flask api from consumer
  • add station history update requests to station add & update
  • create a testintegration service

Resources Used:

• docker tutorial
• Confluent + Docker tutorial
• defining an avro schema
• Processing streamining.io data code example
• Citibike "General Bikeshare Feed Specification" (GBFS) for avro schema
• Docker image and guide for spark, dockerfile and scripts for spark application
• flask/ postgres docker tutorial

Helpful Docker commands:

• docker-compose up, docker-compose down
  • e.g. docker-compose up -d --build
• docker-compose ps: prints status of services
• docker-compose logs
• 'docker inspect web | grep "IPAddress"': gets container ip address
• docker system prune -a use this or run out of storage
• docker-compose run <service-name> bash
  • bash depends on the container (bash has to exist). some just have bin/sh
  • from within the container, can run commands
  • e.g. docker-compose run web python manage.py seed_db
• docker-compose exec <service-name> bash
  • get into an already running service
  • e.g. docker exec -it bikeshare-db psql -U postgres
• if there isn't permission to run something, add chmod
  • chmod +x services/users/entrypoint.sh
• list containers: docker container ls

Helpful database commands:

• recreate the dev database: docker-compose -f docker-compose.yml \ run web python manage.py recreate_db
• get into psql: docker exec -ti $(docker ps -aqf "name=bikeshare-db") psql -U postgres
• run flask tests docker-compose -f docker-compose.yml run web python manage.py test

Helpful flask:

• Use TDD when adding flask routes https://testdriven.io/part-one-restful-routes
• Postman is nice when working on APIs https://www.getpostman.com/

Tips from friends / the internet:

• AWS

  • look at elastic beanstalk or elastic containers
  • it will take patience
  • look into circleci
  • S3 is for static assets
  • use RDS for db

• might want a docker-compose.prod

  • because there could be a different dev database
  • use env variables in docker-compose