Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

20191106 postgres dbapi #432

Merged
merged 4 commits into from
Aug 26, 2020
Merged

20191106 postgres dbapi #432

merged 4 commits into from
Aug 26, 2020

Conversation

peterthomassen
Copy link
Member

No description provided.

@peterthomassen peterthomassen force-pushed the 20191106_postgres_dbapi branch 2 times, most recently from 07fad76 to d032650 Compare August 25, 2020 13:32
@peterthomassen peterthomassen marked this pull request as ready for review August 25, 2020 13:33
@peterthomassen peterthomassen force-pushed the 20191106_postgres_dbapi branch from d032650 to cacb7ec Compare August 25, 2020 14:15
@peterthomassen peterthomassen force-pushed the 20191106_postgres_dbapi branch from cacb7ec to 65e91ec Compare August 25, 2020 14:56
@peterthomassen peterthomassen force-pushed the 20191106_postgres_dbapi branch from 65e91ec to 5c4b999 Compare August 26, 2020 09:27
@peterthomassen
Copy link
Member Author

Migration instructions (tested on sandbox):

  1. docker-compose stop api celery

  2. docker-compose run --rm api python manage.py dumpdata contenttypes --indent=4 --natural-foreign > contenttype.json

  3. docker-compose run --rm api python manage.py dumpdata --exclude contenttypes --exclude desecapi.Donation --exclude desecapi.AuthenticatedAction --exclude desecapi.AuthenticatedBasicUserAction --exclude desecapi.AuthenticatedUserAction --exclude desecapi.AuthenticatedActivateUserAction --exclude desecapi.AuthenticatedChangeEmailUserAction --exclude desecapi.AuthenticatedResetPasswordUserAction --exclude desecapi.AuthenticatedDeleteUserAction --exclude desecapi.AuthenticatedDomainBasicUserAction --exclude desecapi.AuthenticatedRenewDomainBasicUserAction --indent=4 --natural-foreign > everything_else.json

  4. docker-compose stop dbapi

  5. git pull && docker-compose pull # pre-merge: git checkout 20191106_postgres_dbapi && docker-compose build api dbapi

  6. docker-compose up -d dbapi

  7. docker-compose run --rm api bash

  8. in another terminal, run:

    1. docker ps # find container name like desec-stack_api_run_*
    2. docker cp contenttype.json desec-stack_api_run_XXX:/usr/src/app/
    3. docker cp everything_else.json desec-stack_api_run_XXX:/usr/src/app/

  9. back in the api shell, run:

    python manage.py migrate
python manage.py sqlflush | ./manage.py dbshell
python manage.py loaddata contenttype.json
python manage.py loaddata everything_else.json
rm contenttype.json everything_else.json
exit

  10. docker-compose up -d

  11. If all went well: rm contenttype.json everything_else.json

peterthomassen added a commit that referenced this pull request Aug 26, 2020
@peterthomassen
BREAKING feat(dbapi): switch to postgres, fixes #430
1adbf73 
Migration instructions:
#432 (comment)
@peterthomassen peterthomassen force-pushed the 20191106_postgres_dbapi branch from 5c4b999 to 1adbf73 Compare August 26, 2020 10:05
nils-wisiol
nils-wisiol approved these changes Aug 26, 2020
View reviewed changes
Copy link
Contributor

@nils-wisiol nils-wisiol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, but some minor changes are requested

README.md Outdated
@@ -135,10 +135,10 @@ While there are certainly many ways to get started hacking desec-stack, here is
For desec-stack, [docker](https://docs.docker.com/install/linux/docker-ce/ubuntu/) and [docker-compose](https://docs.docker.com/compose/install/) are required.
Further tools that are required to start hacking are git and curl.
Recommended, but not strictly required for desec-stack development is to use certbot along with Let's Encrypt and PyCharm.
jq, httpie, libmariadbclient-dev, python3-dev (>= 3.8) and python3-venv (>= 3.8) are useful if you want to follow this guide.
jq, httpie, libpq-dev, python3-dev (>= 3.8) and python3-venv (>= 3.8) are useful if you want to follow this guide.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

install both clients, mariadb and pq

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

README.md Outdated
The webapp requires nodejs. To install everything you need for this guide except docker and docker-compose, use

sudo apt install certbot curl git httpie jq libmariadbclient-dev nodejs npm python3-dev python3-venv libmemcached-dev
sudo apt install certbot curl git httpie jq libpq-dev nodejs npm python3-dev python3-venv libmemcached-dev
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

install both clients, mariadb and pq

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

api/api/settings.py
'TEST': {
'CHARSET': 'utf8mb4',
'COLLATION': 'utf8mb4_bin',
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's make a test to see if the migration works correctly for utf8 content

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

checked

api/desecapi/serializers.py Outdated
@@ -494,12 +494,6 @@ def is_empty(data_item):
# so for parallel requests, we can get integrity errors due to duplicate keys.
# This will be considered a 429-error, even though re-sending the request will be successful.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adjust comment according to change below

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

dbapi/pg_hba.conf
#host replication all 127.0.0.1/32 scram-sha-256
#host replication all ::1/128 scram-sha-256

host desec desec all scram-sha-256
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's test the permissions on sandbox

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done (locally)

@peterthomassen peterthomassen force-pushed the 20191106_postgres_dbapi branch from 3f91b13 to 50b7bae Compare August 26, 2020 12:41
@peterthomassen peterthomassen merged commit 50b7bae into master Aug 26, 2020
@peterthomassen peterthomassen deleted the 20191106_postgres_dbapi branch August 26, 2020 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nils-wisiol nils-wisiol nils-wisiol approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@peterthomassen @nils-wisiol