Major Database Refactoring

[ewdurbin]

working on resolving #4772

[di]

@ewdurbin Are you auto-generating the migration from the models? Looks to me like there's a bit of disagreement between the two.

[ewdurbin]

@di not completely no. there is currently some disagreement because I wanted to get what I had up for review before departing for a bit! fixing tests and syncing now!

[ewdurbin]

I think that resolves most of the schema and model concerns. Now it's a matter of cleaning up the newly goof'd references. I plan to pick this up tomorrow morning if no-one runs with it.

Contributors: Feel free to pick branch from this and add commits in a separate PR! Just note that you've done so and reference this pull request. I'll happily close and pick up from your work in the morning :)

[ewdurbin]

Could use some help pushing this along. There's something I don't seem to understand in the relationship between Dependency and Release. I'm not sure if that's a modeling issue or an issue with our test factories.

[dstufft]

It looks like some of the other tracebacks are due to project_id being null when a flush is happening. You might have to adjust some queries to handle this change.

[dstufft]

I've fixed a few more of the tests. I think the remaining tests exist because we still have Release.name, File.name, File.version, but nothing is populating them. I see three possible options for fixing it:

1. Adjust all of the queries we make, to pass those values in.
2. Add a trigger in the database to automatically populate those columns from the referenced parent whenever the value is null.
3. Remove Release.name, File.name, and File.version, and update all of our queries to use joins to get that information from the normalized location.

[dstufft]

Oh, and (1) and (2) will need something to make sure that if we update Project.name or Release.version, it gets updated in the references Release and/or File objects.

(3) is maybe the right answer, but we'd have to see how it performed, doing those joins might increase the query duration by a non-trivial amount. I have no idea.

[dstufft]

Error here is that this PR needs updated to deal with https://github.com/pypa/warehouse/blob/b5be03eb54f2f48ef9d26a2aaf9a9a687aa32e91/warehouse/migrations/versions/eeb23d9b4d00_add_squats_table.py.

[dstufft]

I'm hijacking this PR to include a number of other changes that I think we should do, but I've avoided doing due to downtime they would likely require. Since this PR is going to require downtime already, merging all of these changes into a singe PR allows us to just take the downtime once.

[dstufft]

Ok, so the effect of this PR:

• Add a Project.id and Release.id column.
• Replaces all of the FKs to Project and Release to use that new id column.
• Deletes the Admin role, which is no longer being used.
• Updates Role.user, Release.uploader, and JournalEntry.submitted_by to FK using User.id instead of User.username.
  • In the case of Release.uploader, this includes removing a gnarly join and just setting the uploader at upload time.
• Renames packages to projects.
• Renames accounts_user to users.
• Renames accounts_email to user_emails.
• Renames warehouse_admin_* to admin_*.

Replacing all uses of things like Release.name to be Release.project.name means we're now triggering joins (or extra queries) when we weren't previously. I've tried to go through and spot check to see if we've introduced any of those, and fixed any I found. However it's possible that some will still have been missed. We'll want to monitor performance after this to make sure we're not degrading our perf.

I expect this set of migrations will take a bit to run. On the dev db, my make initdb is taking about 10 minutes, and the production database is quite a bit larger.

[ewdurbin]

Given the complexity and potential for a long migration, I'd like to schedule downtime to complete this rollout.

This Saturday morning Eastern time is good for me, but I'd also like it if I could have some additional support from either @dstufft or @di during this time. Does Saturday 2018-11-10T10:00:00-05:00 work for either/both of you?

I'd estimate we schedule two hours of downtime for uploads, but most likely only use about half of that.

[dstufft]

Maybe, there's something on my calendar, but I'm not sure what it is or if I have to go to it. Will have to check and get back.

[di]

I'll be at PyCon CA that weekend -- Sunday would be better for me.

[ewdurbin]

Sunday is toast for me. Committed all afternoon.

@dstufft if something a bit later works for you Saturday I'm good all afternoon (12PM-5PM Eastern)

[ewdurbin]

Scheduled.

https://status.python.org/incidents/vhyxtc7cvf9g

[ewdurbin]

Deployment to prod disabled temporarily. Merging to send to test.pypi.org.