Delete table database versioning (revised PR with unit tests)

[MatMoore]

This takes the changes to versioning.py from #2465, adds unit tests, and reworks the behaviour slightly.

This could do with more integration testing, but I have tested the happy path by running the container locally, and once this is rolled out to API gateway we will do some further testing in order to document the scenarios here: https://dsdmoj.atlassian.net/wiki/spaces/DataPlatform/pages/4557963328/user+journeys+via+API+WIP

New behaviour for delete table

1. We decided Thursday morning that the delete table endpoint should return early if a schema doesn't exist, even if there may be data hanging around. I.e. we shouldn't attempt to delete data for which there is no metadata. In future we will ensure that any generated schemas are written to s3 as a prerequisite of loading data into a table.

2. For now, we will just delete the table & data from the new version. This deviates from our original plan of deleting from all historical versions (i.e. "delete means delete"), but we had some doubts about implementing contradictory behaviour (i.e. creating a new major version while at the same time making breaking changes to existing versions doesn't make sense). I've created 💣 "Delete means delete" when deleting a table #2500 to revisit the delete means delete behaviour. For now the exact logic doesn't matter as long as the endpoint behaves consistently.

So the expected behaviour when deleting a table is now:

1. If the schema is not found, do nothing
2. Create a new version of the database with the new version number in
3. If the glue table is there, delete it from the new version only
4. Data will still be there in historical versions

Copying tables between databases

Originally this had an exclude list of fields that should not be copied over when we copy tables. I've changed this to an include list so that the code is less likely to break completely if amazon starts returning additional fields in the get table response. (When testing this I ran into several fields that weren't handled by the previous version of this code)

Refactor of glue_and_athena_utils

This now contains functions for creating, deleting, and fetching databases and tables. I've renamed some of the existing functions in this module to follow a common naming convention, but kept aliases with the old names.

[MatMoore]

Decision after speaking to @PriyaBasker23

For now, we will just delete the table & data from the new version. This deviates from our original plan of deleting from all historical versions "delete means delete", but we had some doubts about implementing contradictory behaviour (i.e. creating a new major version while at the same time making breaking changes to existing versions doesn't make sense).

Created #2500 to revisit the delete means delete behaviour. For now the exact logic doesn't matter as long as the endpoint behaves consistently.

[MatMoore]

Need to rebase this monday due to conflicts.

Remaining stuff

• Bump the version
• Make sure glue errors are handled in the delete table lambda
• Test the lambda