Add optional subgroups to pyproject.toml

[nenb]

Description

This PR does three things:

1. uses pyproject.toml to manage all optional subgroups (eg torch, dev etc)
2. removes the requirements subdirectory which had caused a lot of confusion and replaces it with a simpler configuration inside .github.
3. simplifies the CI logic

More Details

1. Rather than having a separate bunch of files in a requirements subdirectory for managing our dependencies, we now manage everything via pyproject.toml. This is the modern way of doing Python packaging, and is arguably a lot simpler than having a separate requirements subdirectory. A minor disadvantage is that it does make our pyproject.toml quite large and a bit harder to read. In the future, adding depedencies is simply a matter of adding to the appropriate subgroup in pyproject.toml, or creating a new subgroup if this is necessary. Users can install these subgroups by doing a pip install superduperdb[my_subgroup_name].

2. The original intention for having a separate requirements subdirectory was i) to make our pyproject.toml more readable and ii) so that we could pin our dependencies for CI. We have sacrificed i) for the sake of simplicity (see 1.). For ii), I have now moved these files inside the .github subdirectory, as they are only used for CI and do not need to be visible in the root directory. I have also reduced the number of files to 2 - dev and docs. This will mean that our CI will take a bit longer, but it's a lot simpler to understand and maintain. Further details are inside CONTRIBUTING.md

3. Since we now only have dev and docs requirements files, I have merged the CI into a single step, where linting, type-checking and testing all happen together

Related Issues

#653
#627

Checklist

• Is this code covered by new or existing unit tests or integration tests?
• Did you run make test successfully?
• Do new classes, functions, methods and parameters all have docstrings?
• Were existing docstrings updated, if necessary?
• Was external documentation updated, if necessary?

Additional Notes or Comments