Torch optional

[nenb]

(Taken directly from #695. The work was done by @rec with a minor tweak to the CI by @nenb. A new PR was created for this work due to @nenb 's challenges with git 🤕 )

---

This pull request removes torch from the main dependencies that get installed by default.

With the modifications in these commits, the test suite runs without torch installed, with 59.5% coverage.

We ended up with a lot of requirements files, so in an independent commit at the end, I inlined three requirements sets that are always loaded together in each case, requirements-doc, requirements-lint and requirements-test.

---

We need to get the download way down to deploy the web app, and this commit achieves that.

This does not add any new pip installs like superduperdb[torch] which I don't know how to do.

If someone wants to use torch, they must pip install torch themselves for the moment

[codecov-commenter]

Codecov Report

Patch coverage: 75.00% and project coverage change: -0.06% ⚠️

Comparison is base (86fb7af) 79.65% compared to head (47942d0) 79.60%.
Report is 12 commits behind head on main.

❗ Current head 47942d0 differs from pull request most recent head 2845054. Consider uploading reports for the commit 2845054 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #697      +/-   ##
==========================================
- Coverage   79.65%   79.60%   -0.06%     
==========================================
  Files          79       79              
  Lines        4950     4937      -13     
==========================================
- Hits         3943     3930      -13     
  Misses       1007     1007              

Files Changed	Coverage Δ
superduperdb/ext/transformers/model.py	73.55% <0.00%> (-0.83%)	⬇️
superduperdb/misc/classification.py	28.57% <0.00%> (ø)
superduperdb/vector_search/inmemory.py	98.03% <ø> (ø)
superduperdb/db/query_dataset.py	48.36% <100.00%> (-0.44%)	⬇️
superduperdb/misc/superduper.py	92.18% <100.00%> (-0.24%)	⬇️
superduperdb/vector_search/base.py	95.23% <100.00%> (-0.37%)	⬇️
superduperdb/vector_search/lancedb_client.py	87.91% <100.00%> (-0.14%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.