Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add local cluster #1403

Merged
merged 1 commit into from
Nov 27, 2023
Merged

Add local cluster #1403

merged 1 commit into from
Nov 27, 2023

Conversation

thejumpman2323
Copy link
Contributor

@thejumpman2323 thejumpman2323 commented Nov 25, 2023

Description

fix ##1379

Related Issues

Checklist

  • Is this code covered by new or existing unit tests or integration tests?
  • Did you run make test successfully?
  • Do new classes, functions, methods and parameters all have docstrings?
  • Were existing docstrings updated, if necessary?
  • Was external documentation updated, if necessary?

Additional Notes or Comments

@thejumpman2323 thejumpman2323 self-assigned this Nov 25, 2023
@codecov-commenter
Copy link

codecov-commenter commented Nov 25, 2023

Codecov Report

Attention: 34 lines in your changes are missing coverage. Please review.

Comparison is base (34830a7) 80.33% compared to head (f2687d3) 67.12%.
Report is 127 commits behind head on main.

❗ Current head f2687d3 differs from pull request most recent head 071f1d6. Consider uploading reports for the commit 071f1d6 to get more accurate results

Files Patch % Lines
superduperdb/backends/dask/compute.py 47.05% 18 Missing ⚠️
superduperdb/backends/base/compute.py 68.96% 9 Missing ⚠️
superduperdb/backends/base/query.py 85.71% 3 Missing ⚠️
superduperdb/backends/base/metadata.py 66.66% 2 Missing ⚠️
superduperdb/backends/base/artifact.py 66.66% 1 Missing ⚠️
superduperdb/backends/base/data_backend.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1403       +/-   ##
===========================================
- Coverage   80.33%   67.12%   -13.22%     
===========================================
  Files          95      103        +8     
  Lines        6602     7272      +670     
===========================================
- Hits         5304     4881      -423     
- Misses       1298     2391     +1093     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

superduperdb/base/config.py Show resolved Hide resolved
superduperdb/server/cluster.py Show resolved Hide resolved
None: Run all jobs in local mode i.e simple function call
local: same as above
dask+thread: Run all jobs on a local threaded dask cluster
dask+tcp:<host>:<port>: Run all jobs on a remote dask cluster
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks inconsistent with the above. Please make uniform.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do u mean?
can u write the edited version here :)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be:

"local": same as above

None: Run vector search on local
http://<host>:<port>: Connect a remote vector search service

`cdc` options:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What not add these in the parameter doc-strings?

:param backfill_batch_size: The number of rows to backfill at a time
for vector-search loading
:param compute: The URI for compute i.e 'local', 'dask+tcp://localhost:8786'
None: Run all jobs in local mode i.e simple function call
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ':' at the end is confusing -- is that part of the string or part of the explanation. You need to add "quotation" marks.

@thejumpman2323 thejumpman2323 merged commit ea8ec01 into superduper-io:main Nov 27, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement Local Threaded Cluster for Easy Debugging Repair, Update, and Clean superduperdb.cli.serve
5 participants