Create flat and sequential docs structure #1168

blythed · 2023-10-26T09:53:30Z

This PR aims to reboot the documentation:

Add details about SQL
Add more details about power-user/ production features
....

codecov-commenter · 2023-11-01T10:37:14Z

Codecov Report

Attention: 48 lines in your changes are missing coverage. Please review.

Comparison is base (34830a7) 80.33% compared to head (7c201ce) 80.43%.
Report is 9 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1168      +/-   ##
==========================================
+ Coverage   80.33%   80.43%   +0.09%     
==========================================
  Files          95       97       +2     
  Lines        6602     6684      +82     
==========================================
+ Hits         5304     5376      +72     
- Misses       1298     1308      +10

Files	Coverage Δ
superduperdb/base/config.py	`100.00% <ø> (ø)`
superduperdb/components/model.py	`93.66% <100.00%> (+11.00%)`	⬆️
superduperdb/ext/openai/model.py	`96.09% <ø> (+0.64%)`	⬆️
superduperdb/ext/torch/model.py	`76.80% <ø> (ø)`
superduperdb/misc/superduper.py	`76.25% <ø> (ø)`
superduperdb/base/build.py	`65.85% <66.66%> (-1.65%)`	⬇️
superduperdb/base/logger.py	`86.95% <80.00%> (-3.05%)`	⬇️
superduperdb/cli/serve.py	`39.28% <50.00%> (+4.28%)`	⬆️
superduperdb/cli/stack.py	`55.55% <55.55%> (ø)`
superduperdb/components/stack.py	`41.53% <41.53%> (ø)`

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

thejumpman2323

Amazing !!

thejumpman2323 · 2023-11-02T18:24:41Z

docs/hr/content/docs/03_configuration.md

+```bash
+$ export SUPERDUPERDB_DATA_BACKEND='mongodb://localhost:27018/documents'
+$ python -c 'import superduperdb; print(superduperdb.CFG.databackend)'
+mongodb://localhost:27018/documents


We should add more example of configuration,
like how to configure vector search

we should also add example of environment variable configuration
SUPERDUPERDB_ with multi level configuration example cluster.distributed = True

thejumpman2323 · 2023-11-02T18:27:34Z

docs/hr/content/docs/04_connecting.md

+
+```python
+from superduperdb import superduper
+db = superduper()


We should showcase
db = pymongo.client()

db = superduper(db)

thejumpman2323 · 2023-11-02T18:29:36Z

docs/hr/content/docs/07_datalayer_functionality.md

+
+### `db.show`
+
+This methods displays which `Component` instances are registered with the system.


we should give link to Components md file ?

thejumpman2323 · 2023-11-02T18:31:43Z

docs/hr/content/docs/12_sql_query_API.md

+)
+```
+
+### Vector-search


This could be a seperate vector search md file?

thejumpman2323 · 2023-11-02T18:31:57Z

docs/hr/content/docs/12_sql_query_API.md

+)
+```
+
+### Coming soon: support for raw-sql


already merged

thejumpman2323 · 2023-11-02T18:36:00Z

docs/hr/content/docs/19_apply_models.mdx

+
+m = Pipeline(task='sentiment-analysis')
+
+m.predict(


Should we have a Listener md file?

fnikolai · 2023-11-03T00:22:26Z

docs/hr/content/docs/04_connecting.md

+```python
+from superduperdb import CFG
+
+CFG.artifact_store = 'filesystem://./data'


What other fields are supported in CFG ?

fazlulkarimweb · 2023-11-03T06:41:54Z

I think a glossary.md would be beneficial. It would reduce the size of the documentation in the long run as we don't have to explain every terminology every time. We can just use hyperlinks. From an SEO point of view, it would be great as well! As it is a new technology, a glossary explaining the core vocabulary, like vectors, artifacts, and components, is necessary, in my opinion.

For example: Milvus Glossary , Product FAQ

jieguangzhou

The code in the notebook is old. Some of the new versions of the code no longer work and need to be updated.
I think we missing a real (or fake) use case to deploy SuperDuperDB for production. Not just a notebook. For example, we had a database, how to deploy superduperdb to handle it, maybe command line, or other?
There is a lack of a brief process introduction of superduperdb, what happened after adding the model, what happened after adding the listener, etc. Let users understand the general operating mechanism

jieguangzhou · 2023-11-03T06:58:44Z

docs/hr/content/docs/01_intro.md

+
+- [Applying models](19_apply_models.mdx)
+- [Vector search](22_vector_search.mdx)
+- [Example Q&A application](/docs/use_cases/items/question_the_docs)


jieguangzhou · 2023-11-03T07:03:54Z

docs/hr/content/docs/26_developer_vs_production_mode.md

+- A [**change-data-capture** service](29_change_data_capture.md)
+- A [**vector-search** service](30_vector_comparison_service.md), which finds similar vectors, given an input vector
+
+In the following pages, we describe how to set-up these independent services.


Miss the link or information about the following pages

jieguangzhou · 2023-11-03T07:17:59Z

docs/hr/content/docs/09_document_encoder_abstraction.md

+fields:
+
+```python
+s = Schema('my-schema', fields={'my-text': 'str', 'my-image': my_image_encoder})


Maybe we add the context to let users know what this is and how to use it.

jieguangzhou · 2023-11-03T07:31:42Z

docs/hr/content/docs/14_referring_to_data_from_diverse_sources.md

+)
+```
+
+Now when the data is loaded from the database, it is loaded as text:


I think we should explain to the user what's going on here. Because we must assume that users are not familiar with related concepts, such as encoder and decoder in Encoder, and uri->bytes

jieguangzhou · 2023-11-03T07:40:14Z

docs/hr/content/docs/17_ai_apis.md

It is recommended to add a custom API wrapper demo here.

api = 'https://xxxxxxx:12345/predict' class ModelA(Model): ...... def predict(self, x): return label db.add(ModelA(api))

Then users can quickly use superduperdb to test their model services like this

jieguangzhou · 2023-11-03T07:48:12Z

docs/hr/content/docs/25_creating_stacks_of_functionality.md

+)
+
+db.add(
+    Stack(


import Stack?

blythed · 2023-11-03T10:38:30Z

The code in the notebook is old. Some of the new versions of the code no longer work and need to be updated.

I think we missing a real (or fake) use case to deploy SuperDuperDB for production. Not just a notebook. For example, we had a database, how to deploy superduperdb to handle it, maybe command line, or other?

There is a lack of a brief process introduction of superduperdb, what happened after adding the model, what happened after adding the listener, etc. Let users understand the general operating mechanism

Hi @jieguangzhou thanks for this in depth feedback. I'm working on adding 1, 3. (I realize the notebooks/ use-cases are out-of-date). For 2. what would you suggest? A mini-code-base?

This was referenced Oct 26, 2023

Design document and draft to lead rehaul of documentation #1167

Closed

Complete all sections of documentation walkthrough #1076

Closed

blythed force-pushed the docs/revamp branch from 6dc1ccf to c6183a4 Compare November 1, 2023 10:36

blythed force-pushed the docs/revamp branch 2 times, most recently from 25d496e to a327af8 Compare November 1, 2023 16:33

thejumpman2323 suggested changes Nov 2, 2023

View reviewed changes

fnikolai reviewed Nov 3, 2023

View reviewed changes

jieguangzhou reviewed Nov 3, 2023

View reviewed changes

blythed force-pushed the docs/revamp branch 2 times, most recently from e7eec38 to 942e5c4 Compare November 3, 2023 19:02

Create flat and sequential docs structure

7c201ce

blythed force-pushed the docs/revamp branch from 942e5c4 to 7c201ce Compare November 5, 2023 11:01

blythed marked this pull request as ready for review November 5, 2023 11:06

blythed merged commit 9ba310c into superduper-io:main Nov 5, 2023

blythed deleted the docs/revamp branch June 1, 2024 10:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create flat and sequential docs structure #1168

Create flat and sequential docs structure #1168

blythed commented Oct 26, 2023 •

edited

Loading

codecov-commenter commented Nov 1, 2023 •

edited

Loading

thejumpman2323 left a comment

thejumpman2323 Nov 2, 2023

thejumpman2323 Nov 2, 2023

thejumpman2323 Nov 2, 2023

thejumpman2323 Nov 2, 2023

thejumpman2323 Nov 2, 2023

thejumpman2323 Nov 2, 2023

fnikolai Nov 3, 2023

fazlulkarimweb commented Nov 3, 2023

jieguangzhou left a comment •

edited

Loading

jieguangzhou Nov 3, 2023

jieguangzhou Nov 3, 2023

jieguangzhou Nov 3, 2023

jieguangzhou Nov 3, 2023

jieguangzhou Nov 3, 2023 •

edited

Loading

jieguangzhou Nov 3, 2023

blythed commented Nov 3, 2023


		### `db.show`

		This methods displays which `Component` instances are registered with the system.

+              )
+              db.add(
+                  Stack(


		m = Pipeline(task='sentiment-analysis')

		m.predict(

Create flat and sequential docs structure #1168

Create flat and sequential docs structure #1168

Conversation

blythed commented Oct 26, 2023 • edited Loading

codecov-commenter commented Nov 1, 2023 • edited Loading

Codecov Report

thejumpman2323 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fazlulkarimweb commented Nov 3, 2023

jieguangzhou left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jieguangzhou Nov 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blythed commented Nov 3, 2023

blythed commented Oct 26, 2023 •

edited

Loading

codecov-commenter commented Nov 1, 2023 •

edited

Loading

jieguangzhou left a comment •

edited

Loading

jieguangzhou Nov 3, 2023 •

edited

Loading