Skip to content

Commit

Permalink
docs(README): Copy improvements (#261)
Browse files Browse the repository at this point in the history
  • Loading branch information
timonv authored Sep 1, 2024
1 parent a1250c1 commit fb381b8
Showing 1 changed file with 28 additions and 7 deletions.
35 changes: 28 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,12 +80,14 @@ With Swiftide, you can build your AI application from idea to production in a fe
<img src="https://github.com/bosun-ai/swiftide/blob/master/images/rag-dark.svg" alt="RAG" width="100%" >
</div>

While working with other Python-based tooling, frustrations arose around performance, stability, and ease of use. Thus, Swiftide was born. Indexing performance went from tens of minutes to a few seconds.
While working with other Python-based tooling, frustrations arose around performance, stability, and ease of use. Thus, Swiftide was born. Swiftide's goal is to offer a fully fledged retrieval augmented generation library, that is fast, easy-to-use, reliable and easy-to-extend.

Part of the [bosun.ai](https://bosun.ai) project. An upcoming platform for autonomous code improvement.

We <3 feedback: project ideas, suggestions, and complaints are very welcome. Feel free to open an issue or contact us on [discord](https://discord.gg/3jjXYen9UY).

**Great starting points are this readme, [swiftide.rs](https://swiftide.rs), [the examples folder](https://github.com/bosun-ai/swiftide/tree/master/examples), our blog at [bosun.ai](https://bosun.ai), and in depth tutorials at [swiftide-tutorial](https://github.com/bosun-ai/swiftide-tutorial).**

> [!CAUTION]
> Swiftide is under heavy development and can have breaking changes while we work towards 1.0. Documentation here might fall short of all features, and despite our efforts be slightly outdated. Expect bugs. We recommend to always keep an eye on our [github](https://github.com/bosun-ai/swiftide) and [api documentation](https://docs.rs/swiftide/latest/swiftide/). If you found an issue or have any kind of feedback we'd love to hear from you in an issue.
Expand All @@ -103,6 +105,8 @@ We <3 feedback: project ideas, suggestions, and complaints are very welcome. Fee

## Example

Indexing a local code project, chunking into smaller pieces, enriching the nodes with metadata, and persisting into [Qdrant](https://qdrant.tech):

```rust
indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["rs"]))
.with_default_llm_client(openai_client.clone())
Expand All @@ -127,13 +131,29 @@ indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["rs"]))
.await?;
```

Querying for an example on how to use the query pipeline:

```rust
query::Pipeline::default()
.then_transform_query(GenerateSubquestions::from_client(
openai_client.clone(),
))
.then_transform_query(Embed::from_client(
openai_client.clone(),
))
.then_retrieve(qdrant.clone())
.then_answer(Simple::from_client(openai_client.clone()))
.query("How can I use the query pipeline in Swiftide?")
.await?;
```

_You can find more examples in [/examples](https://github.com/bosun-ai/swiftide/tree/master/examples)_

<p align="right">(<a href="#readme-top">back to top</a>)</p>

## Vision

Our goal is to create a fast, extendable platform for data indexing and querying to further the development of automated LLM applications, with an easy-to-use and easy-to-extend api.
Our goal is to create a fast, extendable platform for Retrieval Augmented Generation to further the development of automated AI applications, with an easy-to-use and easy-to-extend api.

<p align="right">(<a href="#readme-top">back to top</a>)</p>

Expand All @@ -146,7 +166,8 @@ Our goal is to create a fast, extendable platform for data indexing and querying
- Splitting and merging pipelines
- Jinja-like templating for prompts
- Store into multiple backends
- Integrations with OpenAI, Groq, Redis, Qdrant, Ollama, FastEmbed-rs, Fluvio, and Treesitter
- Integrations with OpenAI, Groq, Redis, Qdrant, Ollama, FastEmbed-rs, Fluvio, LanceDB, and Treesitter
- Evaluate pipelines with RAGAS
- Sparse vector support for hybrid search
- `tracing` supported for logging and tracing, see /examples and the `tracing` crate for more information.

Expand All @@ -158,7 +179,7 @@ Our goal is to create a fast, extendable platform for data indexing and querying
| **Loading data** | Files <br> Scraping <br> Fluvio <br> Other pipelines and streams |
| **Transformers and metadata generation** | Generate Question and answerers for both text and code (Hyde) <br> Summaries, titles and queries via an LLM <br> Extract definitions and references with tree-sitter |
| **Splitting and chunking** | Markdown <br> Code (with tree-sitter) |
| **Storage** | Qdrant <br> Redis |
| **Storage** | Qdrant <br> Redis <br> LanceDB |

<p align="right">(<a href="#readme-top">back to top</a>)</p>

Expand All @@ -172,7 +193,7 @@ Make sure you have the rust toolchain installed. [rustup](https://rustup.rs) Is

To use OpenAI, an API key is required. Note that by default `async_openai` uses the `OPENAI_API_KEY` environment variables.

Other integrations will need to be installed accordingly.
Other integrations might have their own requirements.

### Installation

Expand Down Expand Up @@ -210,7 +231,7 @@ Nodes have a path, chunk and metadata. Currently metadata is copied over when ch
Additionally, several generic transformers are implemented. They take implementers of `SimplePrompt` and `EmbedModel` to do their things.

> [!NOTE]
> No integrations are enabled by default as some are code heavy. Either cherry-pick the integrations you need or use the "all" feature flag.
> No integrations are enabled by default as some are code heavy. We recommend you to cherry-pick the integrations you need. By convention flags have the same name as the integration they represent.
> [!WARNING]
> Due to the performance, chunking before adding metadata gives rate limit errors on OpenAI very fast, especially with faster models like 3.5-turbo. Be aware.
Expand Down Expand Up @@ -242,7 +263,7 @@ Swiftide is in a very early stage and we are aware that we lack features for the
If you have a great idea, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!

If you just want to contribute (bless you!), see [our issues](https://github.com/bosun-ai/swiftide/issues).
If you just want to contribute (bless you!), see [our issues](https://github.com/bosun-ai/swiftide/issues) or join us on Discord.

1. Fork the Project
2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
Expand Down

0 comments on commit fb381b8

Please sign in to comment.