docs(rustdocs): rewrite the initial landing page

README.md

@@ -66,15 +66,13 @@ Blazing fast data pipelines for Retrieval Augmented Generation written in Rust
 
 <!-- [![Product Name Screen Shot][product-screenshot]](https://example.com) -->
 
-**Swiftide** is a straightforward, easy-to-use, easy-to-extend asynchronous data indexing and processing library. It is designed to be used in a RAG (Retrieval Augmented Generation) system. It is built to be fast and efficient, with a focus on parallel processing and asynchronous operations.
+Swiftide is a data indexing and processing library, tailored for Retrieval Augmented Generation (RAG). When building applications with large language models (LLM), these LLMs need access to external resources. Data needs to be transformed, enriched, split up, embedded, and persisted. It is build in Rust, using parallel, asynchronous streams and is blazingly fast.
 
 <div align="center">
-  <a href="https://github.com/bosun-ai/swiftide">
     <img src="https://github.com/bosun-ai/swiftide/blob/master/images/rag-dark.svg" alt="RAG" width="100%" >
-  </a>
 </div>
 
-While working with other Python-based tooling, frustrations arose around performance, stability, and ease of use. Thus, Swiftide was born. Indexing performance went from multiple tens of minutes to a few seconds.
+While working with other Python-based tooling, frustrations arose around performance, stability, and ease of use. Thus, Swiftide was born. Indexing performance went from tens of minutes to a few seconds.
 
 Part of the [bosun.ai](https://bosun.ai) project. An upcoming platform for autonomous code improvement.
 
@@ -114,18 +112,19 @@ indexing::Pipeline::from_loader(FileLoader::new(".").with_extensions(&["rs"]))
 
 ## Features
 
-- Extremely fast streaming indexing pipeline with async, parallel processing
+- Fast streaming indexing pipeline with async, parallel processing
 - Integrations with OpenAI, Redis, Qdrant, FastEmbed, and Treesitter
 - A variety of loaders, transformers, and embedders and other common, generic tools
 - Bring your own transformers by extending straightforward traits
+- Splitting and merging pipelines
 - Store into multiple backends
 - `tracing` supported for logging and tracing, see /examples and the `tracing` crate for more information.
 
 <p align="right">(<a href="#readme-top">back to top</a>)</p>
 
 ## Vision
 
-Our goal is to create an extremely fast, extendable platform for data indexing and querying to further the development of automated LLM applications, with an easy-to-use and easy-to-extend api.
+Our goal is to create afast, extendable platform for data indexing and querying to further the development of automated LLM applications, with an easy-to-use and easy-to-extend api.
 
 <p align="right">(<a href="#readme-top">back to top</a>)</p>
 
@@ -198,7 +197,7 @@ See the [open issues](https://github.com/bosun-ai/swiftide/issues) for a full li
 
 ## Contributing
 
-Swiftide is in a very early stage and we are aware that we do lack features for the wider community. Contributions are very welcome. :tada:
+Swiftide is in a very early stage and we are aware that we lack features for the wider community. Contributions are very welcome. :tada:
 
 If you have a great idea, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
 Don't forget to give the project a star! Thanks again!

swiftide/Cargo.toml

@@ -8,6 +8,9 @@ keywords = ["llm", "rag", "ai", "data", "openai"]
 description = "Blazing fast, streaming document and code indexation"
 categories = ["asynchronous"]
 repository = "https://github.com/bosun-ai/swiftide-rs"
+homepage = "https://swiftide.rs"
+
+[badges]
 
 [dependencies]
 anyhow = { version = "1.0", features = ["backtrace"] }
@@ -110,3 +113,4 @@ multiple_crate_versions = "allow"
 
 [package.metadata.docs.rs]
 all-features = true
+cargo-args = ["-Zunstable-options", "-Zrustdoc-scrape-examples"]

swiftide/src/lib.rs

@@ -1,12 +1,54 @@
-//! Swiftide - Document and code indexation for retrieval augmented generation
+//! # Swiftide
 //!
-//! Swiftide is a straightforward, easy-to-use, easy-to-extend asynchronous file indexing and processing system. It is designed to be used in a RAG (Retrieval Augmented Generation) system. It is built to be fast and efficient, with a focus on parallel processing and asynchronous operations.
+//! Swiftide is a data indexing and processing library, tailored for Retrieval Augmented Generation (RAG). When building applications with large language models (LLM), these LLMs need access to external resources. Data needs to be transformed, enriched, split up, embedded, and persisted. It is build in Rust, using parallel, asynchronous streams and is blazingly fast.
 //!
-//! Part of the bosun.ai project. An upcoming platform for autonomous code improvement.
+//! Part of the [bosun.ai](https://bosun.ai) project. An upcoming platform for autonomous code improvement.
 //!
 //! We <3 feedback: project ideas, suggestions, and complaints are very welcome. Feel free to open an issue.
 //!
-//! # Feature flags
+//! Read more about the project on the [swiftide website](https://swiftide.rs)
+//!
+//! ## Features
+//!
+//! - Extremely fast streaming indexing pipeline with async, parallel processing
+//! - Integrations with `OpenAI`, `Redis`, `Qdrant`, `FastEmbed`, `Treesitter` and more
+//! - A variety of loaders, transformers, and embedders and other common, generic tools
+//! - Bring your own transformers by extending straightforward traits
+//! - Splitting and merging pipelines
+//! - Store into multiple backends
+//! - `tracing` supported for logging and tracing, see /examples and the `tracing` crate for more information.
+//!
+//! ## Example
+//!
+//! ```no_run
+//! use swiftide::loaders::FileLoader;
+//! use swiftide::transformers::{ChunkMarkdown, Embed, MetadataQAText};
+//! use swiftide::integrations::qdrant::Qdrant;
+//! use swiftide::indexing::Pipeline;
+//! # use anyhow::Result;
+//!
+//! # #[tokio::main]
+//! # async fn main() -> Result<()> {
+//! # let qdrant_url = "url";
+//! # let openai_client = swiftide::integrations::openai::OpenAI::builder().build()?;
+//!  Pipeline::from_loader(FileLoader::new(".").with_extensions(&["md"]))
+//!          .then_chunk(ChunkMarkdown::from_chunk_range(10..512))
+//!          .then(MetadataQAText::new(openai_client.clone()))
+//!          .then_in_batch(10, Embed::new(openai_client.clone()))
+//!          .then_store_with(
+//!              Qdrant::try_from_url(qdrant_url)?
+//!                  .batch_size(50)
+//!                  .vector_size(1536)
+//!                  .collection_name("swiftide-examples".to_string())
+//!                  .build()?,
+//!          )
+//!          .run()
+//!          .await
+//! # }
+//! ```
+//!
+//! ## Feature flags
+//!
 //! Swiftide has little features enabled by default as there are some dependency heavy
 //! integrations.
 //!