[Question]: How to split or seperate nodes Doucment data?

[iriye]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

JSONReader = download_loader("JSONReader")
loader = JSONReader()
index = VectorStoreIndex.from_documents(documents, service_context=service_context)
when I use this code documents prompt size very big, How to split documents text i or how to seperate nodes?

[dosubot]

🤖

Hello,

Based on your question, it seems you want to split your large documents into smaller chunks when using the VectorStoreIndex.from_documents method in LlamaIndex. While LlamaIndex does not provide a built-in method to do this, you can implement this functionality yourself before passing the documents to the from_documents method.

Here is a simple example of how you can do this:

from llama_index.schema import Document

def split_document(document: Document, chunk_size: int):
    text = document.text
    chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]
    return [Document(text=chunk, metadata=document.metadata) for chunk in chunks]

# Assume `documents` is your list of large documents
smaller_documents = []
for document in documents:
    smaller_documents.extend(split_document(document, chunk_size=1000))

index = VectorStoreIndex.from_documents(smaller_documents)

In this example, split_document is a function that splits the text of a document into chunks of a specified size (in this case, 1000 characters), and creates a new Document object for each chunk. The metadata of the original document is preserved for each chunk.

Please adjust the chunk_size according to your needs. This is just a simple example and might not cover all your needs, especially if your documents have complex structures or specific requirements.

Additionally, you might want to consider the chunk_size parameter in LlamaIndex's SentenceSplitter class. This parameter is used to determine the maximum number of tokens that each chunk of text can contain when the text is being split into chunks. The larger the chunk_size, the larger each chunk of text will be, and vice versa.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• [Question]: What Is The Optomial Document Size
• How to form nodes (in GptSimpleVectorIndex form) based on paragraphs not based on fixed token size?
• llama_index/indices/base.py
• llama_index/text_splitter/token_splitter.py
• llama_index/text_splitter/sentence_splitter.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

You can also just decrease the chunk size in the service context

from llama_index import ServiceContext
service_context = ServiceContext.from_defaults(chunk_size=512, ...)
index = VectorStoreIndex.from_documents(documents, service_context=service_context)
query_engine = index.as_query_engine(similarity_top_k=2)