Documents from DataFrame

[stargeysir]

Hi team,
I am currently building a small project by myself where all text data is stored in a DataFrame, not individual documents. Is it possible to to turn rows in a DataFrame into documents as well, so I can use a PreProcessor before updating my InMemoryDocumentStore?
I have managed to write to the InMemoryDocumentStore after turning my DataFrame into a list of dictionaries, but it seems I need the Document format first if I wish to use a PreProcessor.
Thanks!

[ZanSara]

Hey @stargeysir! Depends a lot on how your dataframe looks like. There's a node that does something similar with CSV files: you can take inspiration from that and create your own custom node that converts your dataframes into Documents: https://github.com/deepset-ai/haystack/blob/61f85b08435b51f9bad12b06fad1c3102ad84c4a/haystack/nodes/file_converter/csv.py

[hiwaveSupport]

Hello - just bumping this up. I tried the workaround described.
I convert my single column df to dict to Document.

`docs_dicts = filtered_df.to_dict(orient="records")

docs = []
for dictionary in docs_dicts:
docs.append(Document.from_dict(dictionary))

print(docs)`

Generates the Document object, with auto-assigned ids.
These generated IDs seem to be duplicated.
When I try to use the Document object to load into a pipeline.

indexing_pipeline.run({"splitter":{"documents":docs}})

I get the following error:
`/usr/local/lib/python3.10/dist-packages/haystack/document_stores/in_memory/document_store.py in write_documents(self, documents, policy)
337 if policy != DuplicatePolicy.OVERWRITE and document.id in self.storage.keys():
338 if policy == DuplicatePolicy.FAIL:
--> 339 raise DuplicateDocumentError(f"ID '{document.id}' already exists.")
340 if policy == DuplicatePolicy.SKIP:
341 logger.warning("ID '{document_id}' already exists", document_id=document.id)

DuplicateDocumentError: ID '93724f75f58c2f2583c9b756c8664555712354a80b5a8517c5ea2009f06d5f43' already exists.`

Is there a way to re-generate or generate unique IDs only when going from dictionary to Document?

[charlielu05]

do you have duplicates in your content column?
by default I think the Document class creates an unique id using _create_id.
you might want to either remove duplicates or create your own unique id for each row of your dataframe.