How to save embedding_model and document_store locally ?

[arthurbarros]

I wonder if it is possible to do something similar to reader.save(directory="my_path") but for a EmbeddingRetriever

[Timoeller]

Hey @arthurbarros
the retrievers are not trained (for now) so there is not much need to store them.
The underlying model like "deepset/sentence_bert" is downloaded and cached when first calling the EmbeddingRetriever, so you only download it once.

Though saving the Retrievers might be an interesting feature if you are in an offline setting or copy the model between machines - working with the cache files directly is rather cumbersome. Could you elaborate why you need this functionality, so we can better prioritize this feature?

[Timoeller]

Closing this now. Feel free to reopen

[violetcodes]

Hi @Timoeller,
I have used dpr retriever on colab using gpu (following tute 6) for qa on my documents, I want to use this model on my local (or instance) without a gpu. document_store.update_embedding(retriever) #document_store: FAISSDocumentStore, retriever: DensePassageRetriever on instance is taking much longer time (it is low config machine). This step needs to be run only once. If I could save retriever, or document_store (with its updated embeddings) and copy to my instance, I have to only load them without running bottleneck step and every time, demo would be fast.
am I doing something wrong?

also couldn't use document_store.save("filepath") because FAISSDocumentStore.load() needs sql_url, why don't we need sql_url during initialization but it is essential during loading?

[Weilin37]

I'm also wondering how to save the document store once we've taken the time to generate the embeddings and loading them.

[tholor]

Hey @Weilin37,

What type of document store are you using?

If you use ElasticsearchDocumentStore the data will be automatically persisted by elasticsearch.

If you use FAISS, the embeddings are stored in a FAISS Index. You can save that index via FAISSDocumentStore.save("file_path"). All remaining data (e.g the document text and metadata) is automatically kept persistent in SQL. You can load it back via FAISSDocumentStore.load(faiss_file_path="file_path", sql_url= "sqlite:///")

Hope this helps!

[Weilin37]

Thanks! This is definitely the answer I was looking for! (I am using FAISS)

[AzureAlph]

Hi @tholor,

Just curious if that also applies for document_store that already had update_embeddings applied? Using Faiss by the way.
I did try your example but when I loaded it, the embeddings where empty. I haven't tried all examples in the documentation yet so maybe im missing something

[tholor]

Yes, absolutely.

[AzureAlph]

So I was trying this out in colab
document_store = FAISSDocumentStore()
document_store.write_documents(datadict)
retriever = EmbeddingRetriever(document_store=document_store,
embedding_model="deepset/sentence_bert",
use_gpu=False)

document_store.update_embeddings(retriever)
document_store.save("testfile_path")
document_store2 = FAISSDocumentStore()
document_store2.load(faiss_file_path="testfile_path", sql_url= "sqlite:///")

but document_store2 always comes up empty. Please guide me where I went wrong. I think its in the sql part but I am not also familiar with it

[tholor]

Can you please try:

document_store = FAISSDocumentStore(sql_url= "sqlite:///haystack_test_faiss.db")
document_store.write_documents(datadict)
retriever = EmbeddingRetriever(document_store=document_store,
embedding_model="deepset/sentence_bert", use_gpu=False)
document_store.update_embeddings(retriever)
document_store.save("testfile_path")
document_store2 = FAISSDocumentStore.load(faiss_file_path="testfile_path", sql_url= "sqlite:///haystack_test_faiss.db")

[AzureAlph]

It worked! Thank you.

Also I noticed that after applying 'update_embedding', when I try to check the embedding value of items via 'get_all_documents' within the document_store, embedding values remain empty. Is it the expected behavior or am I misunderstanding how update_embedding works?

[tholor]

You are totally right, the embeddings should not remain empty. @tanaysoni can you please check this and verify that saving/loading is working as expected?

[tanaysoni]

Hi @AzureAlph, currently, the FAISSDocumentStore does not return embeddings with the documents by default.

With #615, an option to get the embeddings using the return_embedding parameter would get added to the FAISSDocumentStore.get_all_documents().

[harikc456]

In Google Colab I have used the below code to update the embedding and save the document_store and the retriever.

document_store = FAISSDocumentStore(faiss_index_factory_str="Flat", sql_url= "sqlite:///haystack_test_faiss.db")
from haystack.retriever.dense import DensePassageRetriever
retriever = DensePassageRetriever(document_store=document_store,
                                  query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
                                  passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",
                                  max_seq_len_query=64,
                                  max_seq_len_passage=256,
                                  batch_size=16,
                                  use_gpu=True,
                                  embed_title=True,
                                  use_fast_tokenizers=True)
document_store.update_embeddings(retriever)
retriever.save("retriever.pt")
document_store.save("faiss.index")

Later in my local machine, I have the following snippet which loads the document store and the saved retriever

document_store = FAISSDocumentStore.load("faiss.index",sql_url="sqlite:///haystack_test_faiss.db", index = None)
retriever = DensePassageRetriever.load("retriever.pt", document_store=document_store)
reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=True)
finder = Finder(reader, retriever)
prediction = finder.get_answers(question="Who is the sister of Sansa?", top_k_retriever=10, top_k_reader=5)

But This returns 0 candidates by the retriever. Even though the same was answered in Google Colab

Am I loading the files wrong?

[benscottie]

@tholor I am experiencing the same problem as above. Is there a solution? I'm wondering if setting index to None has anything to do with it.

[Timoeller]

Hey @benscottie thanks for bringing this up here. Could you open a separate issue with a more detailed explanation of your setup? I could imaging that different faiss versions for saving and loading might complicate it.
Or do you save and load in one colab environment?

[benscottie]

hey @Timoeller thanks for your response. I am saving and loading in one environment and having the issue. Ultimately I would like to be able to save and load in separate environments. I will open a separate issue

[fingoldo]

document_store2 = FAISSDocumentStore.load(faiss_file_path="testfile_path", sql_url= "sqlite:///haystack_test_faiss.db")

Hi, does your example mean that, if sql_url was not specified from the start, documents are not persisted anywhere and it's not possible to load them gain? If I'm wrong, where can I find them? According to the docs, default path should be "sqlite:///faiss_document_store.db", but I can't find such file, faiss_document_store.db, anywhere on my PC.

[tholor]

Hey @fingoldo,
We recently changed the save/load behaviour for FAISS (see #1459). With the latest master version, you should indeed have the default sqlite:///faiss_document_store.db and find a file accordingly on your system. This was not the case in older haystack versions where we used the inmemory based sqllite as default.

You still need to call FAISSDocumentStore.save() to save also the FAISS index and the configuration. Just having the SQL DB won't be enough to restore your doc store.

Hope this helps.

[fingoldo]

Thank you very much for the clarification... I have installed farm-haystack recently, a few days ago. After reading ~2M small documents, updating embeddings and calling .save(), I can't find the .db file anywhere, but the RAM footprint is 18Gbs. So probably I had an earlier version somehow which is in-memory SQL. I guess the only option I have is to reinstall the haystack & reindex from scratch...

[tholor]

Yes, the above PR was just merged 4 days ago (so after our latest 0.10. pypy release). I'd suggest you re-install directly from master via:

git clone https://github.com/deepset-ai/haystack.git
cd haystack
pip install --editable .

[fingoldo]

I apologize, is changing of vector_dim of a FAISS store supported, without reimporting all the documents? For example, when I want to test many models to see which one gives the most sensible results. 'cause I tried applying "sentence-transformers/all-MiniLM-L6-v2" and it turned out to be a mismatch between default vector_dim of the store (768) and the actual dimension size of the retriever's model (384. is there some standard way to query this number from the retriever, by the way?). When I changed FAISS_store.vector_dim = 384 and tried update_embeddings() again, it was the same error, so I guess changing is not supported at all or must be done at store init step.

[fingoldo]

Also, it seems that you have a small inaccuracy in the code.

file:
haystack\haystack\document_store\faiss.py

method:
def load(cls, index_path: Union[str, Path], config_path: Optional[Union[str, Path]] = None):

code:

        if not config_path:
            index_path = Path(index_path)
            faiss_init_params_path = index_path.with_suffix(".json")

        init_params: dict = {}
        try:
            with open(faiss_init_params_path, 'r') as ipp:
                init_params = json.load(ipp)
        except OSError as e:

if I provide config_path, load fails as faiss_init_params_path is not initialized.

[tholor]

is changing of vector_dim of a FAISS store supported, without reimporting all the documents?

No, that's currently not supported as FAISS doesn't allow any such permutations on an existing index. Most transformers use 768 as the dim - however, I see your point when switching to smaller retriever models.
I think we can try to find a way to delete the FAISS index under the hood and repopulate it with embeddings from the new retriever using the documents that are still stored in SQL. Created a new issue for this: #1505
Let me know if you are interested in contributing here.

if I provide config_path, load fails as faiss_init_params_path is not initialized.

@ZanSara Can you please check?

[fingoldo]

t transformers use 768 as the dim - however, I see your point when switching to smaller retriever models.
I think we can try to find a way to delete the FAISS index under the hood and repopulate it with embeddi

Will things change if I switch to Milvus? I mean, that "index" parameter in many of the doc stores. Can it be used somehow to achieve that goal of testing multiple retrievers (and, therefore, embedding models) on the same document set?

[fingoldo]

t transformers use 768 as the dim - however, I see your point when switching to smaller retriever models.
I think we can try to find a way to delete the FAISS index under the hood and repopulate it with embeddi

Will things change if I switch to Milvus? I mean, that "index" parameter in many of the doc stores. Can it be used somehow to achieve that goal of testing multiple retrievers (and, therefore, embedding models) on the same document set?

I'd be willing to contribute something useful, but I'm not experienced at all in contributing (

[tholor]

I'd be willing to contribute something useful, but I'm not experienced at all in contributing (

How about you give it a shot and if you don't succeed, we can still take over in a couple of weeks?
I added some first thoughts on the design here.

You can find some tips for contributing to Haystack here. In short: If you create an early "work in progress" PR, we can also try to help and give feedback. We recommend using our CI for running the tests until #1353 is finished (which will simplify running local tests).

[lalitpagaria]

@tholor @fingoldo
I have bit hacky solution -

faiss_doc_store.faiss_indexes[index].reset()
faiss_doc_store.faiss_indexes.pop(index)
faiss_doc_store.vector_dim = 384
# To update config (Optional step) need to implement `to_dict()` method
# faiss_doc_store.set_config(**faiss_doc_store.to_dict())
# Now call `update_embedding`

[ZanSara]

Also, it seems that you have a small inaccuracy in the code.

file:
haystack\haystack\document_store\faiss.py

method:
def load(cls, index_path: Union[str, Path], config_path: Optional[Union[str, Path]] = None):

code:

        if not config_path:
            index_path = Path(index_path)
            faiss_init_params_path = index_path.with_suffix(".json")

        init_params: dict = {}
        try:
            with open(faiss_init_params_path, 'r') as ipp:
                init_params = json.load(ipp)
        except OSError as e:

if I provide config_path, load fails as faiss_init_params_path is not initialized.

@fingoldo The bug should be fixed now, check it out. Thank you for spotting it!