Slow semantic index process

[Illia-M]

I have 221587 records\data to index for search and when I try change index model I need reindex whole data.
Now I see that it cannot utilize more then 1 core and process need about 4-7 day to create full index, but because of interaction(we have blackouts time to time in Ukraine) I cannot finish it for 3 weeks 🫠
Any plan to parallelize that reindex process or some other way to speedup?
I has 4 cores, 8 threads Intel CPU with 32 GB RAM that cannot be utilized(
Gramps 5.2.3
Gramps Web API 2.6.0
Gramps Web Frontend 24.12.0
Gramps QL 0.3.0
Sifts 1.0.0
locale: uk
multi-tree: false
task queue: true
OCR: true
chat: true

[DavidMStraub]

Unfortunately, I had to explicitly set OMP_NUM_THREADS to 1 because PyTorch multiprocessing does not work with celery threads. We could think about a standalone script that can use multiprocessing.

I guess it would be much faster on GPU, but I don't have one to try...

[DavidMStraub]

By the way, the Web API supports incremental reindexing, i.e. you can compute it in several batches if it's interrupted. We should add a button to the UI to trigger that (trivial to implement)

[Illia-M]

Maybe some explicit way to set how many threads to use in reindex? In UI or env vars

[Illia-M]

And yes - some way to "continue where you stop" will help)

[DavidMStraub]

Maybe some explicit way to set how many threads to use in reindex? In UI or env vars

No, that doesn't work in the app itself due to the PyTorch vs Celery issue. It would have to be a standalone script.