locked profile after verdi storage maintain

[t-reents]

Hi,

I wanted to back up my AiiDA installation and I used verdi storage maintain --full to improve the performance of the repository back up. During the process, I lost my ssh connection and the packing of the loose files got interrupted. When I try to access the profile again via any verdi command, the profile is still locked and blocked due to a process with a given pid number.

Is there any possibility to unlock the profile?

Thanks a lot in advance!

[mbercx]

Pinging @ramirezfranciscof, since I believe you worked on the storage maintain command?

[t-reents]

I checked the source code and a naive assumption is that it might be sufficient to delete the .lock and .pid files in the access directory as it seems that the locking state is checked based on the existence of such files.

But, I would wait for your answer since I am not sure whether there could be other problems due to an interrupted maintain command.

[sphuber]

Correct, the locking is just done based on this lock file. So once you remove that, you can access the profile again. Note, that you should only do this if you are very sure that the process that locked it is no longer active and the lock file was not cleaned due to an improper shutdown, as in your case.

[t-reents]

Thanks a lot @sphuber

[t-reents]

@sphuber A small follow-up question. When I rerun the storage maintain command, I receive an error that the pack -1 already exists. It also states that this might be related to an aborted storage maintain which is true in my case. Is it also possible to delete the existing files in the packs directory and to rerun the storage maintain afterwards?

[sphuber]

That is a more delicate situation. When AiiDA is running, whenever it writes new files, it writes them as "loose" files, i.e., simple single files on disk. What storage maintain does is pack all these single files together into pack files which is more efficient. It is possible that this operation will delete the loose file as soon as it was appended to the pack file. This would meaning that deleting the pack file would cause data loss.

We should check the implementation of the disk-objectstore, which is used for the file repository, to see exactly what it does. Should be here: https://github.com/aiidateam/disk-objectstore/blob/develop/disk_objectstore/container.py#L1305

[sphuber]

The comments in the pack_all_loose suggest that deletion of loose files is not done there, once they are added to the packs, but that is left as a later manual operation.

The storage maintain command will call through to DiskObjectStoreRepositoryBackend.maintain, see: https://github.com/aiidateam/aiida-core/blob/main/aiida/repository/backend/disk_object_store.py#L144
It is clear that that calls first pack_all_loose and only when completed will call clean_container which will take care of deleting loose files that have been added to pack files.

So, it looks like it is written such that if the packing fails, as in your case, we should be able to delete those pack files and restart the packing, as the loose files will still be there. But I am not sure if there may be other problems. For example, the container keeps track of pack file indices and other data in an sqlite database. I am not sure if its state will still be consistent if you remove the pack files.

If this data is really important, here is what you could do. Create a backup of the repository folder, in its entirety. Then, delete the pack files and try to rerun verdi storage maintain. In the worst case, you can always restore the repository with its current state. Even though it is in an inconsistent state, at least all data should still be there. We can then try to delve deeper in the code and see what else needs to be reset.

[t-reents]

@sphuber Sorry for contacting you again but I further investigated the problem and I have some additional questions.

After rerunning the storage maintain based on the interrupted repacking, I realized that one pack is actually missing. Moreover, two packs show a smaller file size and I would only expect this for one (the last one) pack. Something seems to have failed after removing the pack -1 and rerunning the command. I will close the additional thread #6025 as it is obvious that there are problems while loading the arrays in case of missing files.

I had some further looks into the source code and I think that the SQL database which keeps track of the pack indices is actually making problems as you assumed. I think the problem is not directly related to the index as this is determined based on the existing packs: https://github.com/aiidateam/disk-objectstore/blob/develop/disk_objectstore/container.py#L258-L283
In my opinion, the problem arises here: https://github.com/aiidateam/disk-objectstore/blob/develop/disk_objectstore/container.py#L1455
The hash-keys of the loose files are stored with the corresponding pack index in the database. I think this information is also used when one tries to access the data of an array at a later point, if I am not mistaken.

This being said, I think it would be problematic to remove the pack files (no matter whether they are only packed or alreay repacked) as the loose files are already linked to a pack and I think it is not checked whether these packs do still exist: https://github.com/aiidateam/disk-objectstore/blob/develop/disk_objectstore/container.py#LL1334C1-L1351C60
So I think that it would not be possible to create new pack files. Moreover, as far as I understood, AiiDA would also try to read the data, e.g. of an array, from these (deleted) packs and not from the loose files, based on the SQL database.

In my current back-up, the packs already exist but as discussed, all loose files do still exist. My idea would be the following: Remove all packs and also delete the packs.idx (SQL database) file. As far as I remember, I never ran the storage maintain command before (I don't see any packs in my previous backup). My question would be whether other processes do also access the database. If this is not the case, I think it should work out when a new SQL database is created as it is done when init_container is called: https://github.com/aiidateam/disk-objectstore/blob/develop/disk_objectstore/container.py#L400
Then it should be possible to rerun the storage maintain command to pack the loose files and everything would be in a consistent state again. In case the database is accessed by other processes and contains additional information, it would probably not be possible to choose this approach.

If you are busy at the moment, please take your time. But I wanted to discuss this before doing such a step as it is not easy to get the full overview of all the processes.

[sphuber]

Thanks for the detailed investigation. The approach that you describe could work, but you will probably get an exception when you call init_container. I believe this raises if the directory of the container is not empty. So it won't actually just restore the database file.

Did you say you have a complete backup of the repository from before you called storage maintain first? In that case, your best bet would just to restore that copy and start from scratch with the maintenance. That should be fail-safe.

If you don't have that, and all you have is the half-way state where some files are already packed (and so the database may already contain those entries) but you still have all loose files as well. You might try the following approach (read in full before starting):

• Create full backup of the current repository state (container directory)
• Delete all pack files
• Delete packs.idx
• Now delete the original container directory
• Load an interactive shell and call init_container manually on the container directory. This should recreate an "empty" container.
• Now copy the contents of the loose folder of the backup to the newly initialized loose folder.
• Copy over config.json from the backed up container to the newly initialized container.

This last step is important as the container_id key is stored in AiiDA's postgresql database and is cross-referenced to check the database and object-store container belong to each other.

Again, not sure that this will work, but I think it might. Best bet is still to restore the original backup before the maintenance call.

[t-reents]

Actually, I did not want to call init_container, I just used it as an example method which shows the setup of the database file.

Unfortunately, I only have the backup where the packing was done and the process got interrupted during the repacking. So the database should already contain entries.

Before I try the second approach you mentioned, do I have to downgrade AiiDA again to version 2.0.4 which is the version where the backup was created? I just realized the data problem after upgrading to version 2.3. As this problem is only related to the file repository and not directly to the postgresql database, I was wondering whether it would work out without downgrading.

[sphuber]

There have been no database migrations between 2.0.4 and 2.3, so I think it shouldn't matter.

[t-reents]

It seems to work again. Thanks a lot for your great support!

[sphuber]

Thanks to you for the very detailed reports and analysis. Glad it is back to working properly