AiiDA import is eager on disk space

[waychal]

I am trying to import data from https://archive.materialscloud.org/file/2019.0044/v2/automatic_wannier_provenance.aiida (23.3 GBs) on a ubuntu machine having 16 GBs RAM, 4 VCPUs and 500 GBs mounted volume.

While importing, it gives below error:

IOError: [Errno 28] No space left on device

In extracted data, the dump of the DB is 500MB (uncompressed), and the file repository is 247GB.

[sphuber]

This is not so much an import problem I think but just a generic problem with AiiDA's repository implementation. Especially the fact that there is no built-in data deduplication nor an option to bundle and compress content stored in the repository. Please correct me if I'm wrong but I think the actual repository on disk of the original database is also 247 GB, is that possible @giovannipizzi ? The contents are just compressed for the export file, but the import then needs to decompress it again to copy it to the repository of the profile that is importing it.

[ltalirz]

Closing this as a duplicate of point 3. in #2399