Import/export improvements

[szoupanos]

Here we can list some of the import/export improvements that could be implemented with the integration of import/export with the sharing implementation. The following open main issues should be addressed (independently on in parallel):

• Improve the performance of import/export (memory and maybe speed). The current implementation of import/export is based on creating or loading a big JSON file. The result is the need to have this file in memory during import or export which is cumbersome with big export files. We should move to a streaming protocol that has much smaller footprint.
• Import/export should be as backend independent as possible and code duplication should be minimised.

Sub-issues can/should be created from the list above. Moreover it can be extended.

Smaller issues like the ones mentioned in #110 should be looked at and if possible implemented. However, not all of them should be included to the import/export mechanism. Like the data transformation mechanism.

[giovannipizzi]

Using this as a tracker for the long-term project

[yakutovicha]

During the code week (03.12.2018 - 07.12.2018) together with @szoupanos and @lekah we agreed to do the following steps for the start:

• Split the file aiida/orm/importexport.py into import.py and export.py
• Rewrite import and export in a more modular way

[yakutovicha]

#493 is related to the export/import improvements

[ltalirz]

Issues related to import/export performance are better tracked in the import/export project https://github.com/aiidateam/aiida-core/projects/5

Just to clarify that no information gets lost by closing this issue:

• issue with export format requiring json file to be read in one go: continues to be tracked in export archive format forces reading of data.json file to memory #493
• making implementation backend-independent: I guess @CasperWA is aware of it, probably no need to open a separate issue for it
• Rewrite import and export in a more modular way: already in place; more in Rework export archive extraction #3242