Verdi export create fails when encountering sealed nodes

[jinchang]

We ran an export to create a backup of our database (using the --input-calc-forward flag from our starting data nodes). There are, however, still running processes which have corresponding unsealed nodes and we, therefore, get this exception (traceback below):

aiida.tools.importexport.common.exceptions.ExportValidationError: All ProcessNodes must be sealed before they can be exported. Node(s) with PK(s): 15625, 15632, 15636, 15639, 15642, 15643, 15644, 15645, 926, 927, 928, 929, 802, 803, 804, 799, 15646, 15529, 15544, 15548, 15551, 15554, 15555, 15556, 15557, 15558 is/are not sealed.

whilst it's good that these nodes are not exported it does mean that the backup fails and we're unable to make a backup in the current state.

Would it be possible to have AiiDA just warn about these nodes and continue with the export?

Traceback:

Traceback (most recent call last):
  File "/srv/mair/virtualenvs/mair/bin/verdi", line 8, in <module>
    sys.exit(verdi())
  File "/srv/mair/virtualenvs/mair/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/srv/mair/virtualenvs/mair/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/srv/mair/virtualenvs/mair/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/mair/virtualenvs/mair/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/mair/virtualenvs/mair/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/srv/mair/virtualenvs/mair/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/srv/mair/virtualenvs/mair/lib/python3.7/site-packages/aiida/cmdline/utils/decorators.py", line 68, in wrapper
    return wrapped(*args, **kwargs)
  File "/srv/mair/virtualenvs/mair/lib/python3.7/site-packages/aiida/cmdline/commands/cmd_export.py", line 138, in create
    export_function(entities, outfile=output_file, **kwargs)
  File "/srv/mair/virtualenvs/mair/lib/python3.7/site-packages/aiida/tools/importexport/dbexport/__init__.py", line 512, in export
    export_tree(what, folder=folder, silent=silent, **kwargs)
  File "/srv/mair/virtualenvs/mair/lib/python3.7/site-packages/aiida/tools/importexport/dbexport/__init__.py", line 395, in export_tree
    check_process_nodes_sealed(process_nodes)
  File "/srv/mair/virtualenvs/mair/lib/python3.7/site-packages/aiida/tools/importexport/dbexport/utils.py", line 275, in check_process_nodes_sealed
    'Node(s) with PK(s): {} is/are not sealed.'.format(', '.join(str(pk) for pk in nodes - sealed_nodes))

[sphuber]

If a node is not sealed, that means that it corresponds to a process that is still active. Exporting it is dangerous because the resulting archive will have an inconsistent provenance graph. This is why we forbid them from being exported and it is not likely that we will change this. If your goal is to create a backup of your AiiDA installation, there are better ways to do this. Please refer to the documentation on backups that should explain in detail how to create efficient backups and how to restore them. Let me know if you run into trouble there of have any questions.

[muhrin]

Hey Seb, I wrote this with @jinchang last night. It is indeed for backup/replication purposes but it would be great to be able to do an export so that we're not forced to take the whole DB at once. For example, I do some development where I import this snapshot of our 'master' database into my existing one keeping everything else that's already in there. Not to mention that an export is significantly less laborious than a backup.

So what I would love to see here is for unsealed nodes and descendents to simply be excluded. What do you think?

Happy to look into making the code changes if we decide to go this way.

[sphuber]

I see now, apologies for the misunderstanding. An option to omit unsealed from the export instead of just printing a warning is significantly different and that is something that we could look into.

So what I would love to see here is for unsealed nodes and descendents to simply be excluded. What do you think?

Since v1.0.0 we have put very strict rules on what sub-graphs are allowed to exported. The contents of an archive should represent "consistent" graphs. What that means is explained in detail in the documentation. For example, it is forbidden to export a calculation node without its inputs, because whoever would import that archive would have a calculation node with broken provenance. All this to say that adding an option to skip unsealed nodes maybe possible, but under these constraints.

[muhrin]

Yup, I thought this would be the case. So, I'll have a look and report back when I see you next week. In essence my 'ignore unsealed calculations and any descendents' policy shouldn't violate the consistent graphs rule.