use relative keynames in storage and export files

[iandonnelly]

• simpleini
• ni
• tcl
• dump
• yajl with cascading keys
• xmltool (not really relevant, is superseded by xerces plugin)

I am currently having a problem with kdb import. When I try to import a file:
cat backup.ecf | sudo kdb import system/restore
None of the keys are added from backup.ecf (which is in dump format).

If I have a file such as backup.txt in line format,
cat backup.txt | sudo kdb import system/restore line
Works as expected.

Can anybody confirm?

[markus2330]

Without the content of backup.ecf it could be any issue, but I think you stumbled over the fact that dump only uses absolute names and that kdb import cannot rename keys.

See export_dump.md (please also write about that in the tutorials):

$ kdb set system/export_test/a b
create a new key system/export_test/a with string b

$ kdb export system/export_test > e.ecf

$ cat e.ecf
kdbOpen 1
ksNew 1
keyNew 21 2
system/export_test/ab
keyEnd
ksEnd

$ kdb rm system/export_test/a

$ kdb get system/export_test/a
Did not find key

$ cat e.ecf | kdb import system/export_test

$ kdb get system/export_test/a
b

$ cat e.ecf | kdb import system/export_test_renamed

$ kdb get system/export_test_renamed/a
Did not find key

$ kdb get system/export_test/a
b

Does this answer the question or did you stumble over another issue?

[iandonnelly]

That was it, I thought I specifically checked that but apparently not well enough, working now. Other formats don't have this limitation, correct?

[iandonnelly]

I want to add that I think that we should handle this better though. Rather than not importing at all, we should also print a message to the console informing the user of the error.

[markus2330]

There are some plugins that use absolute names, it should be mentioned in the contract, I added a TODO in CONTRACT.ini to add an explicit field for that information.

It is not really an error, because the user requested that just keys below system/export_test_renamed should be imported (but there are none).

Its like doing

  kdb ls system/export_test_renamed

and warning about not getting keys below system/export_test

We could, however, print information about which keys are going to be dropped in verbose mode.

Currently, they are not dropped, and kdb get system/export_test/a will be set, so lets keep the issue open and let us work out these situations.

[markus2330]

@easyxtarget you are right that it is not very obvious that dump has different behaviour than the others. The advantage dump has that it can contain user + system keys in one export file. (Currently, however, is this not possible to create such files because export does not have the feature)

So this absolute/relative stuff is certainly worth some considerations, but not for this release. (We need a backwards compatible solution anyway)

[markus2330]

further plugins that use absolute names: ni, tcl, xml

[markus2330]

ni is fixed in b91ba11

so we still have (at least): tcl, xmltool and dump

[markus2330]

Some notes: doc/decisions/relative.md describes a bit more the background.

And as scattert in some todos (e.g. TOOLS) the idea is that kdb export does an export for each namespace, making development of plugins easier.

E.g. when you do kdb export /hello ni you would get following output:

kdb import user/hello ni << UNIQUE_EOF
relative/path/below/user/hello
...
UNIQUE_EOF
kdb import system/hello ni << UNIQUE_EOF
relative/path/below/system/hello
...
UNIQUE_EOF

So one can simply execute the exported file and gets everything reimported without having to care about which parts are exported with which format.
(we

As said for dump we need some compatibility so that old dump files (with absolute pathes) can still be read and even be written if not configured to use the new-style.

[markus2330]

@sanssecours do you have something ready for this PR? E.g. the fix for cascading export/import?

[sanssecours]

@sanssecours do you have something ready for this PR? E.g. the fix for cascading export/import?

No. Otherwise I would have opened a pull request.

[markus2330]

Ok, I actually meant "which could be ready for this release" (which is unlikely to happen today or tomorrow, too many issues are still open). Maybe we should at least output a warning that cascading import/export currently is broken for non-relative plugins? Or even tag plugins if they are relative or absolute and have different import/export code. (I would prefer to avoid it, but it seems unlikely that all plugins will be fixed anytime soon, or will they?)

[sanssecours]

Ok, I actually meant "which could be ready for this release" (which is unlikely to happen today or tomorrow, too many issues are still open).

Sorry, I do not think so. Unless the release is in a few weeks/months.

Maybe we should at least output a warning that cascading import/export currently is broken for non-relative plugins? Or even tag plugins if they are relative or absolute and have different import/export code. (I would prefer to avoid it, but it seems unlikely that all plugins will be fixed anytime soon, or will they?)

I think tagging the sounds like a good idea. Although, ultimately I think we should just fix this issue. You are right though: I assume it will take quite some time until I am done with this.

[markus2330]

Ok, thanks for the update. You might over-estimate it: Once you have done one of the plugins, the others are really similar.

[markus2330]

This is also a serious issue for the web service, where the real path does not match what the user believes the absolute path is.

[markus2330]

I close this issue as dump and xmltool are soon only relevant as legacy formats. (See #2330 #1327)