linenumbers for validation

[markus2330]

Currently, when validation fails you know which key is the problematic one. This is enough for most use-cases (e.g. when working with tools such as kdb), but not ideal when directly working with configuration files.

In a7544a5 I propose that storage plugins can add the line numbers where they found each key. This meta data should be added to validation errors (if available).

The downsides are that it increases memory footprint and, again, needs changes to every storage plugin that wants to support it. (Most parsers do not have this information available at all, so for them it won't work anyway).

@sanssecours @tom-wa What do you think?

Btw. Elektra seems to be one of the few tool that properly supports validation of multiple files at once. @tom-wa can you search for others as related work? See also discussion here

[tom-wa]

i like that idea.
but we could also add a function to storage plugins providing/returning additional informations for a key and export it. let's assume i'd have no way of passing a line number from the parser to the storage plugin, the storage plugin would still have other relevant informations. bad example: the ini plugin could still tell which section a key would belong to, what position within the section the key has, etc. ofc a line number would be better, but if it's not available it's still better than nothing

[markus2330]

I also thought about having a search for the line after-the-fact. But I think this would lead to poor results. When the parser does not support it, I am afraid you simply cannot get the feature in a way so that is reliable and useful. (You easily could get confused e.g. by multi-line comments)

For quite some plugins, however, the line numbers could be added easily. (e.g. for tcl https://gist.github.com/sehe/1425972)

Another question is if we use start line/end line or even start column/end column, too?

[markus2330]

Btw. as global validation plugin you even lose the information in which file an option was (and obviously also won't know which storage plugin was involved). A similar issue would also happen with a multi-file plugin resolver.

So we should even add the file names during parsing (next to line information). The file name only takes a pointer for every key, so it should not matter much. The line-number information would be something about 50 byte per key (depending on how much information we add and how well @mpranj can optimize keyNew for meta-data). Ini already adds about 3 meta-keys per key, so you would not notice the overhead of the line information.

For some other plugins that currently do not add any meta-data at all, an option to disable the adding conffile-based information might be useful: to not cause overhead for people who do not edit files directly anyway.

[markus2330]

@bauhaus93 maybe this can be realized in the new TOML plugin? Maybe the order can directly be the line number?

@sanssecours any new thoughts on that?

[sanssecours]

Maybe the order can directly be the line number?

For JSON (and YAML flow style) that will not work, since more that one key can appear on the same line. For example the code

["one", "two"]

mapped to the namespace user translates to

user/#0: one
user/#1: two

, where the line number of both the key user/#0 and user/#1 is 1. As far as I can tell TOML also supports (nested) arrays, so the same will be true for that language too.

@sanssecours any new thoughts on that?

Storing the line number should not be that hard, although we have to consider that keys, can also span multiple lines. For example, if we map the following YAML code

root:
  level1:
    level2:
      value

to the namespace user, then the only key root/level1/level2 spans from line 1 till line 3. That possibility should apply to TOML as well, as far as I know. For example, I assume that the following TOML data:

[clients]
data = [ ["gamma", "delta"], [1, 2] ]

maps to the key set:

user/clients/data/#0/#0: gamma
user/clients/data/#0/#1: delta
user/clients/data/#1/#0: 1
user/clients/data/#1/#1: 2

, if we mount it directly at the namespace user. In the code above all keys span from line 1 to line 2 in the TOML file.

[markus2330]

Thank you for the detailed explanation!

[markus2330]

Won't be implemented soon.