Dissect: * and & reference modifier, breaking change for 7.0

[ph]

In the current dissect implementation you can use a named skip field as a reference key, we want to change the behavior to use the * instead of using a named skip field this will make it more obvious and will be closer to pointers and reference in programming language. You will still be able to have a named skip field but the value will not be accessible.

Extract from the specification

? - Named skip key instructs the parser to not include this result in the final result set. Behaves identical to an empty skip key %{} but may be used to help with human readability. The ? modifier must be placed to the left of the key name. see example below

* and & reference modifiers. This modifier requires two keys with the same name present in the dissect pattern. One key with the * and another with the &. This instructs the parser that the value discovered by the * is to be used as the key name for the value discovered by the corresponding & key. These modifiers must be placed on the left of the key name. see example below

[ruflin]

Will there be a migration path? Assuming we use it in some of our modules, how could we make sure that Beats 7 still works with ES 6?

[ph]

@ruflin I don't think there will be issues with beats7 and ES6, are you talking about possible issues with the ingest node the dissect processor in ES6? FYI ES6.5 will already use the new syntax.

For a migration path, this is another thing, I could make both syntax works in 6.5 and remove it in 7.0. I need to evaluate the complexity of it.

But note the change should be minimal, %{?mykey} to %{*mykey}

Also we don't use that specific syntax in our modules, its a niche use case :)

[ruflin]

Thanks for the details. As we don't use the feature yet, it should not be an issue. Assuming we start using it in one of the 6.x modules it would mean we have to about the dissect processor for 7.0. As we do the processing on our end, it would not have any effect if the users uses ES 6.x or 7.x.

[ph]

It would nice to do the following at the same time: #8123