Minor branching modification to matching policy (route branch)

[johnhtodd]

Is your feature request related to a problem? Please describe.
It seems that there could be a better branching function easily implemented in the match/non-matching concept.

Right now, there is a matching function which has "include" and "exclude" criteria. Also, there is a "policy" concept, which says what to do with messages that do not meet the matching criteria, but that policy: concept has a critical flaw - it can only jump to the routes: specified at the end of the statement, which is ALSO where all the matching results are sent after the transformations. This model essentially forces branching behaviors closer to ingestion in order to avoid this inadvertent combining of routing.

Describe the solution you'd like
It seems that instead of a "policy" statement, you could have a "route-unmatched:" statement that would give the ability to fork queries based on their meeting or not meeting the "include:" criteria. It is implicit that anything that meets the "matching: include:" statements will then have the transformations or aggregations or other actions applied to them, and then they can proceed to a next route statement. Adding "route-unmatched:" would allow immediate branching to a different path.

(also: is "exclude:" just another way of saying that the message is unmatched?)

Example of how this path logic has to work today:

# Take in data from a dnstap stream.

  - name: dnsdist-from-outside
    dnstap:
      listen-ip: 0.0.0.0
      listen-port: 59310
    routes: [ check-for-google, main-process ]

# This is a "dead-end" check - once we add the tag, we don't want to put
#  it back in our main processing flow.  We want to send it to a dnstap forwarder
#  and end the process.  Therefore, we MUST split the stream from dnsdist-from-outside
#  and duplicate the ENTIRE data set to be evaluated twice. 

  - name: check-for-google
    dnsmessage:
      matching:
        include:
          dns.qname: "^.*\\.google\\.com$"
      policy: "block-unmatched"
      transforms:
        atags:
          tags: [ "TXT:Le-Goog" ]
    routes: [ send-to-dnstap-forwarder ]

  - name: send-to-dnstap-forwarder
    dnstap:
      <specify dnstap receiver etc.>

  - name: main-process
     dnsmessage:
      <stuff below here that we want to do>

Here's the example below of how it could look with a slightly different method:

We get flexibility if this is what things look like - messages are not split, and actions taken on matching messages are not necessarily inserted back into the single pathway. This permits forking on match, but keeping most messages in a single unduplicated stream for lower processing overhead.

# Take in data from a dnstap stream.

  - name: dnsdist-from-outside
    dnstap:
      listen-ip: 0.0.0.0
      listen-port: 59310
    routes: [ check-for-google ]

  - name: check-for-google
    dnsmessage:
      matching:
        include:
          dns.qname: "^.*\\.google\\.com$"
        route-unmatched: [ main-process ]    <-- this is the key line
      transforms:
        atags:
          tags: [ "TXT:Le-Goog" ]
    routes: [ send-to-dnstap-forwarder ]

  - name: send-to-dnstap-forwarder
    dnstap:
      <specify dnstap receiver etc.>

  - name: main-process
     dnsmessage:
      <stuff below here that we want to do>

Describe alternatives you've considered
It is possible to achieve the same outcomes by creating multiple route destinations, each leading to a specific conclusion. This seems quite wasteful, and means every packet has to be evaluated many times over, and also requires strict include/exclude matching sets as the contexts become deeper.

[dmachard]

I have this type of feature in mind, but I am not yet clear on how to specify properly it in the YAML configuration.
I would like to find a better description between routes, drop-policy and route-unmatched

[dmachard]

Perhaps something like that, which is generic:

To apply specific routes

routing-policy:
  accepted: [ stanzaA ]
  discarded: [ stanzaB ] 

To discard unmatched packet

routing-policy:
  accepted: [ stanzaA ]
  discarded: [  ] 

And finally, the default routing for backward compatibility with any checks

routing-policy:
  default: [ stanzaA, stanzaB, .... ] 

[johnhtodd]

The model you describe in the last comment seems to be a good way of approaching the issue, of course assuming that there can (as usual) be multiple paths specified in the routing output for each option. Example:

routing-policy:
  accepted: [ stanzaA, stanzaC ]
  discarded: [ stanzaB, stanzaD ] 

Question: Would objects that match an "exclude:" statement automatically go to the "discarded:" route path? Are "exclude:" rules considered first, then, before "include:" rules?

[dmachard]

Multiple path is supported as usual.
Here the new syntax implemented

    routing-policy:
      dropped: [ outputfile ]
      default: [ console ]

Question: Would objects that match an "exclude:" statement automatically go to the "discarded:" route path? Are
"exclude:" rules considered first, then, before "include:" rules?

Exclude and include rules works together with a logical AND.
A DNS message will be dropped/discarded if the rules are not satisfied (include AND exclude)

[dmachard]

Completed but need documentations

[dmachard]

implemented in #719
The pipeline mode with be the default mode for next release.