Field Reference: handle special characters

docs/static/field-reference.asciidoc

@@ -4,7 +4,8 @@
 It is often useful to be able to refer to a field or collection of fields by name. To do this,
 you can use the Logstash field reference syntax.
 
-The syntax to access a field specifies the entire path to the field, with each fragment wrapped in square brackets.
+The syntax to access a field specifies the entire path to the field, with each fragment wrapped in square brackets;
+when a field name contains square brackets, they must be properly <<formal-grammar-escape-sequences, _escaped_>>.
 
 _Field References_ can be expressed literally within <<conditionals,_Conditional_>> statements in your pipeline configurations,
 as string arguments to your pipeline plugins, or within sprintf statements that will be used by your pipeline plugins:
@@ -133,3 +134,15 @@ embeddedFieldReference
   ;
 
 An _Embedded Field Reference_ is a _Field Reference_ that is itself wrapped in square brackets (`[` and `]`), and can be a component of a _Composite Field Reference_.
+
+[float]
+[[formal-grammar-escape-sequences]]
+=== Escape Sequences
+
+In order to reference a field whose name contains a character that has special meaning in the field reference grammar, it needs to be escaped.
+Logstash can be globally configured to use one of two field reference escape modes:
+
+ - `NONE` (default): no escape sequence processing is done; fields containing literal square brackets cannot be referenced by the Event API.
+ - `PERCENT`: URI-style percent encoding of UTF-8 bytes; the left square bracket (`[`) is expressed as `%5B`, and the right square bracket (`]`) is expressed as `%5D`.
+// NOTE: the following is _also_ HTML-escaped in the asciidoc source document so that browsers rendering the HTML will unwrap one escape and leave the remaining.
+ - `AMPERSAND`: HTML-style ampersand encoding (`&#` + decimal unicode codepoint + `;`); the left square bracket (`[`) is expressed as `&amp;#91;`, and the right square bracket (`]`) is expressed as `&amp;#93;`.

docs/static/settings-file.asciidoc

@@ -178,6 +178,17 @@ Values other than `disabled` are currently considered BETA, and may produce unin
 | When set to `true`, quoted strings will process the following escape sequences: `\n` becomes a literal newline (ASCII 10). `\r` becomes a literal carriage return (ASCII 13). `\t` becomes a literal tab (ASCII 9). `\\` becomes a literal backslash `\`. `\"` becomes a literal double quotation mark. `\'` becomes a literal quotation mark.
 | `false`
 
+| `config.field_reference.escape_style`
+a| _EXPERIMENTAL_ setting that provides a way to reference fields that contain <<formal-grammar-escape-sequences,field reference special characters>> `[` and `]`.
+
+Current options are:
+
+* `PERCENT`: URI-style `%`+`HH` hexadecimal encoding of UTF-8 bytes (`[` -> `%5B`; `]` -> `%5D`)
+* `AMPERSAND`: HTML-style `&#`+`DD`+`;` encoding of decimal Unicode code-points (`[` -> `&amp;#91;`; `]` -> `&amp;#91;`)
+* `NONE`: field names containing special characters _cannot_ be referenced.
+
+| `NONE`
+
 | `modules`
 | When configured, `modules` must be in the nested YAML structure described above this table.
 | None

logstash-core/lib/logstash/agent.rb

@@ -66,6 +66,14 @@ def initialize(settings = LogStash::SETTINGS, source_loader = nil)
     # Generate / load the persistent uuid
     id
 
+    field_reference_escape_style_setting = settings.get_setting('config.field_reference.escape_style')
+    if field_reference_escape_style_setting.set?
+      logger.warn(I18n.t("logstash.settings.experimental.set", canonical_name: field_reference_escape_style_setting.name))
+    end
+    field_reference_escape_style = field_reference_escape_style_setting.value
+    logger.debug("Setting global FieldReference escape style: #{field_reference_escape_style}")
+    org.logstash.FieldReference::set_escape_style(field_reference_escape_style)
+
     # Initialize, but do not start the webserver.
     @webserver = LogStash::WebServer.from_settings(@logger, self, settings)
 

logstash-core/lib/logstash/environment.rb

@@ -49,6 +49,7 @@ module Environment
            Setting::Boolean.new("config.reload.automatic", false),
            Setting::TimeValue.new("config.reload.interval", "3s"), # in seconds
            Setting::Boolean.new("config.support_escapes", false),
+            Setting::String.new("config.field_reference.escape_style", "NONE", true, %w(NONE PERCENT AMPERSAND)),
            Setting::Boolean.new("metric.collect", true),
             Setting::String.new("pipeline.id", "main"),
            Setting::Boolean.new("pipeline.system", false),

logstash-core/lib/logstash/runner.rb

@@ -87,6 +87,11 @@ class LogStash::Runner < Clamp::StrictCommand
     :default => LogStash::SETTINGS.get_default("config.string"),
     :attribute_name => "config.string"
 
+  option ["--field-reference-escape-style"], "STYLE",
+         I18n.t("logstash.runner.flag.field-reference-escape-style"),
+         :default => LogStash::SETTINGS.get_default("config.field_reference.escape_style"),
+         :attribute_name => "config.field_reference.escape_style"
+
   # Module settings
   option ["--modules"], "MODULES",
     I18n.t("logstash.runner.flag.modules"),

logstash-core/locales/en.yml

@@ -219,6 +219,31 @@ en:
           "%{default_output}"
           If you wish to use both defaults, please use
           the empty string for the '-e' flag.
+        field-reference-escape-style: |+
+          Use the given STYLE when parsing field
+          references. This allows you to reference fields
+          whose name includes characters that are
+          meaningful in a field reference including square
+          brackets (`[` and `]`).
+
+          This feature is EXPERIMENTAL, and implementations
+          are subject to change.
+
+          Available escape styles are:
+           - `NONE`: escape sequences in field references
+             are not processed, which means fields that
+             contain special characters cannot be
+             referenced.
+           - `PERCENT`: characters may be encoded with
+             URI-style percent notation represeting UTF-8
+             bytes (`[` is `%5B`; `]` is `%5D`).
+             Unlike URI-encoding, literal percent characters
+             do not need to be escaped unless followed by a
+             sequence of 2 capital hexadecimal characters.
+           - `AMPERSAND`: characters may be encoded with
+             HTML-style ampersand-hash encoding notation
+             representing decimal unicode codepoints
+             (`[` is `[`; `]` is `]`).
         modules: |+
           Load Logstash modules.
           Modules can be defined using multiple instances
@@ -431,4 +456,8 @@ en:
         ambiguous: >-
           Both `%{canonical_name}` and its deprecated alias `%{deprecated_alias}` have been set.
           Please only set `%{canonical_name}`
+      experimental:
+        set: >-
+          The setting `%{canonical_name}` is EXPERIMENTAL and its implementation is subject to change
+          in a future release of Logstash
 

logstash-core/src/main/java/org/logstash/FieldReference.java

@@ -27,8 +27,10 @@
 import java.util.Map;
 
 import java.util.concurrent.ConcurrentHashMap;
+import java.util.stream.Collectors;
 
 import org.jruby.RubyString;
+import org.logstash.util.EscapeHandler;
 
 /**
  * Represents a reference to another field of the event {@link Event}
@@ -45,19 +47,39 @@ public static class IllegalSyntaxException extends RuntimeException {
         }
     }
 
+    private static EscapeHandler ESCAPE_HANDLER = EscapeHandler.NONE;
+
+    public static void setEscapeStyle(final String escapeStyleSpec) {
+        final EscapeHandler newEscapeHandler;
+        switch(escapeStyleSpec) {
+            case "NONE":
+                newEscapeHandler = EscapeHandler.NONE;
+                break;
+            case "PERCENT":
+                newEscapeHandler = EscapeHandler.PERCENT;
+                break;
+            case "AMPERSAND":
+                newEscapeHandler = EscapeHandler.AMPERSAND;
+                break;
+            default:
+                throw new IllegalArgumentException(String.format("Invalid escape style: `%s`", escapeStyleSpec));
+        }
+        ESCAPE_HANDLER = newEscapeHandler;
+    }
+
     /**
      * This type indicates that the referenced that is the metadata of an {@link Event} found in
-     * {@link Event#metadata}.
+     * {@link Event#getMetadata()}.
      */
     public static final int META_PARENT = 0;
 
     /**
-     * This type indicates that the referenced data must be looked up from {@link Event#metadata}.
+     * This type indicates that the referenced data must be looked up from {@link Event#getMetadata()}.
      */
     public static final int META_CHILD = 1;
 
     /**
-     * This type indicates that the referenced data must be looked up from {@link Event#data}.
+     * This type indicates that the referenced data must be looked up from {@link Event#getData()}.
      */
     private static final int DATA_CHILD = -1;
 
@@ -73,11 +95,6 @@ public static class IllegalSyntaxException extends RuntimeException {
      */
     private static final StrictTokenizer TOKENIZER = new StrictTokenizer();
 
-    /**
-     * Unique {@link FieldReference} pointing at the timestamp field in a {@link Event}.
-     */
-    public static final FieldReference TIMESTAMP_REFERENCE = FieldReference.from(Event.TIMESTAMP);
-
     /**
      * Cache of all existing {@link FieldReference} by their {@link RubyString} source.
      */
@@ -90,6 +107,11 @@ public static class IllegalSyntaxException extends RuntimeException {
     private static final Map<String, FieldReference> CACHE =
         new ConcurrentHashMap<>(64, 0.2F, 1);
 
+    /**
+     * Unique {@link FieldReference} pointing at the timestamp field in a {@link Event}.
+     */
+    public static final FieldReference TIMESTAMP_REFERENCE = FieldReference.from(Event.TIMESTAMP);
+
     private final String[] path;
 
     private final String key;
@@ -217,7 +239,9 @@ private static FieldReference parseToCache(final String reference) {
     }
 
     private static FieldReference parse(final CharSequence reference) {
-        final List<String> path = TOKENIZER.tokenize(reference);
+        final List<String> path = TOKENIZER.tokenize(reference).stream()
+                .map(ESCAPE_HANDLER::unescape)
+                .collect(Collectors.toList());
 
         final String key = path.remove(path.size() - 1);
         final boolean empty = path.isEmpty();

logstash-core/src/main/java/org/logstash/util/EscapeHandler.java

@@ -0,0 +1,73 @@
+package org.logstash.util;
+
+import org.apache.commons.codec.DecoderException;
+import org.apache.commons.codec.binary.Hex;
+
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.regex.Pattern;
+
+public interface EscapeHandler {
+    String unescape(String escaped);
+    String escape(String unescaped);
+
+    EscapeHandler NONE = new EscapeHandler() {
+        @Override
+        public String unescape(final String escaped) {
+            return escaped;
+        }
+
+        @Override
+        public String escape(final String unescaped) {
+            return unescaped;
+        }
+    };
+
+    EscapeHandler PERCENT = new EscapeHandler() {
+        private final Pattern ESCAPE_REQUIRED_PERCENT_LITERAL = Pattern.compile("%(?=[0-9A-F]{2})");
+
+        @Override
+        public String escape(final String unescaped) {
+            // When a percent-literal is followed by a pair of hex digits, we must escape it.
+            return ESCAPE_REQUIRED_PERCENT_LITERAL.matcher(unescaped).replaceAll("%25")
+                    .replace("[", "%5B")
+                    .replace("]", "%5D");
+        }
+
+        private final Pattern PERCENT_ENCODED_SEQUENCE = Pattern.compile("%[0-9A-F]{2}");
+        private final Pattern UNESCAPED_PERCENT_LITERAL = Pattern.compile("%(?![0-9A-F]{2})");
+
+        public String unescape(String escaped) {
+            if (!PERCENT_ENCODED_SEQUENCE.matcher(escaped).find()) { return escaped; }
+
+            // In order to support unescaped percent-literals without implementing
+            // our own percent-decoder, we need to detect them and escape them before
+            // handing off to java's URLDecoder.
+            escaped = UNESCAPED_PERCENT_LITERAL.matcher(escaped).replaceAll("%25");
+
+            return URLDecoder.decode(escaped, StandardCharsets.UTF_8);
+        }
+    };
+
+    EscapeHandler AMPERSAND = new EscapeHandler() {
+        private final Pattern AMPERSAND_ENCODED_SEQUENCE = Pattern.compile("&#([0-9]{2,});");
+
+        @Override
+        public String escape(final String unescaped) {
+            return unescaped.replaceAll(AMPERSAND_ENCODED_SEQUENCE.pattern(), "&#$1;")
+                    .replace("[", "[")
+                    .replace("]", "]");
+        }
+
+        @Override
+        public String unescape(final String escaped) {
+            if (!escaped.contains("&")) { return escaped; }
+
+            return AMPERSAND_ENCODED_SEQUENCE.matcher(escaped).replaceAll(matchResult -> {
+                final int codePoint = Integer.parseInt(matchResult.group(1));
+                final char[] chars = Character.toChars(codePoint);
+                return String.copyValueOf(chars);
+            });
+        }
+    };
+}