Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ECS schema for user-agent ingest processor #37727

Merged
merged 9 commits into from
Jan 25, 2019
Merged

Add ECS schema for user-agent ingest processor #37727

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
f7aa7de
Add ECS schema for user-agent injest processor
dakrone Jan 22, 2019
5ba7839
Fix/add documentation
dakrone Jan 22, 2019
abe6ce5
Remove extra trailing brace
dakrone Jan 23, 2019
2608cea
Merge remote-tracking branch 'origin/6.x' into ingest-user-agent-ecs-6x
dakrone Jan 23, 2019
88d78f6
Merge remote-tracking branch 'origin/6.x' into ingest-user-agent-ecs-6x
dakrone Jan 23, 2019
490f438
Merge remote-tracking branch 'origin/6.x' into ingest-user-agent-ecs-6x
dakrone Jan 24, 2019
e88ba82
Merge remote-tracking branch 'origin/6.x' into ingest-user-agent-ecs-6x
dakrone Jan 24, 2019
f087784
Add deprecation warning for deprecated properties
dakrone Jan 25, 2019
9931f27
Merge remote-tracking branch 'origin/6.x' into ingest-user-agent-ecs-6x
dakrone Jan 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions docs/reference/ingest/processors/user-agent.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ The ingest-user-agent module ships by default with the regexes.yaml made availab
| `regex_file` | no | - | The name of the file in the `config/ingest-user-agent` directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with (see below).
| `properties` | no | [`name`, `major`, `minor`, `patch`, `build`, `os`, `os_name`, `os_major`, `os_minor`, `device`] | Controls what properties are added to `target_field`.
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
| `ecs` | no | `false` | Whether to return the output in Elastic Common Schema format. NOTE: ECS format will be the default in Elasticsearch 7.0 and non-ECS format is deprecated.
|======

Here is an example that adds the user agent details to the `user_agent` field based on the `agent` field:
Expand All @@ -31,7 +32,8 @@ PUT _ingest/pipeline/user_agent
"processors" : [
{
"user_agent" : {
"field" : "agent"
"field" : "agent",
"ecs" : true
}
}
]
Expand Down Expand Up @@ -60,13 +62,13 @@ Which returns
"agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
"user_agent": {
"name": "Chrome",
"major": "51",
"minor": "0",
"patch": "2704",
"os_name": "Mac OS X",
"os": "Mac OS X 10.10.5",
"os_major": "10",
"os_minor": "10",
"original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
"version": "51.0.2704",
"os": {
"name": "Mac OS X",
"version": "10.10.5",
"full": "Mac OS X 10.10.5"
},
"device": "Other"
}
}
Expand Down
228 changes: 160 additions & 68 deletions ...ngest-user-agent/src/main/java/org/elasticsearch/ingest/useragent/UserAgentProcessor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@

package org.elasticsearch.ingest.useragent;

import org.apache.logging.log4j.LogManager;
import org.elasticsearch.common.logging.DeprecationLogger;
import org.elasticsearch.ingest.AbstractProcessor;
import org.elasticsearch.ingest.IngestDocument;
import org.elasticsearch.ingest.Processor;
Expand Down Expand Up @@ -47,23 +49,25 @@ public class UserAgentProcessor extends AbstractProcessor {
private final Set<Property> properties;
private final UserAgentParser parser;
private final boolean ignoreMissing;
private final boolean useECS;

public UserAgentProcessor(String tag, String field, String targetField, UserAgentParser parser, Set<Property> properties,
boolean ignoreMissing) {
boolean ignoreMissing, boolean useECS) {
super(tag);
this.field = field;
this.targetField = targetField;
this.parser = parser;
this.properties = properties;
this.ignoreMissing = ignoreMissing;
this.useECS = useECS;
}

boolean isIgnoreMissing() {
return ignoreMissing;
}

@Override
public IngestDocument execute(IngestDocument ingestDocument) throws Exception {
public IngestDocument execute(IngestDocument ingestDocument) {
String userAgent = ingestDocument.getFieldValue(field, String.class, ignoreMissing);

if (userAgent == null && ignoreMissing) {
Expand All @@ -75,71 +79,134 @@ public IngestDocument execute(IngestDocument ingestDocument) throws Exception {
Details uaClient = parser.parse(userAgent);

Map<String, Object> uaDetails = new HashMap<>();
for (Property property : this.properties) {
switch (property) {
case NAME:
if (uaClient.userAgent != null && uaClient.userAgent.name != null) {
uaDetails.put("name", uaClient.userAgent.name);
}
else {
uaDetails.put("name", "Other");
}
break;
case MAJOR:
if (uaClient.userAgent != null && uaClient.userAgent.major != null) {
uaDetails.put("major", uaClient.userAgent.major);
}
break;
case MINOR:
if (uaClient.userAgent != null && uaClient.userAgent.minor != null) {
uaDetails.put("minor", uaClient.userAgent.minor);
}
break;
case PATCH:
if (uaClient.userAgent != null && uaClient.userAgent.patch != null) {
uaDetails.put("patch", uaClient.userAgent.patch);
}
break;
case BUILD:
if (uaClient.userAgent != null && uaClient.userAgent.build != null) {
uaDetails.put("build", uaClient.userAgent.build);
}
break;
case OS:
if (uaClient.operatingSystem != null) {
uaDetails.put("os", buildFullOSName(uaClient.operatingSystem));
}
else {
uaDetails.put("os", "Other");
}

break;
case OS_NAME:
if (uaClient.operatingSystem != null && uaClient.operatingSystem.name != null) {
uaDetails.put("os_name", uaClient.operatingSystem.name);
}
else {
uaDetails.put("os_name", "Other");
}
break;
case OS_MAJOR:
if (uaClient.operatingSystem != null && uaClient.operatingSystem.major != null) {
uaDetails.put("os_major", uaClient.operatingSystem.major);
}
break;
case OS_MINOR:
if (uaClient.operatingSystem != null && uaClient.operatingSystem.minor != null) {
uaDetails.put("os_minor", uaClient.operatingSystem.minor);
}
break;
case DEVICE:
if (uaClient.device != null && uaClient.device.name != null) {
uaDetails.put("device", uaClient.device.name);
}
else {
uaDetails.put("device", "Other");
}
break;
if (useECS) {
// Parse the user agent in the ECS (Elastic Common Schema) format
for (Property property : this.properties) {
switch (property) {
case ORIGINAL:
uaDetails.put("original", userAgent);
break;
case NAME:
if (uaClient.userAgent != null && uaClient.userAgent.name != null) {
uaDetails.put("name", uaClient.userAgent.name);
} else {
uaDetails.put("name", "Other");
}
break;
case VERSION:
StringBuilder version = new StringBuilder();
if (uaClient.userAgent != null && uaClient.userAgent.major != null) {
version.append(uaClient.userAgent.major);
if (uaClient.userAgent.minor != null) {
version.append(".").append(uaClient.userAgent.minor);
if (uaClient.userAgent.patch != null) {
version.append(".").append(uaClient.userAgent.patch);
if (uaClient.userAgent.build != null) {
version.append(".").append(uaClient.userAgent.build);
}
}
}
uaDetails.put("version", version.toString());
}
break;
case OS:
if (uaClient.operatingSystem != null) {
Map<String, String> osDetails = new HashMap<>(3);
if (uaClient.operatingSystem.name != null) {
osDetails.put("name", uaClient.operatingSystem.name);
StringBuilder sb = new StringBuilder();
if (uaClient.operatingSystem.major != null) {
sb.append(uaClient.operatingSystem.major);
if (uaClient.operatingSystem.minor != null) {
sb.append(".").append(uaClient.operatingSystem.minor);
if (uaClient.operatingSystem.patch != null) {
sb.append(".").append(uaClient.operatingSystem.patch);
if (uaClient.operatingSystem.build != null) {
sb.append(".").append(uaClient.operatingSystem.build);
}
}
}
osDetails.put("version", sb.toString());
osDetails.put("full", uaClient.operatingSystem.name + " " + sb.toString());
}
uaDetails.put("os", osDetails);
}
}
break;
case DEVICE:
if (uaClient.device != null && uaClient.device.name != null) {
uaDetails.put("device", uaClient.device.name);
} else {
uaDetails.put("device", "Other");
}
break;
}
}
} else {
// Deprecated format, removed in 7.0
for (Property property : this.properties) {
switch (property) {
case NAME:
if (uaClient.userAgent != null && uaClient.userAgent.name != null) {
uaDetails.put("name", uaClient.userAgent.name);
} else {
uaDetails.put("name", "Other");
}
break;
case MAJOR:
if (uaClient.userAgent != null && uaClient.userAgent.major != null) {
uaDetails.put("major", uaClient.userAgent.major);
}
break;
case MINOR:
if (uaClient.userAgent != null && uaClient.userAgent.minor != null) {
uaDetails.put("minor", uaClient.userAgent.minor);
}
break;
case PATCH:
if (uaClient.userAgent != null && uaClient.userAgent.patch != null) {
uaDetails.put("patch", uaClient.userAgent.patch);
}
break;
case BUILD:
if (uaClient.userAgent != null && uaClient.userAgent.build != null) {
uaDetails.put("build", uaClient.userAgent.build);
}
break;
case OS:
if (uaClient.operatingSystem != null) {
uaDetails.put("os", buildFullOSName(uaClient.operatingSystem));
} else {
uaDetails.put("os", "Other");
}

break;
case OS_NAME:
if (uaClient.operatingSystem != null && uaClient.operatingSystem.name != null) {
uaDetails.put("os_name", uaClient.operatingSystem.name);
} else {
uaDetails.put("os_name", "Other");
}
break;
case OS_MAJOR:
if (uaClient.operatingSystem != null && uaClient.operatingSystem.major != null) {
uaDetails.put("os_major", uaClient.operatingSystem.major);
}
break;
case OS_MINOR:
if (uaClient.operatingSystem != null && uaClient.operatingSystem.minor != null) {
uaDetails.put("os_minor", uaClient.operatingSystem.minor);
}
break;
case DEVICE:
if (uaClient.device != null && uaClient.device.name != null) {
uaDetails.put("device", uaClient.device.name);
} else {
uaDetails.put("device", "Other");
}
break;
}
}
}

Expand Down Expand Up @@ -199,8 +266,14 @@ UserAgentParser getUaParser() {
return parser;
}

public boolean isUseECS() {
return useECS;
}

public static final class Factory implements Processor.Factory {

private static final DeprecationLogger deprecationLogger = new DeprecationLogger(LogManager.getLogger(UserAgentProcessor.class));

private final Map<String, UserAgentParser> userAgentParsers;

public Factory(Map<String, UserAgentParser> userAgentParsers) {
Expand All @@ -215,6 +288,7 @@ public UserAgentProcessor create(Map<String, Processor.Factory> factories, Strin
String regexFilename = readStringProperty(TYPE, processorTag, config, "regex_file", IngestUserAgentPlugin.DEFAULT_PARSER_NAME);
List<String> propertyNames = readOptionalList(TYPE, processorTag, config, "properties");
boolean ignoreMissing = readBooleanProperty(TYPE, processorTag, config, "ignore_missing", false);
boolean useECS = readBooleanProperty(TYPE, processorTag, config, "ecs", false);

UserAgentParser parser = userAgentParsers.get(regexFilename);
if (parser == null) {
Expand All @@ -236,13 +310,31 @@ public UserAgentProcessor create(Map<String, Processor.Factory> factories, Strin
properties = EnumSet.allOf(Property.class);
}

return new UserAgentProcessor(processorTag, field, targetField, parser, properties, ignoreMissing);
if (useECS == false) {
deprecationLogger.deprecated("setting [ecs] to false for non-common schema " +
"format is deprecated and will be removed in 7.0, set to true to use the non-deprecated format");
}

return new UserAgentProcessor(processorTag, field, targetField, parser, properties, ignoreMissing, useECS);
}
}

enum Property {

NAME, MAJOR, MINOR, PATCH, OS, OS_NAME, OS_MAJOR, OS_MINOR, DEVICE, BUILD;
NAME,
// Deprecated in 6.7 (superceded by VERSION), to be removed in 7.0
@Deprecated MAJOR,
dakrone marked this conversation as resolved.
Show resolved Hide resolved
@Deprecated MINOR,
@Deprecated PATCH,
OS,
// Deprecated in 6.7 (superceded by just using OS), to be removed in 7.0
@Deprecated OS_NAME,
@Deprecated OS_MAJOR,
@Deprecated OS_MINOR,
DEVICE,
@Deprecated BUILD, // Same deprecated as OS_* above
ORIGINAL,
VERSION;

public static Property parseProperty(String propertyName) {
try {
Expand Down
8 changes: 7 additions & 1 deletion ...gent/src/test/java/org/elasticsearch/ingest/useragent/UserAgentProcessorFactoryTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ public void testBuildDefaults() throws Exception {

Map<String, Object> config = new HashMap<>();
config.put("field", "_field");
config.put("ecs", true);

String processorTag = randomAlphaOfLength(10);

Expand All @@ -90,6 +91,7 @@ public void testBuildDefaults() throws Exception {
assertThat(processor.getUaParser().getDevicePatterns().size(), greaterThan(0));
assertThat(processor.getProperties(), equalTo(EnumSet.allOf(UserAgentProcessor.Property.class)));
assertFalse(processor.isIgnoreMissing());
assertTrue(processor.isUseECS());
}

public void testBuildWithIgnoreMissing() throws Exception {
Expand All @@ -98,6 +100,7 @@ public void testBuildWithIgnoreMissing() throws Exception {
Map<String, Object> config = new HashMap<>();
config.put("field", "_field");
config.put("ignore_missing", true);
config.put("ecs", true);

String processorTag = randomAlphaOfLength(10);

Expand All @@ -118,6 +121,7 @@ public void testBuildTargetField() throws Exception {
Map<String, Object> config = new HashMap<>();
config.put("field", "_field");
config.put("target_field", "_target_field");
config.put("ecs", true);

UserAgentProcessor processor = factory.create(null, null, config);
assertThat(processor.getField(), equalTo("_field"));
Expand All @@ -130,6 +134,7 @@ public void testBuildRegexFile() throws Exception {
Map<String, Object> config = new HashMap<>();
config.put("field", "_field");
config.put("regex_file", regexWithoutDevicesFilename);
config.put("ecs", true);

UserAgentProcessor processor = factory.create(null, null, config);
assertThat(processor.getField(), equalTo("_field"));
Expand Down Expand Up @@ -164,6 +169,7 @@ public void testBuildFields() throws Exception {
Map<String, Object> config = new HashMap<>();
config.put("field", "_field");
config.put("properties", fieldNames);
config.put("ecs", true);

UserAgentProcessor processor = factory.create(null, null, config);
assertThat(processor.getField(), equalTo("_field"));
Expand All @@ -179,7 +185,7 @@ public void testInvalidProperty() throws Exception {

ElasticsearchParseException e = expectThrows(ElasticsearchParseException.class, () -> factory.create(null, null, config));
assertThat(e.getMessage(), equalTo("[properties] illegal property value [invalid]. valid values are [NAME, MAJOR, MINOR, "
+ "PATCH, OS, OS_NAME, OS_MAJOR, OS_MINOR, DEVICE, BUILD]"));
+ "PATCH, OS, OS_NAME, OS_MAJOR, OS_MINOR, DEVICE, BUILD, ORIGINAL, VERSION]"));
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there also an OS_VERSION here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the os map, the os property now returns the entire map (sub fields aren't selectable)


public void testInvalidPropertiesType() throws Exception {
Expand Down
Loading