Skip to content

Commit

Permalink
Add support for additional fields from V2 ALB logs (elastic#304)
Browse files Browse the repository at this point in the history
  • Loading branch information
kaiyan-sheng authored Oct 12, 2020
1 parent 596abdb commit cae646a
Show file tree
Hide file tree
Showing 10 changed files with 127 additions and 17 deletions.
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ go 1.12

require (
github.com/blang/semver v3.5.1+incompatible
github.com/elastic/elastic-package v0.0.0-20201001110805-0bb695cf2b70
github.com/elastic/elastic-package v0.0.0-20201012164813-861bb9387609
github.com/elastic/package-registry v0.12.0
github.com/magefile/mage v1.10.0
github.com/pkg/errors v0.9.1
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,8 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/elastic/elastic-package v0.0.0-20201001110805-0bb695cf2b70 h1:GV6BO1olp6KHgWnL2KOElxTSXxi5zfaiKcYAqZPZmJE=
github.com/elastic/elastic-package v0.0.0-20201001110805-0bb695cf2b70/go.mod h1:u7Hvc2PyfZBOfidOA5JuC4HOeBd7Ms4Ox1fQ+Wa/CRQ=
github.com/elastic/elastic-package v0.0.0-20201012164813-861bb9387609 h1:/qKEFsMwebx9USAUSl6frxWKOWEPYzLYVc1zMmy+UyE=
github.com/elastic/elastic-package v0.0.0-20201012164813-861bb9387609/go.mod h1:u7Hvc2PyfZBOfidOA5JuC4HOeBd7Ms4Ox1fQ+Wa/CRQ=
github.com/elastic/go-elasticsearch/v7 v7.9.0 h1:UEau+a1MiiE/F+UrDj60kqIHFWdzU1M2y/YtBU2NC2M=
github.com/elastic/go-elasticsearch/v7 v7.9.0/go.mod h1:OJ4wdbtDNk5g503kvlHLyErCgQwwzmDtaFC4XyOxXA4=
github.com/elastic/go-ucfg v0.8.4-0.20200415140258-1232bd4774a6 h1:Ehbr7du4rSSEypR8zePr0XRbMhO4PJgcHC9f8fDbgAg=
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
http 2018-07-02T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 10.0.0.1:80 0.000 0.001 0.000 200 200 34 366 "GET http://www.example.com:80/ HTTP/1.1" "curl/7.46.0" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 "Root=1-58337262-36d228ad5d99923122bbe354" "-" "-" 0 2018-07-02T22:22:48.364000Z "forward,redirect" "-" "-" "10.0.0.1:80" "200" "-" "-"
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
{
"expected": [
{
"cloud": {
"provider": "aws"
},
"tracing": {
"trace": {
"id": "Root=1-58337262-36d228ad5d99923122bbe354"
}
},
"@timestamp": "2018-07-02T22:23:00.186Z",
"http": {
"request": {
"method": "get",
"body": {
"bytes": 34
},
"referrer": "http://www.example.com:80/"
},
"version": "1.1",
"response": {
"body": {
"bytes": 366
},
"status_code": 200
}
},
"source": {
"port": "2817",
"ip": "192.168.131.39"
},
"aws": {
"elb": {
"trace_id": "Root=1-58337262-36d228ad5d99923122bbe354",
"matched_rule_priority": "0",
"type": "http",
"request_processing_time": {
"sec": 0.0
},
"response_processing_time": {
"sec": 0.0
},
"target_port": [
"10.0.0.1:80"
],
"protocol": "http",
"target_status_code": [
"200"
],
"name": "app/my-loadbalancer/50dc6c495c0c9188",
"backend": {
"port": "80",
"http": {
"response": {
"status_code": 200
}
},
"ip": "10.0.0.1"
},
"target_group": {
"arn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067"
},
"backend_processing_time": {
"sec": 0.001
},
"action_executed": [
"forward",
"redirect"
]
}
},
"event": {
"start": "2018-07-02T22:22:48.364000Z",
"end": "2018-07-02T22:23:00.186Z",
"category": "web",
"kind": "event",
"outcome": "success"
},
"user_agent": {
"original": "curl/7.46.0"
}
}
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ processors:
%{TIMESTAMP_ISO8601:event.start}
\"(?:-|%{DATA:_tmp.actions_executed})\"
\"(?:-|%{DATA:aws.elb.redirect_url})\"
\"(?:-|%{DATA:aws.elb.error.reason})\"
\"(?:-|%{DATA:aws.elb.error.reason})\"( \"(?:-|%{DATA:_tmp.target_port})\")?( \"(?:-|%{DATA:_tmp.target_status_code})\")?( \"(?:-|%{DATA:aws.elb.classification})\")?( \"(?:-|%{DATA:aws.elb.classification_reason})\")?
# TCP from Network Load Balancers (v2 Load Balancers)
- >-
Expand Down Expand Up @@ -143,6 +143,18 @@ processors:
separator: ','
ignore_missing: true

- split:
field: '_tmp.target_port'
target_field: 'aws.elb.target_port'
separator: ' '
ignore_missing: true

- split:
field: '_tmp.target_status_code'
target_field: 'aws.elb.target_status_code'
separator: ' '
ignore_missing: true

- date:
field: '_tmp.timestamp'
formats:
Expand Down
20 changes: 20 additions & 0 deletions packages/aws/data_stream/elb_logs/fields/fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,26 @@
type: keyword
description: |
The error reason if the executed action failed.
- name: target_port
type: keyword
description: >
List of IP addresses and ports for the targets that processed this request.
- name: target_status_code
type: keyword
description: >
List of status codes from the responses of the targets.
- name: classification
type: keyword
description: >
The classification for desync mitigation.
- name: classification_reason
type: keyword
description: >
The classification reason code.
- name: destination.domain
type: keyword
description: Destination domain.
Expand Down
4 changes: 4 additions & 0 deletions packages/aws/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,8 @@ For network load balancer, please follow [enable access log for network load bal
| aws.elb.backend_processing_time.sec | The total time in seconds since the connection is sent to the backend till the backend starts responding. | float |
| aws.elb.chosen_cert.arn | The ARN of the chosen certificate presented to the client in TLS/SSL connections. | keyword |
| aws.elb.chosen_cert.serial | The serial number of the chosen certificate presented to the client in TLS/SSL connections. | keyword |
| aws.elb.classification | The classification for desync mitigation. | keyword |
| aws.elb.classification_reason | The classification reason code. | keyword |
| aws.elb.connection_time.ms | The total time of the connection in milliseconds, since it is opened till it is closed. | long |
| aws.elb.error.reason | The error reason if the executed action failed. | keyword |
| aws.elb.incoming_tls_alert | The integer value of TLS alerts received by the load balancer from the client, if present. | keyword |
Expand All @@ -248,6 +250,8 @@ For network load balancer, please follow [enable access log for network load bal
| aws.elb.ssl_cipher | The SSL cipher used in TLS/SSL connections. | keyword |
| aws.elb.ssl_protocol | The SSL protocol used in TLS/SSL connections. | keyword |
| aws.elb.target_group.arn | The ARN of the target group handling the request. | keyword |
| aws.elb.target_port | List of IP addresses and ports for the targets that processed this request. | keyword |
| aws.elb.target_status_code | List of status codes from the responses of the targets. | keyword |
| aws.elb.tls_handshake_time.ms | The total time for the TLS handshake to complete in milliseconds once the connection has been established. | long |
| aws.elb.tls_named_group | The TLS named group. | keyword |
| aws.elb.trace_id | The contents of the `X-Amzn-Trace-Id` header. | keyword |
Expand Down
2 changes: 1 addition & 1 deletion packages/aws/manifest.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 1.0.0
name: aws
title: AWS
version: 0.3.8
version: 0.3.9
license: basic
description: AWS Integration
type: integration
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,12 +95,6 @@
"error": {
"message": "field [@timestamp] doesn't exist"
}
},
{
"message": "",
"error": {
"message": "Provided Grok expressions do not match field value: []"
}
}
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -1128,12 +1128,6 @@
"url": {
"original": "/v2/some"
}
},
{
"message": "",
"error": {
"message": "Provided Grok expressions do not match field value: []"
}
}
]
}

0 comments on commit cae646a

Please sign in to comment.