[1.x] Uniformity across domain name breakdown fields (#981) (#994)

Co-authored-by: Mathieu Martin <[email protected]>

CHANGELOG.next.md

@@ -26,6 +26,7 @@ Thanks, you're awesome :-) -->
 
 * Expanded field set definitions for `source.*` and `destination.*`. #967
 * Provided better guidance for mapping network events. #969
+* Added the field `.subdomain` under `client`, `destination`, `server`, `source` and `url`, to match its presence at `dns.question.subdomain`. #981
 
 #### Deprecated
 

docs/field-details.asciidoc

@@ -407,6 +407,21 @@ example: `example.com`
 
 // ===============================================================
 
+| client.subdomain
+| The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain.  In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain.
+
+For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.
+
+type: keyword
+
+
+
+example: `east`
+
+| extended
+
+// ===============================================================
+
 | client.top_level_domain
 | The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com".
 
@@ -967,6 +982,21 @@ example: `example.com`
 
 // ===============================================================
 
+| destination.subdomain
+| The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain.  In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain.
+
+For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.
+
+type: keyword
+
+
+
+example: `east`
+
+| extended
+
+// ===============================================================
+
 | destination.top_level_domain
 | The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com".
 
@@ -5058,6 +5088,21 @@ example: `example.com`
 
 // ===============================================================
 
+| server.subdomain
+| The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain.  In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain.
+
+For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.
+
+type: keyword
+
+
+
+example: `east`
+
+| extended
+
+// ===============================================================
+
 | server.top_level_domain
 | The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com".
 
@@ -5397,6 +5442,21 @@ example: `example.com`
 
 // ===============================================================
 
+| source.subdomain
+| The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain.  In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain.
+
+For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.
+
+type: keyword
+
+
+
+example: `east`
+
+| extended
+
+// ===============================================================
+
 | source.top_level_domain
 | The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com".
 
@@ -6321,6 +6381,21 @@ example: `https`
 
 // ===============================================================
 
+| url.subdomain
+| The subdomain portion of a fully qualified domain name includes all of the names except the host name under the registered_domain.  In a partially qualified domain, or if the the qualification level of the full name cannot be determined, subdomain contains all of the names below the registered domain.
+
+For example the subdomain portion of "www.east.mydomain.co.uk" is "east". If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com", the subdomain field should contain "sub2.sub1", with no trailing period.
+
+type: keyword
+
+
+
+example: `east`
+
+| extended
+
+// ===============================================================
+
 | url.top_level_domain
 | The effective top level domain (eTLD), also known as the domain suffix, is the last part of the domain name. For example, the top level domain for example.com is "com".
 

generated/beats/fields.ecs.yml

@@ -302,6 +302,20 @@
         list (http://publicsuffix.org). Trying to approximate this by simply taking
         the last two labels will not work well for TLDs such as "co.uk".'
       example: example.com
+    - name: subdomain
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: 'The subdomain portion of a fully qualified domain name includes
+        all of the names except the host name under the registered_domain.  In a partially
+        qualified domain, or if the the qualification level of the full name cannot
+        be determined, subdomain contains all of the names below the registered domain.
+
+        For example the subdomain portion of "www.east.mydomain.co.uk" is "east".
+        If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com",
+        the subdomain field should contain "sub2.sub1", with no trailing period.'
+      example: east
+      default_field: false
     - name: top_level_domain
       level: extended
       type: keyword
@@ -709,6 +723,20 @@
         list (http://publicsuffix.org). Trying to approximate this by simply taking
         the last two labels will not work well for TLDs such as "co.uk".'
       example: example.com
+    - name: subdomain
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: 'The subdomain portion of a fully qualified domain name includes
+        all of the names except the host name under the registered_domain.  In a partially
+        qualified domain, or if the the qualification level of the full name cannot
+        be determined, subdomain contains all of the names below the registered domain.
+
+        For example the subdomain portion of "www.east.mydomain.co.uk" is "east".
+        If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com",
+        the subdomain field should contain "sub2.sub1", with no trailing period.'
+      example: east
+      default_field: false
     - name: top_level_domain
       level: extended
       type: keyword
@@ -4105,6 +4133,20 @@
         list (http://publicsuffix.org). Trying to approximate this by simply taking
         the last two labels will not work well for TLDs such as "co.uk".'
       example: example.com
+    - name: subdomain
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: 'The subdomain portion of a fully qualified domain name includes
+        all of the names except the host name under the registered_domain.  In a partially
+        qualified domain, or if the the qualification level of the full name cannot
+        be determined, subdomain contains all of the names below the registered domain.
+
+        For example the subdomain portion of "www.east.mydomain.co.uk" is "east".
+        If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com",
+        the subdomain field should contain "sub2.sub1", with no trailing period.'
+      example: east
+      default_field: false
     - name: top_level_domain
       level: extended
       type: keyword
@@ -4427,6 +4469,20 @@
         list (http://publicsuffix.org). Trying to approximate this by simply taking
         the last two labels will not work well for TLDs such as "co.uk".'
       example: example.com
+    - name: subdomain
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: 'The subdomain portion of a fully qualified domain name includes
+        all of the names except the host name under the registered_domain.  In a partially
+        qualified domain, or if the the qualification level of the full name cannot
+        be determined, subdomain contains all of the names below the registered domain.
+
+        For example the subdomain portion of "www.east.mydomain.co.uk" is "east".
+        If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com",
+        the subdomain field should contain "sub2.sub1", with no trailing period.'
+      example: east
+      default_field: false
     - name: top_level_domain
       level: extended
       type: keyword
@@ -5337,6 +5393,20 @@
 
         Note: The `:` is not part of the scheme.'
       example: https
+    - name: subdomain
+      level: extended
+      type: keyword
+      ignore_above: 1024
+      description: 'The subdomain portion of a fully qualified domain name includes
+        all of the names except the host name under the registered_domain.  In a partially
+        qualified domain, or if the the qualification level of the full name cannot
+        be determined, subdomain contains all of the names below the registered domain.
+
+        For example the subdomain portion of "www.east.mydomain.co.uk" is "east".
+        If the domain has multiple levels of subdomain, such as "sub2.sub1.example.com",
+        the subdomain field should contain "sub2.sub1", with no trailing period.'
+      example: east
+      default_field: false
     - name: top_level_domain
       level: extended
       type: keyword

generated/csv/fields.csv

@@ -30,6 +30,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
 1.7.0-dev,true,client,client.packets,long,core,,12,Packets sent from the client to the server.
 1.7.0-dev,true,client,client.port,long,core,,,Port of the client.
 1.7.0-dev,true,client,client.registered_domain,keyword,extended,,example.com,"The highest registered client domain, stripped of the subdomain."
+1.7.0-dev,true,client,client.subdomain,keyword,extended,,east,The subdomain of the domain.
 1.7.0-dev,true,client,client.top_level_domain,keyword,extended,,co.uk,"The effective top level domain (com, org, net, co.uk)."
 1.7.0-dev,true,client,client.user.domain,keyword,extended,,,Name of the directory the user is a member of.
 1.7.0-dev,true,client,client.user.email,keyword,extended,,,User email address.
@@ -80,6 +81,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
 1.7.0-dev,true,destination,destination.packets,long,core,,12,Packets sent from the destination to the source.
 1.7.0-dev,true,destination,destination.port,long,core,,,Port of the destination.
 1.7.0-dev,true,destination,destination.registered_domain,keyword,extended,,example.com,"The highest registered destination domain, stripped of the subdomain."
+1.7.0-dev,true,destination,destination.subdomain,keyword,extended,,east,The subdomain of the domain.
 1.7.0-dev,true,destination,destination.top_level_domain,keyword,extended,,co.uk,"The effective top level domain (com, org, net, co.uk)."
 1.7.0-dev,true,destination,destination.user.domain,keyword,extended,,,Name of the directory the user is a member of.
 1.7.0-dev,true,destination,destination.user.email,keyword,extended,,,User email address.
@@ -478,6 +480,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
 1.7.0-dev,true,server,server.packets,long,core,,12,Packets sent from the server to the client.
 1.7.0-dev,true,server,server.port,long,core,,,Port of the server.
 1.7.0-dev,true,server,server.registered_domain,keyword,extended,,example.com,"The highest registered server domain, stripped of the subdomain."
+1.7.0-dev,true,server,server.subdomain,keyword,extended,,east,The subdomain of the domain.
 1.7.0-dev,true,server,server.top_level_domain,keyword,extended,,co.uk,"The effective top level domain (com, org, net, co.uk)."
 1.7.0-dev,true,server,server.user.domain,keyword,extended,,,Name of the directory the user is a member of.
 1.7.0-dev,true,server,server.user.email,keyword,extended,,,User email address.
@@ -519,6 +522,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
 1.7.0-dev,true,source,source.packets,long,core,,12,Packets sent from the source to the destination.
 1.7.0-dev,true,source,source.port,long,core,,,Port of the source.
 1.7.0-dev,true,source,source.registered_domain,keyword,extended,,example.com,"The highest registered source domain, stripped of the subdomain."
+1.7.0-dev,true,source,source.subdomain,keyword,extended,,east,The subdomain of the domain.
 1.7.0-dev,true,source,source.top_level_domain,keyword,extended,,co.uk,"The effective top level domain (com, org, net, co.uk)."
 1.7.0-dev,true,source,source.user.domain,keyword,extended,,,Name of the directory the user is a member of.
 1.7.0-dev,true,source,source.user.email,keyword,extended,,,User email address.
@@ -637,6 +641,7 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Normalization,Example,Description
 1.7.0-dev,true,url,url.query,keyword,extended,,,Query string of the request.
 1.7.0-dev,true,url,url.registered_domain,keyword,extended,,example.com,"The highest registered url domain, stripped of the subdomain."
 1.7.0-dev,true,url,url.scheme,keyword,extended,,https,Scheme of the url.
+1.7.0-dev,true,url,url.subdomain,keyword,extended,,east,The subdomain of the domain.
 1.7.0-dev,true,url,url.top_level_domain,keyword,extended,,co.uk,"The effective top level domain (com, org, net, co.uk)."
 1.7.0-dev,true,url,url.username,keyword,extended,,,Username of the request.
 1.7.0-dev,true,user,user.domain,keyword,extended,,,Name of the directory the user is a member of.