Convert to Python 3 (elastic#674)

Notable changes:

* `date` field examples are now quoted in the schemas/*.yml files. This avoids them being interpreted and output in a different format.
* Changed the TravisCI distro to use, to get Python 3.6
* Virtualenv forces the use of Python3
* Update the Python version used in contributor guidelines
* avoid aliasing problems by renaming some `dict` function param names to `dct`
* Python 3 adjustments:
  * added parens for function calls such as `print` and `exit`
  * Updated all dependencies in scripts/requirements.txt to most recent versions
  * simplify a `sorted` lambda
  * use `yaml.safe_load`
  * Force sorting in a few places where Python 2's deterministic dict ordering was accidentally giving us sorted values
  * `assertEquals` => `assertEqual`

.travis.yml

@@ -5,10 +5,12 @@ language: go
 os:
 - linux
 
+dist: bionic
+
 go:
 - 1.13.x
 
-before_install:
+install:
 - make setup
 
 addons:

CONTRIBUTING.md

@@ -9,7 +9,7 @@ happen through Pull Requests submitted through Git.
 You need these tools to contribute to ECS:
 
 * [Git](https://git-scm.com/)
-* [Python 2.7](https://www.python.org/)
+* [Python 3.6+](https://www.python.org/)
 * [Go 1.13](https://golang.org/)
 
 ## Steps to contribute

Makefile

@@ -56,6 +56,7 @@ fmt: ve
 # Alias to generate everything.
 .PHONY: generate
 generate: legacy_use_cases codegen generator
+	$(PYTHON) --version
 
 # Run the new generator
 .PHONY: generator
@@ -107,7 +108,7 @@ test:
 .PHONY: ve
 ve: build/ve/bin/activate
 build/ve/bin/activate: scripts/requirements.txt
-	@test -d build/ve || virtualenv build/ve
+	@test -d build/ve || virtualenv -p python3 build/ve
 	@build/ve/bin/pip install -Ur scripts/requirements.txt
 	@touch build/ve/bin/activate
 

docs/field-details.asciidoc

@@ -1180,7 +1180,7 @@ In case the two timestamps are identical, @timestamp should be used.
 
 type: date
 
-example: `2016-05-23 08:05:34.857000`
+example: `2016-05-23T08:05:34.857Z`
 
 | core
 
@@ -1256,7 +1256,7 @@ In normal conditions, assuming no tampering, the timestamps should chronological
 
 type: date
 
-example: `2016-05-23 08:05:35.101000`
+example: `2016-05-23T08:05:35.101Z`
 
 | core
 

generated/beats/fields.ecs.yml

@@ -981,7 +981,7 @@
         your agent''s or pipeline''s ability to keep up with your event source.
 
         In case the two timestamps are identical, @timestamp should be used.'
-      example: 2016-05-23 08:05:34.857000
+      example: '2016-05-23T08:05:34.857Z'
     - name: dataset
       level: core
       type: keyword
@@ -1035,7 +1035,7 @@
 
         In normal conditions, assuming no tampering, the timestamps should chronologically
         look like this: `@timestamp` < `event.created` < `event.ingested`.'
-      example: 2016-05-23 08:05:35.101000
+      example: '2016-05-23T08:05:35.101Z'
       default_field: false
     - name: kind
       level: core

generated/csv/fields.csv

@@ -118,13 +118,13 @@ ECS_Version,Indexed,Field_Set,Field,Type,Level,Example,Description
 1.5.0-dev,true,event,event.action,keyword,core,user-password-change,The action captured by the event.
 1.5.0-dev,true,event,event.category,keyword,core,authentication,Event category. The second categorization field in the hierarchy.
 1.5.0-dev,true,event,event.code,keyword,extended,4648,Identification code for this event.
-1.5.0-dev,true,event,event.created,date,core,2016-05-23 08:05:34.857000,Time when the event was first read by an agent or by your pipeline.
+1.5.0-dev,true,event,event.created,date,core,2016-05-23T08:05:34.857Z,Time when the event was first read by an agent or by your pipeline.
 1.5.0-dev,true,event,event.dataset,keyword,core,apache.access,Name of the dataset.
 1.5.0-dev,true,event,event.duration,long,core,,Duration of the event in nanoseconds.
 1.5.0-dev,true,event,event.end,date,extended,,event.end contains the date when the event ended or when the activity was last observed.
 1.5.0-dev,true,event,event.hash,keyword,extended,123456789012345678901234567890ABCD,Hash (perhaps logstash fingerprint) of raw field to be able to demonstrate log integrity.
 1.5.0-dev,true,event,event.id,keyword,core,8a4f500d,Unique ID to describe the event.
-1.5.0-dev,true,event,event.ingested,date,core,2016-05-23 08:05:35.101000,Timestamp when an event arrived in the central data store.
+1.5.0-dev,true,event,event.ingested,date,core,2016-05-23T08:05:35.101Z,Timestamp when an event arrived in the central data store.
 1.5.0-dev,true,event,event.kind,keyword,core,alert,The kind of the event. The highest categorization field in the hierarchy.
 1.5.0-dev,true,event,event.module,keyword,core,apache,Name of the module this data is coming from.
 1.5.0-dev,false,event,event.original,keyword,core,Sep 19 08:26:10 host CEF:0|Security| threatmanager|1.0|100| worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2spt=1232,Raw text message of entire event.

generated/ecs/ecs_flat.yml

@@ -1543,7 +1543,7 @@ event.created:
     agent''s or pipeline''s ability to keep up with your event source.
 
     In case the two timestamps are identical, @timestamp should be used.'
-  example: 2016-05-23 08:05:34.857000
+  example: '2016-05-23T08:05:34.857Z'
   flat_name: event.created
   level: core
   name: created
@@ -1628,7 +1628,7 @@ event.ingested:
 
     In normal conditions, assuming no tampering, the timestamps should chronologically
     look like this: `@timestamp` < `event.created` < `event.ingested`.'
-  example: 2016-05-23 08:05:35.101000
+  example: '2016-05-23T08:05:35.101Z'
   flat_name: event.ingested
   level: core
   name: ingested

generated/ecs/ecs_nested.yml

@@ -615,8 +615,8 @@ client:
   group: 2
   name: client
   nestings:
-  - geo
   - as
+  - geo
   - user
   prefix: client.
   short: Fields about the client side of a network connection, used with server.
@@ -1183,8 +1183,8 @@ destination:
   group: 2
   name: destination
   nestings:
-  - geo
   - as
+  - geo
   - user
   prefix: destination.
   short: Fields about the destination side of a network connection, used with source.
@@ -1759,7 +1759,7 @@ event:
         your agent''s or pipeline''s ability to keep up with your event source.
 
         In case the two timestamps are identical, @timestamp should be used.'
-      example: 2016-05-23 08:05:34.857000
+      example: '2016-05-23T08:05:34.857Z'
       flat_name: event.created
       level: core
       name: created
@@ -1845,7 +1845,7 @@ event:
 
         In normal conditions, assuming no tampering, the timestamps should chronologically
         look like this: `@timestamp` < `event.created` < `event.ingested`.'
-      example: 2016-05-23 08:05:35.101000
+      example: '2016-05-23T08:05:35.101Z'
       flat_name: event.ingested
       level: core
       name: ingested
@@ -5335,8 +5335,8 @@ server:
   group: 2
   name: server
   nestings:
-  - geo
   - as
+  - geo
   - user
   prefix: server.
   short: Fields about the server side of a network connection, used with client.
@@ -5861,8 +5861,8 @@ source:
   group: 2
   name: source
   nestings:
-  - geo
   - as
+  - geo
   - user
   prefix: source.
   short: Fields about the source side of a network connection, used with destination.