Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add readme to geo containment alert covering test alert setup #89625

Merged
merged 24 commits into from
Feb 4, 2021
Merged
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
0c12f2d
Add readme for operating test threshold alert scenario
Nov 6, 2020
4004228
Update mappings to cover latest available fields
Nov 6, 2020
45bb7a5
More details, updates
Nov 6, 2020
369a286
Update index params
Nov 6, 2020
5a607d7
Add more detail around map creation and add supporting images
Nov 9, 2020
fb70faa
Review feedback
Nov 11, 2020
13a5f3c
Merge remote-tracking branch 'upstream/master' into add-geothreshold-…
Nov 11, 2020
4be4d74
Merge remote-tracking branch 'upstream/master' into add-geothreshold-…
Nov 16, 2020
dcc2daa
Update config flag path
Nov 16, 2020
9786004
Pull last bullet out of code block
Nov 23, 2020
7eaf306
Merge remote-tracking branch 'upstream/master' into add-geothreshold-…
Nov 23, 2020
7d6e023
Merge remote-tracking branch 'upstream/master' into add-geothreshold-…
Dec 21, 2020
f488f53
Update config setting to latest in readme
Dec 21, 2020
fcff7a9
Merge remote-tracking branch 'upstream/master' into add-geo-containme…
Jan 28, 2021
d68bf03
Move readme and revise for containment alert type
Jan 28, 2021
d50886b
Merge remote-tracking branch 'upstream/master' into add-geo-containme…
Feb 1, 2021
640f615
Add containment view screenshot
Feb 1, 2021
e0282c4
Review feedback. Add note indicating 2 second interval is faster than…
Feb 2, 2021
f9d0374
Review feedback. Revise to output alerts to server logs. Update langu…
Feb 3, 2021
d0fb850
Merge remote-tracking branch 'upstream/master' into add-geo-containme…
Feb 3, 2021
61c16a6
Review feedback. Add more explicit map/layer instructions
Feb 3, 2021
3ac8c00
Merge remote-tracking branch 'upstream/master' into add-geo-containme…
Feb 3, 2021
18c425a
Minor corrections/additions
Feb 3, 2021
fc534f3
Merge branch 'master' into add-geo-containment-alert-readme
kibanamachine Feb 4, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
## Instructions for loading & observing data

There are several steps required to set up geo containment alerts for testing in a way
that allows you to view triggered alerts as they happen. These instructions outline
how to load test data, but really these steps can be used to load any data for geo
containment alerts so long as you have the following data:
- An index containing a`geo_point` field and a `date` field. This data is presumed to
be dynamic (updated).
- An index containing `geo_shape` data, such as boundary data, bounding box data, etc.
This data is presumed to be static (not updated). Shape data matching the query is
harvested once when the alert is created and anytime after when alert is re-enabled
after disablement
The ability for containment alerts to monitor data requires there be somewhat "real time"
data streaming in as indicated by the `date` field.

### 1. Set experimental flag to enable containment alerts
- Your `kibana.yml` config file is located in the `config/` dir in the base of your kibana
project. To edit it, open this file in your editor of choice, add the line described in
the next step to the bottom of the file (or really anywhere) and save. For more details
on different config modifications or on how to make production config modifications,
see [the current docs](https://www.elastic.co/guide/en/kibana/current/settings.html)
- Set the following configuration settings in your `config/kibana.yml`:
`xpack.stack_alerts.enableGeoAlerting: true`

### 2. Run ES/Kibana dev env with ssl enabled
- In two terminals, run the normal commands to launch both elasticsearch and kibana but
append `--ssl` to the end of each as an arg, i.e.:
- `yarn es snapshot --ssl # Runs Elasticsearch`
- `yarn start --ssl # Runs Kibana`

### 3. Get an MTA data api key
- You'll need to obtain an NYC MTA api key, you can request this
key [here](https://docs.google.com/forms/d/e/1FAIpQLSfGUZA6h4eHd2-ImaK5Q_I5Gb7C3UEP5vYDALyGd7r3h08YKg/viewform?hl=en&formkey=dG9kcGIxRFpSS0NhQWM4UjA0V0VkNGc6MQ#gid=0)

### 4. Get trackable point data (MTA bus data) into elasticsearch
- You'll be using the script: `https://github.com/thomasneirynck/mtatracks` to harvest
live bus data to populate the system. Clone the repo and follow the instructions in
the readme to set up.
- Using the MTA key you obtained in the previous step, the final command to run
in a local terminal should look something like the following. This script loads large
quantities of data the frequency listed below (20000ms = 20s) or higher:
`node ./load_tracks.js -a <YOUR_API_KEY> -f 20000`

### 5. Open required Kibana tabs
There are 3 separate tabs you'll need for a combination of loading and viewing the
data. Since you'll be jumping between them, it might be easiest to just open them
upfront. Each is preceded by `https://localhost:5601/<your dev env prefix>/app/`:
- Stack Management > Index Patterns: `management/kibana/indexPatterns`
- Stack Management > Alerts & Actions: `management/insightsAndAlerting/triggersActions/alerts`
- Maps: `maps`

### 6 Create map to monitor alerts
- Go to the Maps app and create a new map
- Using GeoJSON Upload, upload the GeoJSON file located in the folder of the previously
cloned `mta_tracks` repo: `nyc-neighborhoods.geo.json`. Accept all of the default
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iso. having this feel synthetic wrt nyc boundaries: what about a geojson of something more "real world". Like for example, Times Square or perimeter around UN, and just check if vehicles enter/leave those areas.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was actually the only one I held off on since it's more related to the file contained in https://github.com/thomasneirynck/mtatracks. What we might want to do is throw up a separate PR against that repo, get something together that we agree on, then do a small PR to update the reference in this one. If we're going to craft something a little nicer, it would also be worth rethinking the tracks so that they hit the major NYC sites in the GeoJSON file

Copy link
Contributor Author

@kindsun kindsun Feb 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx, merged!

settings and add the layer.
- You may want to click your newly added layer and select "Fit to data" so you can see the
boundaries you've added.
_ When finished uploading and adding the layer, save the map using a name of your
choice.
- Keep the Maps tab open, you'll come back to this

### 7. Create index pattern for generated tracks
- Go to the index pattern tab to create a new index pattern.
- Give it the index name `mtatracks*`
- For `Time field` select `@timestamp`
- Click `Create index pattern`
- Leave this tab open, you'll come back to this

### 8. Create containment alert
- Go to the Alerts tab and click `Create Alert` > `Tracking containment`
- Fill the side bar form top to bottom. This _should_ flow somewhat logically. In the top
section, set both `Check every` and `Notify every` to `1 minute`.
For `Notify`, leave
on default selected option `Only on status change`, this will notify only on newly
contained entities.
**Please note that `2 seconds` is an unusually quick interval but done here for demo
purposes. With real world data, setting an appropriate interval speed is highly dependent
upon the quantity, update frequency and complexity of data handled.**
- The default settings for `Select Entity` will mostly be correct. Select `mta_tracks*`
as the index you'd like to track. Use the defaults populated under
`Select entity` > `INDEX`, update `Select entity` > `BY` to `vehicle_ref`.
- For `Select boundary` > `INDEX`, select `nyc-neighborhoods` and all populated defaults.
- Under `Actions`, create an `Server log` action, then create a `Connector` which you can simply name
`Log test`.
- For `Run when`, the default `Tracking containment met` will work here. This will track
only points that are newly contained in the boundaries.
- Leave the log level at `Info`
- For the message, use the following sample message or one of your own:
```
Entity: {{context.entityId}} with document ID: {{context.entityDocumentId}} has been recorded at location: {{context.entityLocation}} in boundary: {{context.containingBoundaryName}}({{context.containingBoundaryId}}) at {{context.entityDateTime}}. This was detected by the alerting framework at: {{context.detectionDateTime}}.
```
- At the bottom right, click `Save`. Your alert should now be created!
- You should now be able to see alerts generated in your Kibana console log.

### 9. Visually confirm your alerts with Maps
- Creating layers
- Using the source data below, you can create the following layers:
- Boundary data (`nyc-neighborhoods`)
- Boundary layer
- Original tracks data (`mtatracks*`)
- Last known location
- Geo-line track
- Boundary layer
- This layer should already be added from when you uploaded the GeoJSON
file earlier. If it's not already added, it can be added by selecting `Documents`
> `Index patterns` > `nyc-neighborhoods` then accept the defaults and add the layer.
- Vehicle tracks
- Add `Tracks` > `Index patterns` > `mtatracks*`, accept the defaults selected and set `Entity` > `entity_id`. Add the layer and style appropriately.
- Last known location
- Add `Documents` > `Index patterns` > `mtatracks*` and select `Show top hits per entity`
- For `Entity` select `entity_id` and add the layer.
- The only required setting on the following screen is to set `Sorting` to sort on `@timestamp`
- Update time scope of data
- Changing the refresh rate `Refresh every`: `4 seconds` keeps the layers updated and in particular
shows the latest values obtained in the `Top hits` layer
- The time picker should already be set to the default `15 minutes`, this is a good default but
can be adjusted up or down to see more or less data respectively
- General tips
- Style layers with contrasting colors to clearly see each
- Consider using icons for the `Top hits` vehicle movement layer
- Consider adding tooltips to layers to better understand the data in your layers.
- Save your Map anytime you've made any layer adjustments