-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add readme to geo containment alert covering test alert setup #89625
Merged
kindsun
merged 24 commits into
elastic:master
from
kindsun:add-geo-containment-alert-readme
Feb 4, 2021
Merged
Changes from all commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
0c12f2d
Add readme for operating test threshold alert scenario
4004228
Update mappings to cover latest available fields
45bb7a5
More details, updates
369a286
Update index params
5a607d7
Add more detail around map creation and add supporting images
fb70faa
Review feedback
13a5f3c
Merge remote-tracking branch 'upstream/master' into add-geothreshold-…
4be4d74
Merge remote-tracking branch 'upstream/master' into add-geothreshold-…
dcc2daa
Update config flag path
9786004
Pull last bullet out of code block
7eaf306
Merge remote-tracking branch 'upstream/master' into add-geothreshold-…
7d6e023
Merge remote-tracking branch 'upstream/master' into add-geothreshold-…
f488f53
Update config setting to latest in readme
fcff7a9
Merge remote-tracking branch 'upstream/master' into add-geo-containme…
d68bf03
Move readme and revise for containment alert type
d50886b
Merge remote-tracking branch 'upstream/master' into add-geo-containme…
640f615
Add containment view screenshot
e0282c4
Review feedback. Add note indicating 2 second interval is faster than…
f9d0374
Review feedback. Revise to output alerts to server logs. Update langu…
d0fb850
Merge remote-tracking branch 'upstream/master' into add-geo-containme…
61c16a6
Review feedback. Add more explicit map/layer instructions
3ac8c00
Merge remote-tracking branch 'upstream/master' into add-geo-containme…
18c425a
Minor corrections/additions
fc534f3
Merge branch 'master' into add-geo-containment-alert-readme
kibanamachine File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
123 changes: 123 additions & 0 deletions
123
x-pack/plugins/stack_alerts/public/alert_types/geo_containment/readme.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,123 @@ | ||
## Instructions for loading & observing data | ||
|
||
There are several steps required to set up geo containment alerts for testing in a way | ||
that allows you to view triggered alerts as they happen. These instructions outline | ||
how to load test data, but really these steps can be used to load any data for geo | ||
containment alerts so long as you have the following data: | ||
- An index containing a`geo_point` field and a `date` field. This data is presumed to | ||
be dynamic (updated). | ||
- An index containing `geo_shape` data, such as boundary data, bounding box data, etc. | ||
This data is presumed to be static (not updated). Shape data matching the query is | ||
harvested once when the alert is created and anytime after when alert is re-enabled | ||
after disablement | ||
The ability for containment alerts to monitor data requires there be somewhat "real time" | ||
data streaming in as indicated by the `date` field. | ||
|
||
### 1. Set experimental flag to enable containment alerts | ||
- Your `kibana.yml` config file is located in the `config/` dir in the base of your kibana | ||
project. To edit it, open this file in your editor of choice, add the line described in | ||
the next step to the bottom of the file (or really anywhere) and save. For more details | ||
on different config modifications or on how to make production config modifications, | ||
see [the current docs](https://www.elastic.co/guide/en/kibana/current/settings.html) | ||
- Set the following configuration settings in your `config/kibana.yml`: | ||
`xpack.stack_alerts.enableGeoAlerting: true` | ||
|
||
### 2. Run ES/Kibana dev env with ssl enabled | ||
- In two terminals, run the normal commands to launch both elasticsearch and kibana but | ||
append `--ssl` to the end of each as an arg, i.e.: | ||
- `yarn es snapshot --ssl # Runs Elasticsearch` | ||
- `yarn start --ssl # Runs Kibana` | ||
|
||
### 3. Get an MTA data api key | ||
- You'll need to obtain an NYC MTA api key, you can request this | ||
key [here](https://docs.google.com/forms/d/e/1FAIpQLSfGUZA6h4eHd2-ImaK5Q_I5Gb7C3UEP5vYDALyGd7r3h08YKg/viewform?hl=en&formkey=dG9kcGIxRFpSS0NhQWM4UjA0V0VkNGc6MQ#gid=0) | ||
|
||
### 4. Get trackable point data (MTA bus data) into elasticsearch | ||
- You'll be using the script: `https://github.com/thomasneirynck/mtatracks` to harvest | ||
live bus data to populate the system. Clone the repo and follow the instructions in | ||
the readme to set up. | ||
- Using the MTA key you obtained in the previous step, the final command to run | ||
in a local terminal should look something like the following. This script loads large | ||
quantities of data the frequency listed below (20000ms = 20s) or higher: | ||
`node ./load_tracks.js -a <YOUR_API_KEY> -f 20000` | ||
|
||
### 5. Open required Kibana tabs | ||
There are 3 separate tabs you'll need for a combination of loading and viewing the | ||
data. Since you'll be jumping between them, it might be easiest to just open them | ||
upfront. Each is preceded by `https://localhost:5601/<your dev env prefix>/app/`: | ||
- Stack Management > Index Patterns: `management/kibana/indexPatterns` | ||
- Stack Management > Alerts & Actions: `management/insightsAndAlerting/triggersActions/alerts` | ||
- Maps: `maps` | ||
|
||
### 6 Create map to monitor alerts | ||
- Go to the Maps app and create a new map | ||
- Using GeoJSON Upload, upload the GeoJSON file located in the folder of the previously | ||
cloned `mta_tracks` repo: `nyc-neighborhoods.geo.json`. Accept all of the default | ||
settings and add the layer. | ||
- You may want to click your newly added layer and select "Fit to data" so you can see the | ||
boundaries you've added. | ||
_ When finished uploading and adding the layer, save the map using a name of your | ||
choice. | ||
- Keep the Maps tab open, you'll come back to this | ||
|
||
### 7. Create index pattern for generated tracks | ||
- Go to the index pattern tab to create a new index pattern. | ||
- Give it the index name `mtatracks*` | ||
- For `Time field` select `@timestamp` | ||
- Click `Create index pattern` | ||
- Leave this tab open, you'll come back to this | ||
|
||
### 8. Create containment alert | ||
- Go to the Alerts tab and click `Create Alert` > `Tracking containment` | ||
- Fill the side bar form top to bottom. This _should_ flow somewhat logically. In the top | ||
section, set both `Check every` and `Notify every` to `1 minute`. | ||
For `Notify`, leave | ||
on default selected option `Only on status change`, this will notify only on newly | ||
contained entities. | ||
**Please note that `2 seconds` is an unusually quick interval but done here for demo | ||
purposes. With real world data, setting an appropriate interval speed is highly dependent | ||
upon the quantity, update frequency and complexity of data handled.** | ||
- The default settings for `Select Entity` will mostly be correct. Select `mta_tracks*` | ||
as the index you'd like to track. Use the defaults populated under | ||
`Select entity` > `INDEX`, update `Select entity` > `BY` to `vehicle_ref`. | ||
- For `Select boundary` > `INDEX`, select `nyc-neighborhoods` and all populated defaults. | ||
- Under `Actions`, create an `Server log` action, then create a `Connector` which you can simply name | ||
`Log test`. | ||
- For `Run when`, the default `Tracking containment met` will work here. This will track | ||
only points that are newly contained in the boundaries. | ||
- Leave the log level at `Info` | ||
- For the message, use the following sample message or one of your own: | ||
``` | ||
Entity: {{context.entityId}} with document ID: {{context.entityDocumentId}} has been recorded at location: {{context.entityLocation}} in boundary: {{context.containingBoundaryName}}({{context.containingBoundaryId}}) at {{context.entityDateTime}}. This was detected by the alerting framework at: {{context.detectionDateTime}}. | ||
``` | ||
- At the bottom right, click `Save`. Your alert should now be created! | ||
- You should now be able to see alerts generated in your Kibana console log. | ||
|
||
### 9. Visually confirm your alerts with Maps | ||
- Creating layers | ||
- Using the source data below, you can create the following layers: | ||
- Boundary data (`nyc-neighborhoods`) | ||
- Boundary layer | ||
- Original tracks data (`mtatracks*`) | ||
- Last known location | ||
- Geo-line track | ||
- Boundary layer | ||
- This layer should already be added from when you uploaded the GeoJSON | ||
file earlier. If it's not already added, it can be added by selecting `Documents` | ||
> `Index patterns` > `nyc-neighborhoods` then accept the defaults and add the layer. | ||
- Vehicle tracks | ||
- Add `Tracks` > `Index patterns` > `mtatracks*`, accept the defaults selected and set `Entity` > `entity_id`. Add the layer and style appropriately. | ||
- Last known location | ||
- Add `Documents` > `Index patterns` > `mtatracks*` and select `Show top hits per entity` | ||
- For `Entity` select `entity_id` and add the layer. | ||
- The only required setting on the following screen is to set `Sorting` to sort on `@timestamp` | ||
- Update time scope of data | ||
- Changing the refresh rate `Refresh every`: `4 seconds` keeps the layers updated and in particular | ||
shows the latest values obtained in the `Top hits` layer | ||
- The time picker should already be set to the default `15 minutes`, this is a good default but | ||
can be adjusted up or down to see more or less data respectively | ||
- General tips | ||
- Style layers with contrasting colors to clearly see each | ||
- Consider using icons for the `Top hits` vehicle movement layer | ||
- Consider adding tooltips to layers to better understand the data in your layers. | ||
- Save your Map anytime you've made any layer adjustments |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
iso. having this feel synthetic wrt nyc boundaries: what about a geojson of something more "real world". Like for example, Times Square or perimeter around UN, and just check if vehicles enter/leave those areas.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was actually the only one I held off on since it's more related to the file contained in https://github.com/thomasneirynck/mtatracks. What we might want to do is throw up a separate PR against that repo, get something together that we agree on, then do a small PR to update the reference in this one. If we're going to craft something a little nicer, it would also be worth rethinking the tracks so that they hit the major NYC sites in the GeoJSON file
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI: thomasneirynck/mtatracks#2
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thx, merged!