Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Autonomous System (AS) fields #341

Merged
merged 1 commit into from
Jul 8, 2019

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Feb 23, 2019

This adds fields for Autonomous System (AS) information. It's common to enrich
events containing IP addresses with information about the AS controlling that
address.

It fields are in a reusable object that can be attached to source, destination, client,
and server. For example:

  • source.as.number
  • source.as.organization.name

Some future fields we might want in this object are (I'm considering them now just to make sure nothing we do now precludes future growth):

  • cidr - the IP range advertised by the AS (e.g. 216.90.108.0/24)
  • organization.country_code_iso - country code where the org is registered
  • registry - what registry the AS is with (e.g. ARIN)
  • registration_date - When the AS was first registered
  • last_update - When the AS registration was last updated.

Some resources to see what kind of data is available.

fields.yml Outdated
system number (ASN) uniquely identifies each network on the Internet.
example: 15169

- name: organization.name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think organization_name would be a better name, unless there will be more properties than just name under the organization property.

Copy link
Contributor

@praseodym praseodym Feb 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I missed the suggestion of organization.country_code. Maybe that information is better suited for the geo field group, though.

edit: and there’s already an organization field group so that #328 might actually solve this implicitly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that the source.geo field might be populated with the location of the IP whereas the details under source.autonomous_system.organization would be about the organization (like perhaps the country where the organization is headquartered.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could have something like source.autonomous_system.organization.geo, but maybe that's a bit over the top.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

geo is a “reusable object” so it could be used under organization. The current structure in this PR won’t preclude this from specified in the schema at a later time. This PR will get us the bare minimum fields for using the geoip processor with the AS database.

@dainperkins
Copy link
Contributor

couple of thoughts if its not too late:

  1. AS is a specific routing term, its commonly referenced when talking about e.g. ARIN lookups, but its also relevant for other routing protocols and is really only globally significant for public BGP (assuming no one started trading other protocols publicly while I wasn't looking)

  2. We might want to consider AS for network as well for router logs, route changes etc. (network.as?) or possibly splitting up the taxonomies to be more technically accurate (or put it all under network and allow that to be nested under source / dest etc)

I have to consider a bit more to relearn my way down the bgp/routing stack, but I can immediately see NPM desires to track routing changes by AS (particularly in dual homed BGP setups) and usign a single organization header to describe a single AS country code (as opposed to src/dst AS & related info)...

Copy link
Contributor

@webmat webmat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the links, @andrewkroh. I wasn't familiar with these datasets, and the links were extremely useful.

Looking at some of the possibilities of what could go under autonomous systems, the final field names will indeed get very long. I wonder if we should do like with geo, and contract the name to as? E.g. source.as.number, source.as.organization.*.

After looking at a few records on the ARIN website and looking at the RIR files linked from the team-cymru.com page, I agree we can start with .number and .organization.name for now. To continue with your example nested under source, here's what it could look like:

  • source.as.number
  • source.as.organization.name
    • This leaves room for other org metadata, such as organization.handle

I would defer discussion on the nesting of the country code for now. It's not in the default free Maxmind DB, so I suppose the need is not pressing yet.

For future growth, I think the current proposal is fine. We can discuss other fields when the need arises.

So to get this merged:

  • We need to discuss whether we shorten the name to as
  • The PR needs to be rebased on top of master
  • We should mark organization as reusable under as, rather than redefining the field explicitly there.

@andrewkroh
Copy link
Member Author

I updated the PR to start address @webmat's actions:

  • We need to discuss whether we shorten the name to as
  • The PR needs to be rebased on top of master
  • We should mark organization as reusable under as, rather than redefining the field explicitly there.

@andrewkroh
Copy link
Member Author

andrewkroh commented Jul 2, 2019

@webmat What's the right way to accomplish the last bullet?

I tried this after remove the organization field from as.yml, but it breaks the generator.

- name: organization
  ...
  reusable:
    top_level: true
    expected:
      - client.as
      - destination.as
      - server.as
      - source.as
Running generator. ECS version 1.1.0-dev
Traceback (most recent call last):
  File "scripts/generator.py", line 43, in <module>
    main()
  File "scripts/generator.py", line 17, in main
    (ecs_nested, ecs_flat) = schema_reader.load_ecs()
  File "/Users/akroh/go/src/github.com/elastic/ecs/scripts/schema_reader.py", line 175, in load_ecs
    finalize_schemas(fields_nested, fields_flat)
  File "/Users/akroh/go/src/github.com/elastic/ecs/scripts/schema_reader.py", line 169, in finalize_schemas
    duplicate_reusable_fieldsets(schema, fields_flat, fields_nested)
  File "/Users/akroh/go/src/github.com/elastic/ecs/scripts/schema_reader.py", line 130, in duplicate_reusable_fieldsets
    if 'nestings' not in fields_nested[new_nesting]:
KeyError: 'client.as'
make: *** [generator] Error 1

@andrewkroh
Copy link
Member Author

I reverted the change to make oranization reusable under as for the time being because it wasn't working as expected. For example the organization fields were missing in the places where as was reused (e.g. source.as.organization.name).

Copy link
Contributor

@webmat webmat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the adjustments, @andrewkroh. I'm surprised the nesting didn't work, but this is not a big deal. I'll check this out next time I play in the code.

We're good to merge once the changelog is rebased :-)

neu5ron added a commit to rocknsm/rock-dashboards that referenced this pull request Jul 7, 2019
This adds fields for Autonomous System (AS) information. It's common to enrich
events containing IP addresses with information about the AS controlling that
address.
@andrewkroh andrewkroh merged commit a35a903 into elastic:master Jul 8, 2019
andrewkroh added a commit to andrewkroh/beats that referenced this pull request Jul 23, 2019
Update module pipelines to enrich events with autonomous system info (number and org name).
All of the modules that had a geoip processor now also add autonomous system info. Field naming
is based on ECS (elastic/ecs#341).
andrewkroh added a commit to elastic/beats that referenced this pull request Jul 23, 2019
Update module pipelines to enrich events with autonomous system info (number and org name).
All of the modules that had a geoip processor now also add autonomous system info. Field naming
is based on ECS (elastic/ecs#341).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants