Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Populate NetBox inventory via passively-gathered network traffic metadata #135

Closed
mmguero opened this issue Dec 7, 2022 · 2 comments
Closed

Populate NetBox inventory via passively-gathered network traffic metadata #135

mmguero opened this issue Dec 7, 2022 · 2 comments
Assignees
@mmguero
Labels
enhancement New feature or request netbox Related to Malcolm's use of NetBox
Milestone
v24.06.0

Comments

@mmguero
Copy link
Collaborator

mmguero commented Dec 7, 2022 •
edited
Loading

Feature-tracking issue dependent on #131

If the LOGSTASH_NETBOX_AUTO_POPULATE environment variable in ./config/logstash.env is set to true, uninventoried devices with private IP addresses (as defined in RFC 1918 and RFC 4193) observed in known network segments will be automatically created in the NetBox inventory based on the information available. This value is set to true by answering Y to "Should Malcolm automatically populate NetBox inventory based on observed network traffic?" during configuration.

However, careful consideration should be made before enabling this feature: the purpose of an asset management system is to document the intended state of a network: with Malcolm configured to populate NetBox with the live network state, a network misconfiguration fault could result in an incorrect documented configuration.

Devices created using this autopopulate method will have their status field set to staged. It is recommended that users periodically review automatically-created devices for correctness and to fill in known details that couldn't be determined from network traffic. For example, the manufacturer field for automatically-created devices will be set based on the organizational unique identifier (OUI) determined from the first three bytes of the observed MAC address, which may not be accurate if the device's traffic was observed across a router. If possible, observed hostnames will be used in the naming of the automatically-created devices, falling back to the device manufacturer otherwise (e.g., MYHOSTNAME @ 10.10.0.123 vs. Schweitzer Engineering @ 10.10.0.123).

Since device autocreation is based on IP address, information about network segments (including virtual routing and forwarding (VRF) and prefixes) must be first manually specified in NetBox in order for devices to be automatically populated.

Although network devices can be automatically created using this method, services should inventoried manually. The Uninventoried Observed Services visualization in the Zeek Known Summary dashboard can help users review network services to be created in NetBox.

See idaholab/Malcolm#135 for more information on this feature.
@mmguero mmguero added enhancement New feature or request netbox Related to Malcolm's use of NetBox labels Dec 7, 2022
@mmguero mmguero added this to Malcolm Dec 7, 2022
@mmguero mmguero mentioned this issue Dec 7, 2022
Closed
@mmguero mmguero moved this to Todo (design) in Malcolm Dec 7, 2022
@mmguero mmguero moved this from Todo (design) to Todo in Malcolm Dec 7, 2022
@mmguero mmguero moved this from Todo to Todo (design) in Malcolm Dec 7, 2022
@mmguero mmguero self-assigned this May 18, 2023
@mmguero mmguero moved this from Todo (design) to In Progress in Malcolm May 31, 2023
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jun 1, 2023
@mmguero
beginning work on populate netbox inventory passively (idaholab#135)
1664c46
@mmguero mmguero added this to the v23.06.0 milestone Jun 1, 2023
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jun 1, 2023
@mmguero
work on populate netbox inventory passively (idaholab#135): reorganiz…
2aed1b3 
…ing logstash pipeline so we have the data we need at autopopulate time
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 5, 2023
@mmguero
work in progress for auto-population, idaholab#135
a103d43
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 5, 2023
@mmguero
work in progress for auto-population, idaholab#135
5673be0
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 6, 2023
@mmguero
work in progress for auto-population, idaholab#135
1002302
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 6, 2023
@mmguero
work in progress for auto-population, idaholab#135
b6ab14b
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 6, 2023
@mmguero
work in progress for auto-population, idaholab#135
5e04fb1
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 6, 2023
@mmguero
work in progress for auto-population, idaholab#135
bb7bd3b
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 6, 2023
@mmguero
work in progress for auto-population, idaholab#135
f9039aa
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 6, 2023
@mmguero
work in progress for auto-population, idaholab#135
bd8e8ba
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 6, 2023
@mmguero
work in progress for auto-population, idaholab#135
a70971b
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 7, 2023
@mmguero
work in progress for auto-population, idaholab#135
68e8ebc
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 7, 2023
@mmguero
work in progress for auto-population, idaholab#135
564ed8c
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 7, 2023
@mmguero
work in progress for auto-population, idaholab#135
d9f93c0
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 7, 2023
@mmguero
work in progress for auto-population, idaholab#135
abe7f07
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 7, 2023
@mmguero
work in progress for auto-population, idaholab#135
b347d63
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 7, 2023
@mmguero
work in progress for auto-population, idaholab#135
00fdc01
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 7, 2023
@mmguero
work in progress for auto-population, idaholab#135
a15da08
@mmguero
Copy link
Collaborator Author

mmguero commented Jul 18, 2023

This is basically complete for v23.07.0, but it will be enhanced/improved in v23.08.0.

@mmguero mmguero removed this from the v23.07.0 milestone Jul 18, 2023
@mmguero mmguero added this to the v23.08.0 milestone Jul 18, 2023
This was referenced Jul 18, 2023
Merged
Closed
Merged
mmguero added a commit to cisagov/Malcolm that referenced this issue Jul 19, 2023
@mmguero
Merge pull request #267 from cisagov/v23.07.0_merge_cisagov
432a90a 
Malcolm v23.07.0 is a feature release with a number of improvements, bux fixes and component updates.

v23.05.1...v23.07.0

* New features
    - scan docker images built via GitHub actions for vulnerabilities using Trivy (idaholab#218)
    - document building and deplolying Malcolm with an AWS AMI image (idaholab#205)
    - handle Arkime field actions (idaholab#200)
    - kubernetes: document how to get running on Amazon EKS (idaholab#194)
    - Populate NetBox inventory via passively-gathered network traffic metadata (basic functionality, work in progress) (idaholab#135)

* Enhancements
    - use .tar.xz instead of .tar.gz for packaging Malcolm docker images for better compression (and smaller ISO file size)
    - Malcolm documentation edits (idaholab#204)
    - add option to enable SSH via password in hedgehog's configure-interfaces.py script (idaholab#158)
    - updated "Network Traffic Analysis with Malcolm" slides
    - use an init container in Kubernetes container startup to ensure necessary directories get created under PersistentVolume objects before startup
    - improvements to identifying source of third-party logs sent via fluent bit
    - don't do unnecessary clone of Zeek plugins, just install using URL
    - parse [bacnet_device_control.log](https://github.com/cisagov/icsnpp-bacnet/#device-control-log-bacnet_device_controllog) produced by the icsnpp-bacnet parser for Zeek

* Bug fixes
    - maxlogins value includes tmux sessions, can lock user out of SSH (idaholab#214)
    - curl rc file for connecting to external OpenSearch without auth enabled causes logstash startup to fail (idaholab#209)
    - failure to parse some suricata alerts due to integer type which should be indexed as long (idaholab#206)
    - netbox-restore doesn't work in Kubernetes (idaholab#202)
    - PCAP File with no `-` in pcapng Fails to Upload (#265)
    - disable NetBox telemetry

* Component version updates
    - Alpine (docker container image base) to [v3.18.0](https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html)
    - Arkime to [v4.3.2](https://github.com/arkime/arkime/blob/8bd9d1ccaf3214eeb07da910c45d6172f9ff4ca8/CHANGELOG#L40-L55)
    - capa to [v6.0.0](https://github.com/mandiant/capa/releases/tag/v6.0.0)
    - filebeat to [v8.8.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.8.2.html)
    - NetBox to [v3.5.4](https://github.com/netbox-community/netbox/releases/tag/v3.5.4)
    - OpenSearch and OpenSearch Dashboards to [v2.8.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.8.0.md)
    - Supercronic to [v0.2.25](https://github.com/aptible/supercronic/releases/tag/v0.2.25)
    - YARA to [v4.3.2](https://github.com/VirusTotal/yara/releases/tag/v4.3.2)
    - Zeek to [v5.2.2](https://github.com/zeek/zeek/releases/tag/v5.2.2)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from [https://malcolm.fyi/](https://malcolm.fyi/docs/download.html).
mmguero added a commit that referenced this issue Jul 19, 2023
@mmguero
Merge pull request #224 from idaholab/v23.07.0_merge_idaholab
924431d 
Malcolm v23.07.0 is a feature release with a number of improvements, bux fixes and component updates.

v23.05.1...v23.07.0

* New features
    - scan docker images built via GitHub actions for vulnerabilities using Trivy (#218)
    - document building and deplolying Malcolm with an AWS AMI image (#205)
    - handle Arkime field actions (#200)
    - kubernetes: document how to get running on Amazon EKS (#194)
    - Populate NetBox inventory via passively-gathered network traffic metadata (basic functionality, work in progress) (#135)

* Enhancements
    - use .tar.xz instead of .tar.gz for packaging Malcolm docker images for better compression (and smaller ISO file size)
    - Malcolm documentation edits (#204)
    - add option to enable SSH via password in hedgehog's configure-interfaces.py script (#158)
    - updated "Network Traffic Analysis with Malcolm" slides
    - use an init container in Kubernetes container startup to ensure necessary directories get created under PersistentVolume objects before startup
    - improvements to identifying source of third-party logs sent via fluent bit
    - don't do unnecessary clone of Zeek plugins, just install using URL
    - parse [bacnet_device_control.log](https://github.com/cisagov/icsnpp-bacnet/#device-control-log-bacnet_device_controllog) produced by the icsnpp-bacnet parser for Zeek

* Bug fixes
    - maxlogins value includes tmux sessions, can lock user out of SSH (#214)
    - curl rc file for connecting to external OpenSearch without auth enabled causes logstash startup to fail (#209)
    - failure to parse some suricata alerts due to integer type which should be indexed as long (#206)
    - netbox-restore doesn't work in Kubernetes (#202)
    - PCAP File with no `-` in pcapng Fails to Upload (cisagov#265)
    - disable NetBox telemetry

* Component version updates
    - Alpine (docker container image base) to [v3.18.0](https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html)
    - Arkime to [v4.3.2](https://github.com/arkime/arkime/blob/8bd9d1ccaf3214eeb07da910c45d6172f9ff4ca8/CHANGELOG#L40-L55)
    - capa to [v6.0.0](https://github.com/mandiant/capa/releases/tag/v6.0.0)
    - filebeat to [v8.8.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.8.2.html)
    - NetBox to [v3.5.4](https://github.com/netbox-community/netbox/releases/tag/v3.5.4)
    - OpenSearch and OpenSearch Dashboards to [v2.8.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.8.0.md)
    - Supercronic to [v0.2.25](https://github.com/aptible/supercronic/releases/tag/v0.2.25)
    - YARA to [v4.3.2](https://github.com/VirusTotal/yara/releases/tag/v4.3.2)
    - Zeek to [v5.2.2](https://github.com/zeek/zeek/releases/tag/v5.2.2)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from [https://malcolm.fyi/](https://malcolm.fyi/docs/download.html).
@mmguero mmguero modified the milestones: v23.08.0, v23.09.0 Aug 16, 2023
@mmguero mmguero modified the milestones: v23.09.0, v23.10.0 Sep 7, 2023
@mmguero mmguero modified the milestones: v23.10.0, v23.11.0 Sep 18, 2023
@mmguero mmguero modified the milestones: v23.11.0, v23.12.0 Nov 9, 2023
@mmguero mmguero added the CISA label Nov 13, 2023
@mmguero mmguero removed this from the v23.12.0 milestone Nov 27, 2023
@mmguero mmguero added this to the z.staging milestone Jan 15, 2024
@mmguero
Copy link
Collaborator Author

mmguero commented Feb 15, 2024

I'm going to close this and open individual issues for improvements to this feature.

@mmguero mmguero closed this as completed Feb 15, 2024
@github-project-automation github-project-automation bot moved this from In Progress to Done in Malcolm Feb 15, 2024
@mmguero mmguero moved this from Done to Released in Malcolm Feb 15, 2024
@mmguero mmguero modified the milestones: z.staging, v24.06.0 Oct 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
enhancement New feature or request netbox Related to Malcolm's use of NetBox
Projects
Malcolm
Status: Released
Milestone
v24.06.0
Development

No branches or pull requests
1 participant
@mmguero