Elastic Stack Release Candidate 1

[dougburks]

• Elasticsearch

  • Elasticsearch 6.1.2
  • max number of threads [2048] for user [elasticsearch] is too low, increase to at least [4096]

• Kibana

  • Kibana 6.1.2
  • metric viz scrollbar issue should be resolved, so replace TSVB metrics with standard metrics
  • enable borders on dashboards and adjust borders where necessary
  • Indicator dashboard - add viz for SSL server_name
  • change Signature_Info to signature_info

• Logstash

  • Logstash 6.1.2
  • pipeline errors related to Suricata - need if [sid] and if [gid] before doing comparisons in 1033_preprocess_snort.conf
  • /etc/logstash/*-template.json causing Elasticsearch to report Deprecated field [template] used, replaced by [index_patterns]
  • /etc/logstash/logstash-template.json causing Elasticsearch to report [_all] is deprecated in 6.0+ and will be removed in 7.0. As a replacement, you can use [copy_to] on mapping fields to create your own catch all field.
  • /etc/logstash/*-template.json causing Elasticsearch to report [_default_] mapping is deprecated since it is not useful anymore now that indexes cannot have more than one type
  • ensure double quotes are escaped/replaced when using csv filter to prevent illegal quoting errors
  • change Signature_Info to signature_info

• ElastAlert

  • need at least version 0.1.26 to support Elasticsearch 6
  • make sure that ElastAlert on master server can query crosscluster *: and get results
  • disable on sensor-only installations

• so-elastic-configure

  • if choosing master-only in Setup, freqserver and domainstats should be disabled
  • if sensor already configured, preserve transport settings in /etc/elasticsearch/elasticsearch.yml

• so-elastic-download

  • Remaster process commented out Docker repo in /etc/apt/sources.list
  • once new securityonion-ossec-rules package is published, add that to Updating existing packages section

• so-allow-view

  • need a script to view ufw status and also custom iptables rules in DOCKER-USER (use the iptbl_stats function in so-allow-elastic, maybe move it to a utility script to be used by both so-allow and so-allow-view)

• Apache

  • add /shorten and /goto Locations into /etc/apache2/sites-available/securityonion.conf and reverse proxy to Kibana - commit

• Sguil

  • make sure pivot uses cross cluster search *:

• Squert

  • make sure pivot uses cross cluster search *:

• sosetup-elastic

  • when configuring sensor only, provide option to send logs to master server rather than storing locally
  • fix verbiage in regard to Elasticsearch disk size when running Custom setup

• sostat

  • include info for sensors connected via ES CCS (master only)

[ghost]

Great looking road map Doug. Exposing the docker so-elastic-net and making easily configurable would allow for accomodating cluster scale out needs. ;) For your perusal: https://thoughts.t37.net/designing-the-perfect-elasticsearch-cluster-the-almost-definitive-guide-e614eabc1a87

[ghost]

To add: for consideration:

Kibana: Home page::

 - Counter for sensor reads ZERO regardless of how many sensors are active or how many ossec agents live/active.
 - Devices counter shows sensors and the master server just fine. 
 - Clarity on the "localhost" within the devices counter list. If this is the host Ubuntu operating system to the docker containers, understood. How can it be renamed ?

[weslambert]

To address your questions:

• Counter for sensor reads ZERO regardless of how many sensors are active or how many ossec agents live/active.
• Devices counter shows sensors and the master server just fine.

Are you sure you have the latest updates? If you click to edit the visualization and check the panel options, you should get the correct results with the index set as *:logstash-* (which should be correct with the latest updates).

• Clarity on the "localhost" within the devices counter list. If this is the host Ubuntu operating system to the docker containers, understood. How can it be renamed ?

This is the host from which the syslog was delivered. This visualization gets the count value by determining the number of unique values for the syslog-host_from field. In the future, for clarification, one possibility may be to enrich this value with a hostname during processing if syslog-host_from is equal to localhost.

Another option would be to update /etc/syslog-ng/syslog-ng.conf with:

options {   
        keep-hostname(yes);   
};  

so that the hostname will be written out as the machine's hostname, instead of "localhost".

Please make sure to pose any other questions or feedback to the mailing list:
https://github.com/Security-Onion-Solutions/security-onion/wiki/MailingLists#mailing-lists

Thanks,
Wes

[ghost]

I have your "pass-thru-cache" enabled for the docker registry and updated all an hour ago.
Found that the index for the visualization reads " :logstash- " time now, after updates to master and sensors.

( I see in the Viz editor that the counter reads correct. Though not on home screen )

A; Found it. The builder index pattern did not match the counter visualization index pattern.

Thx Wes!

[r32rtb]

With Elastic 6.x the mapping type will no longer work, all the reference to type will need to be adjusted if you intend to upgrade to Elastic 6. The use of type:bro_conn and type:bro_dns within the same index will not work.

Multiple mapping types are not supported in indices created in 6.0
The ability to have multiple mapping types per index has been removed in 6.0. New indices will be restricted to a single type. This is the first step in the plan to remove mapping types altogether. Indices created in 5.x will continue to support multiple mapping types.

[dougburks]

Hi @r32rtb ,

Yes, we're aware of the changes in Elastic 6. We've already changed type: to event_type: as part of the upcoming Beta 3 release:
#1172

[dougburks]

submitted for testing:
https://groups.google.com/d/topic/security-onion-testing/bGuFBOozcRY/discussion

[dougburks]

published:
http://blog.securityonion.net/2018/01/security-onion-elastic-stack-release.html

[r32rtb]

Great work guys! Have you thought about scaling this out when you have say 10 SO sensors and say 20K docs per second between all the sensors? I've had to move to a dedicated ES cluster with dedicate logstash servers. Do you think the ssh tunnel to the server will be able to handle?

[dougburks]

Thanks @r32rtb !

Please see:
https://groups.google.com/d/topic/security-onion/EhYIfbwLZRU/discussion

If you have further questions or comments, please use the mailing list for discussion:
https://securityonion.net/wiki/MailingLists

Thanks!