Upgrade ELK stack to v8.x

[caffeinatedpixel]

Closes #74

BeaKer Server Changes:

• Adds the ability to upgrade from Elastic version 7.5.2 to 7.17.9, and subsequently 8.7.0
  • For upgrade installs, v7.17.9 will be installed and then the installer will end. If a user wants to upgrade to v8, the user will have to run the installer again when they're ready to.
  • Elastic v7.17.9 will support Winlogbeat logs from v7.5.2 - v7.17.9
  • Elastic v8.7.0 will support Winlogbeat logs from v7.17.0 - 8.7.0, but already ingested logs using the sysmon-* index will still be visible in Kibana. New logs from <7.17.9 will not be ingested.
• Installer now creates the elastic/kibana service account token automatically, which is now required for Kibana
  • Removed the elasticsearch.username and elasticsearch.password entries in kibana.yml as they are deprecated in favor of the service account token
• Installer now creates required encryption keys for Kibana plugins to work properly
• Installer creates a snapshot repository named beaker and attempts to create a snapshot before upgrading to 8.7.0
• Installer loads index templates, index lifecycle policy, aliases, data streams, and ingest pipelines for the matching Winlogbeat version
  • For Elastic v7.17.9, logs are sent to a data stream that uses the index pattern winlogbeat-7.17.9
  • For Elastic v8.7.0, logs are sent to a data stream that uses the index pattern winlogbeat-8.7.0 , and is processed by an ingest pipeline named winlogbeat-8.7.0-routing
• Updates the Sysmon and Beacon Kibana dashboards to include both the sysmon-YYYY-MM-dd (legacy index pattern), and winlogbeat-%{[agent.version]} pattern.
• Creates the CA before generating any certificates, as the previous usage is now deprecated. Generates CA and certificates using the root user.
• Adds ./kibana/check_kibana.py in order to parse the Kibana status API response and check whether or not Kibana is finished upgrading
  • Not waiting for the saved objects migration will cause the Kibana dashboard upload to fail
• Adds an ELK_STACK_VERSION environment variable to make it easier to detect the currently installed Elastic version
• Adds ./elasticsearch/export_day.sh in order to export a day's worth of data from the winlogbeat-* indices (emulates the daily indices from previous versions)

BeaKer Agent Changes:

• Changes the install directory from C:\Program Files\winlogbeat-* to C:\Program Files\Winlogbeat-BeaKer in order to be able to install newer versions of winlogbeat and better detect the currently installed version
• Adds a parameter, BeatsVersion, that allows the desired version of winlogbeat to be installed to be overriden
• Updates the winlogbeat.yml config as needed for 7.17 and 8.x
• Detects whether or not an intermediate upgrade to 7.17 is needed

Testing (Ubuntu 20.04):

• Upgraded BeaKer v0.13 install and verified that v7.17 was installed and not v8.7.0. Ensured that the service account token, index template, ILM policy, data stream, encryption keys, and dashboards were all created. Ensured that data from the previous version was still visible in the dashboards. Ensured that data from winlogbeat v7.5.2 was still being ingested. Ensured that data from winlogbeat v7.17.9 was being ingested via the winlogbeat-7.17.9 data stream.
• Upgraded the same install to v8.7.0. Ensured that the prompt asking whether to upgrade or reinstall the current version was displayed. Ensured that the prompt asking to create a snapshot before upgrade was displayed and the snapshot was created. Ensured that the ingest pipelines were loaded, Kibana's migration status completed, and that the dashboards were loaded. Ensured that data coming from both winlogbeat 7.17.9 and 8.7.0 was being ingested into Elasticsearch.
• Ensured that a fresh install installs v8.7.0 and completes successfully.

[Zalgo2462]

I've finished reading through the changes. Here's what I found on this first pass:

• I believe there is a bug where BeatsVersion is not passed to the agent install script when the script is started as a normal user.
• The way we've been calling curl to interact with Elastic/ Kibana is insecure. There is a better way to run the command which will prevent leaking the credentials to normal users.
• We need to ensure python3 is available on the host system in shell-lib if we want to use a python script on the Docker host. Alternatively, we could find a way to run check_kibana.py in a Docker container. There may be other workarounds we could pursue such as using a tool like yq or jq in bash.

I also left a handful of questions, and a couple suggestions to add documentation/ comments.

[Zalgo2462]

I'm receiving the following error when trying to perform the initial upgrade from 7.5.2 to 7.17.9:

The exported images archive appears to be empty.

I'm getting this error in generate_installer.sh

I believe this change is producing the error:
f438738#diff-acf40e5c821718823a3526ada2992a23c6dea9dc4c9dfd727f7053074da7feabL104

After the build, I do not have individual, tagged versions of our elastic containers in the results from sudo docker image ls | grep elastic. I'm going to uncomment this line and try again.

Yep that worked. 👍

[Zalgo2462]

Please uncomment this line. It appears that commenting this line prevents docker-compose from labeling the resulting elastic and kibana images with the appropriate version tags.

Consider using the docker-compose.yml syntax for specifying the build-args for each of the containers individually instead of passing the build arg as a flag on the next line.

Example which passes the ELK_STACK_VERSION environment variable in as a build argument:

build:
      context: ./elasticsearch
      args:
        ELK_STACK_VERSION: ${ELK_STACK_VERSION} 

[caffeinatedpixel]

Uncommented the line. I did not convert to docker-compose syntax because it doesn't seem to tag the images as expected. I don't have the bandwidth to look into it at the moment, especially since the current tagging implementation isn't broken.

[Zalgo2462]

My testing environment consists of Zeek watching a Windows domain and sending that data to RITA/ AC-Hunter. As a part of that domain, I have a workstation running a C2 implant alongside the BeaKer agent. I ensured that the workstation was correctly reporting the C2 connection to the existing version of BeaKer (7.5.2).

After working through the bug above, I performed an update to 7.17. I was able to confirm that the new version of the dashboard pulled in the old daily indexes from 7.5.2. Additionally, I was able to confirm that the old daily indexes continued to receive new events from the 7.5.2 winlogbeat agents.

Next, I will upgrade the winlogbeat agent on the workstation running the implant to 7.17.9.

[Zalgo2462]

Would you be open to adding a blurb to the end of the upgrade install depending on the resulting version? A reminder for the user to go upgrade their winlogbeat agents to the appropriate version and then to come back and run the next upgrade (if needed) might be helpful.

[Zalgo2462]

I was able to successfully upgrade winlogbeats to 7.19 and I was able to confirm that it was sending data to the 7.19 data stream in elastic. The upgrade to 7.19 went well all around. I was able to confirm the ILM policies, data streams, index template, and dashboards were all properly upgraded or created.

During the upgrade to 8.x I received the following error:

Running the command in the if statement before this failure case seemed to work okay on its own:

What is odd is that the command in the script seems to work despite the error catch activating. Checking the snapshot repos in the browser reveals that the repo is being created despite what the bash script thinks.

I added -vvv to the curl command to try to see what the issue was.
I saw two interesting things:

• * The requested URL returned error: 500 Internal Server Error
• curl returned 22which means error with the HTTP error code being 400 or above.

In the beaker logs I am seeing:

elasticsearch_1  | "stacktrace": ["org.elasticsearch.repositories.RepositoryVerificationException: [beaker] path  is not accessible on master node",
elasticsearch_1  | "Caused by: org.elasticsearch.ElasticsearchException: failed to create blob container",

[Zalgo2462]

It appears that the elasticsearch group does not have write access to the snapshots folder in the container causing the error above:

Please confirm this is the cause of the error and add a chmod to the Dockerfile to remedy the issue if needed.

[caffeinatedpixel]

Please confirm this is the cause of the error and add a chmod to the Dockerfile to remedy the issue if needed.

Added chmod to give write perms, which matches the permissions of many other folders within the elasticsearch container.
Also added a check to start BeaKer if it isn't already running before creating a snapshot.

[Zalgo2462]

I was able to build a new installer with the latest changes. I copied it over to my system running 7.17 and selected the option to reinstall.

This took care of the snapshot issue, and I was able to re-run the installer to upgrade to 8.x.

The post-installation messages look good to me.

I'll test the final upgrade for the winlogbeat agents tonight.

[Zalgo2462]

When running the powershell install script to upgrade the winlogbeat agent to 8.x, I saw the following error, but I couldn't tell what had actually gone wrong. Everything seemed to still work. I tried to reproduce the error, but I couldn't get it to pop up again.

The 8.x agent appears to work well.