Upgrade es 8.8.0 (#148)

* updated es and logstash version * Updated cws image to use java 11 * Update ReadMe for CI --------- Co-authored-by: Ronny Fray <[email protected]>
NASA-AMMOS · Jun 8, 2023 · 208cc63 · 208cc63
1 parent f4718c3
commit 208cc63
Show file tree

Hide file tree

Showing 10 changed files with 148 additions and 28 deletions.
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
@@ -25,10 +25,10 @@ The `publish-cws-image` job is the CD component of the workflow, triggered upon
     - Image: mariadb:10.3
     - Ports: 3306:3306
 - [**checkout**](https://github.com/marketplace/actions/checkout): This action checks out the repository under `$GITHUB_WORKSPACE`, so the workflow can access it.
-- Set up JDK 8:
+- Set up JDK 11:
   - [**setup-java**](https://github.com/marketplace/actions/setup-java-jdk): This action downloads and sets up a requested version of Java
   - Current configuration:
-      - Java-version: 8
+      - Java-version: 11
       - Distribution: Temurin
       - Cache: Maven
 - **Create open-source certs**:
@@ -37,7 +37,7 @@ The `publish-cws-image` job is the CD component of the workflow, triggered upon
 - **Download Logstash**:
   - [**download-file-action**](https://github.com/marketplace/actions/download-file-to-workspace): This action downloads a file from the internet into the workspace
     - Downloads Logstash using a URL
-    - Renames the file as `logstash-7.16.2.zip`
+    - Renames the file as `logstash-8.8.0.zip`
     - Stores Logstash in appropriate directory
 - **Check for Logstash**:
   - List files in the directory where Logstash is expected
@@ -87,7 +87,7 @@ The `publish-cws-image` job is the CD component of the workflow, triggered upon
 - **Check out the repo**:
   - Utilizes the same `checkout` action to check out the repository again
   - This is done in a new GitHub runner
-- **Set up JDK 8**
+- **Set up JDK 11**
   - Utilizes the same `setup-java` action to set up Java
 - **Log in to Docker Hub**:
   - [**Docker Login**](https://github.com/marketplace/actions/docker-login): This action is used to log in against a Docker registry

diff --git a/.github/workflows/camunda.yml b/.github/workflows/camunda.yml
@@ -49,8 +49,8 @@ jobs:
       - name: Download Logstash
         uses: carlosperate/download-file-action@v1
         with:
-          file-url: https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.16.2-windows-x86_64.zip
-          file-name: logstash-7.16.2.zip
+          file-url: https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-windows-x86_64.zip
+          file-name: logstash-8.8.0.zip
           location: install/logging/
 
       - name: Check for Logstash
@@ -172,8 +172,8 @@ jobs:
       - name: Download Logstash
         uses: carlosperate/download-file-action@v1
         with:
-          file-url: https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.16.2-windows-x86_64.zip
-          file-name: logstash-7.16.2.zip
+          file-url: https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-windows-x86_64.zip
+          file-name: logstash-8.8.0.zip
           location: install/logging/
 
       - name: Check for Logstash

diff --git a/.github/workflows/ldap.yml b/.github/workflows/ldap.yml
@@ -49,8 +49,8 @@ jobs:
       - name: Download Logstash
         uses: carlosperate/download-file-action@v1
         with:
-          file-url: https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.16.2-windows-x86_64.zip
-          file-name: logstash-7.16.2.zip
+          file-url: https://artifacts.elastic.co/downloads/logstash/logstash-8.8.0-windows-x86_64.zip
+          file-name: logstash-8.8.0.zip
           location: install/logging/
 
       - name: Check for Logstash

diff --git a/README.md b/README.md
@@ -23,26 +23,27 @@ See the [wiki](https://github.com/NASA-AMMOS/common-workflow-service/wiki) for m
     - Verify installation using: `mvn -v`
 - [**Docker**](https://docs.docker.com/get-docker/): Used to run external Elasticsearch, and create and configure mariaDB database container
   - Recommended minimum system requirements from Docker Resources window:
-      - CPUs: 4
-      - Memory: 5.00 GB
+      - CPUs: 5
+      - Memory: 14.00 GB
       - Swap: 1 GB
-      - Disk image size: 59.6 GB
+      - Disk image size: 64 GB
 - MariaDB or MySQL database set up on either your local machine or a remote host. You will also need to create the following:
     - A database for CWS to use. `cws_dev` is a good default name.
     - A database user with full access to the above database.
 - [**ITerm2**](https://iterm2.com/): Currently these build scripts include commands to open new terminal windows using ITerm2, so they are best run from that terminal.
-- **Logstash 7.16.2+**: Download Logstash (OSS version) for the Windows platform. Rename the ZIP file from 'logstash-oss-7.16.2-windows-x86_64.zip' to 'logstash-7.16.2.zip' and place in `install/logging/`. This is a temporary workaround while we clean up our installation process. You can find the zip download [here](https://www.elastic.co/downloads/past-releases/logstash-oss-7-16-2).
-- **Elasticsearch 7.16.2+**: CWS requires an externally-configured elasticsearch cluster to be set up. You can use an SSL Secure Elasticsearch with or without authentication, or an Insecure HTTP Elasticsearch.
+- **Logstash 8.8.0+**: Download Logstash for your platform. Uncompress it (only if it is a .tar.gz) and then ZIP back it up with the filename 'logstash-8.8.0.zip' and place in `install/logging/`. This is a temporary workaround while we clean up our installation process. You can find the zip download [here](https://www.elastic.co/downloads/logstash).
+- **Elasticsearch 8.8.0+**: CWS requires an externally-configured elasticsearch cluster to be set up. You can use an SSL Secure Elasticsearch with or without authentication, or an Insecure HTTP Elasticsearch.
   - The "Elasticsearch Setup" instruction below provides a contained Dockerized way of running Elasticsearch. This serves as an alternative to installing Elasticsearch.
 - Tomcat **keystore and truststore files** (needed for CWS web console to work properly):
     - You will need to add your own Tomcat keystore file to this path: `install/.keystore`
     - You will need to add your own truststore file to this path: `install/tomcat_lib/cws_truststore.jks`
     - See: https://tomcat.apache.org/tomcat-9.0-doc/ssl-howto.html
-  - **Java 8 SE JDK**: JDK 9+ deprecated, and JDK 11+ removed, dependencies used by CWS.
+  - **Java 11 JDK**: CWS only runs on JDK 11 now, but planning for JDK 17 soon.
     - For Homebrew users:
-      - Install OpenJDK 8 using: `brew install openjdk@8`.`/usr/local/opt/openjdk@8`
-      - Point Maven to this JDK: `export JAVA_HOME=/usr/local/opt/openjdk@8`
-      - (Optional) Add this to your path: `export PATH="/usr/local/opt/openjdk@8/bin:$PATH"`
+      - Install OpenJDK 11 using: `brew install openjdk@11`
+      - Check the exact version installed using `/usr/libexec/java_home -V`
+      - Add to your Shell startup (e.g. .zprofile): `export JAVA_HOME=$(/usr/libexec/java_home -v X.X.X)`
+        - Replace the X.X.X version above with the OpenJDK 11 output from the `/usr/libexec/java_home -V` command.
 
 
 ### **Development Environment Configuration**

diff --git a/install/dev/bpmn/cmd_sleep_n.bpmn b/install/dev/bpmn/cmd_sleep_n.bpmn
@@ -0,0 +1,117 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:camunda="http://camunda.org/schema/1.0/bpmn" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" id="Definitions_1pjqrkw" targetNamespace="http://bpmn.io/schema/bpmn" exporter="Camunda Modeler" exporterVersion="4.1.1">
+  <bpmn:process id="cmd_sleep_n" name="Command Sleep N seconds" isExecutable="true">
+    <bpmn:startEvent id="StartEvent_1" camunda:asyncBefore="true">
+      <bpmn:outgoing>Flow_1crw3vy</bpmn:outgoing>
+    </bpmn:startEvent>
+    <bpmn:sequenceFlow id="Flow_1crw3vy" sourceRef="StartEvent_1" targetRef="Activity_0dgfqfw" />
+    <bpmn:serviceTask id="Activity_13v4mzy" name="Sleep N sec" camunda:modelerTemplate="jpl.cws.task.CmdLineExecTask" camunda:type="external" camunda:topic="__CWS_CMD_TOPIC__">
+      <bpmn:extensionElements>
+        <camunda:field name="cmdLine">
+          <camunda:expression>bash -c 'sleep ${nSeconds}; echo "This is our sleep ${nSeconds} output"'</camunda:expression>
+        </camunda:field>
+        <camunda:field name="workingDir">
+          <camunda:expression>.</camunda:expression>
+        </camunda:field>
+        <camunda:field name="successExitValues">
+          <camunda:expression>0</camunda:expression>
+        </camunda:field>
+        <camunda:field name="throwOnFailures">
+          <camunda:expression>true</camunda:expression>
+        </camunda:field>
+        <camunda:field name="exitCodeEvents">
+          <camunda:expression>0=success,1=error</camunda:expression>
+        </camunda:field>
+        <camunda:field name="throwOnTruncatedVariable">
+          <camunda:expression>false</camunda:expression>
+        </camunda:field>
+        <camunda:field name="preCondition" />
+        <camunda:field name="onPreConditionFail">
+          <camunda:expression>ABORT_PROCESS</camunda:expression>
+        </camunda:field>
+        <camunda:field name="timeout">
+          <camunda:expression>86400</camunda:expression>
+        </camunda:field>
+        <camunda:field name="retries">
+          <camunda:expression>0</camunda:expression>
+        </camunda:field>
+        <camunda:field name="retryDelay">
+          <camunda:expression>1000</camunda:expression>
+        </camunda:field>
+      </bpmn:extensionElements>
+      <bpmn:incoming>Flow_0wj3rjq</bpmn:incoming>
+      <bpmn:outgoing>Flow_0iisaaq</bpmn:outgoing>
+    </bpmn:serviceTask>
+    <bpmn:sequenceFlow id="Flow_0iisaaq" sourceRef="Activity_13v4mzy" targetRef="Activity_0y8rdd5" />
+    <bpmn:endEvent id="Event_0vmiax7">
+      <bpmn:incoming>Flow_1137p4s</bpmn:incoming>
+    </bpmn:endEvent>
+    <bpmn:sequenceFlow id="Flow_1137p4s" sourceRef="Activity_0y8rdd5" targetRef="Event_0vmiax7" />
+    <bpmn:serviceTask id="Activity_0y8rdd5" name="Log Done" camunda:modelerTemplate="jpl.cws.task.LogTask" camunda:class="jpl.cws.task.LogTask">
+      <bpmn:extensionElements>
+        <camunda:field name="message">
+          <camunda:expression>Process done!</camunda:expression>
+        </camunda:field>
+        <camunda:field name="preCondition">
+          <camunda:expression>none</camunda:expression>
+        </camunda:field>
+        <camunda:field name="onPreConditionFail">
+          <camunda:expression>ABORT_PROCESS</camunda:expression>
+        </camunda:field>
+      </bpmn:extensionElements>
+      <bpmn:incoming>Flow_0iisaaq</bpmn:incoming>
+      <bpmn:outgoing>Flow_1137p4s</bpmn:outgoing>
+    </bpmn:serviceTask>
+    <bpmn:sequenceFlow id="Flow_0wj3rjq" sourceRef="Activity_0dgfqfw" targetRef="Activity_13v4mzy" />
+    <bpmn:serviceTask id="Activity_0dgfqfw" name="Log N seconds" camunda:modelerTemplate="jpl.cws.task.LogTask" camunda:class="jpl.cws.task.LogTask">
+      <bpmn:extensionElements>
+        <camunda:field name="message">
+          <camunda:expression>Now sleeping for ${nSeconds} seconds...</camunda:expression>
+        </camunda:field>
+        <camunda:field name="preCondition">
+          <camunda:expression>none</camunda:expression>
+        </camunda:field>
+        <camunda:field name="onPreConditionFail">
+          <camunda:expression>ABORT_PROCESS</camunda:expression>
+        </camunda:field>
+      </bpmn:extensionElements>
+      <bpmn:incoming>Flow_1crw3vy</bpmn:incoming>
+      <bpmn:outgoing>Flow_0wj3rjq</bpmn:outgoing>
+    </bpmn:serviceTask>
+  </bpmn:process>
+  <bpmndi:BPMNDiagram id="BPMNDiagram_1">
+    <bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="cmd_sleep_n">
+      <bpmndi:BPMNEdge id="Flow_0wj3rjq_di" bpmnElement="Flow_0wj3rjq">
+        <di:waypoint x="370" y="117" />
+        <di:waypoint x="430" y="117" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="Flow_1137p4s_di" bpmnElement="Flow_1137p4s">
+        <di:waypoint x="690" y="117" />
+        <di:waypoint x="752" y="117" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="Flow_0iisaaq_di" bpmnElement="Flow_0iisaaq">
+        <di:waypoint x="530" y="117" />
+        <di:waypoint x="590" y="117" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="Flow_1crw3vy_di" bpmnElement="Flow_1crw3vy">
+        <di:waypoint x="215" y="117" />
+        <di:waypoint x="270" y="117" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="_BPMNShape_StartEvent_2" bpmnElement="StartEvent_1">
+        <dc:Bounds x="179" y="99" width="36" height="36" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="Activity_1e5kiv5_di" bpmnElement="Activity_13v4mzy">
+        <dc:Bounds x="430" y="77" width="100" height="80" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="Event_0vmiax7_di" bpmnElement="Event_0vmiax7">
+        <dc:Bounds x="752" y="99" width="36" height="36" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="Activity_0f331lg_di" bpmnElement="Activity_0y8rdd5">
+        <dc:Bounds x="590" y="77" width="100" height="80" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="Activity_0ne4eiu_di" bpmnElement="Activity_0dgfqfw">
+        <dc:Bounds x="270" y="77" width="100" height="80" />
+      </bpmndi:BPMNShape>
+    </bpmndi:BPMNPlane>
+  </bpmndi:BPMNDiagram>
+</bpmn:definitions>
diff --git a/install/docker/console-db-es-ls-kibana/docker-compose.yml b/install/docker/console-db-es-ls-kibana/docker-compose.yml
@@ -23,7 +23,7 @@ services:
     labels:
       com.example.service: "es"
       com.example.description: "For searching and indexing data"
-    image: elasticsearch:7.16.2
+    image: docker.elastic.co/elasticsearch/elasticsearch:8.8.0
     container_name: cws-es
   #  networks:
   #    - frontend
@@ -35,6 +35,7 @@ services:
       - MAX_MAP_COUNT=262144
       - discovery.type=single-node
       - cluster.name=docker-cluster
+      - xpack.security.enabled=false
       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
     healthcheck:
         test: ["CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1"]
@@ -53,7 +54,7 @@ services:
 #    labels:
 #      com.example.service: "kibana"
 #      com.example.description: "Data visualisation and for log aggregation"
-#    image: kibana:7.16.2
+#    image: kibana:8.8.0
 #    container_name: cws-kibana
 #    ports:
 #      - "5601:5601"
@@ -68,7 +69,7 @@ services:
 #    labels:
 #      com.example.service: "logstash"
 #      com.example.description: "For logging data"
-#    image: logstash:7.16.2
+#    image: logstash:8.8.0
 #    container_name: cws-logstash
 #    volumes:
 #      - ./cws-logstash.conf:/home/cws_user/cws-logstash.conf:ro

diff --git a/install/docker/cws-image/Dockerfile b/install/docker/cws-image/Dockerfile
@@ -1,10 +1,10 @@
 FROM oraclelinux:8
 
 RUN yum update -y && \
-    yum install -y mysql java-1.8.0-openjdk java-1.8.0-openjdk-devel rsync which && \
+    yum install -y mysql java-11-openjdk java-11-openjdk-devel rsync which && \
     yum clean all
 
-ENV JAVA_HOME /usr/lib/jvm/java-1.8.0
+ENV JAVA_HOME /usr/lib/jvm/java-openjdk
 
 ENV TZ=America/Los_Angeles
 RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

diff --git a/install/docker/es-only/docker-compose.yml b/install/docker/es-only/docker-compose.yml
@@ -5,7 +5,7 @@ services:
     labels:
       com.example.service: "es"
       com.example.description: "For searching and indexing data"
-    image: elasticsearch:7.16.2
+    image: docker.elastic.co/elasticsearch/elasticsearch:8.8.0
     container_name: cws-es-only
     networks:
       - es-net
@@ -16,6 +16,7 @@ services:
       - MAX_MAP_COUNT=262144
       - discovery.type=single-node
       - cluster.name=docker-cluster
+      - xpack.security.enabled=false
       - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
     healthcheck:
         test: ["CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1"]
@@ -34,7 +35,7 @@ services:
 #    labels:
 #      com.example.service: "kibana"
 #      com.example.description: "Data visualisation and for log aggregation"
-#    image: kibana:7.16.2
+#    image: kibana:8.8.0
 #    container_name: cws-kibana
 #    ports:
 #      - "5601:5601"

diff --git a/install/docker/worker-ls/docker-compose.yml b/install/docker/worker-ls/docker-compose.yml
@@ -5,7 +5,7 @@ services:
 #    labels:
 #      com.example.service: "logstash"
 #      com.example.description: "For logging data"
-#    image: docker.elastic.co/logstash/logstash:7.16.2
+#    image: docker.elastic.co/logstash/logstash:8.8.0
 #    container_name: cws-worker-logstash
 #    volumes:
 #      - logs-volume:/cws_logs:ro

diff --git a/utils.sh b/utils.sh
@@ -8,7 +8,7 @@
 export CWS_VER='2.4.0'    # update this each CWS release
 export CAMUNDA_VER='7.19.0'
 export TOMCAT_VER='9.0.72'
-export LOGSTASH_VER='7.16.2'
+export LOGSTASH_VER='8.8.0'
 
 # Prints the provided string, tagging with the script that called it
 function print () {