Skip to content

Commit

Permalink
Create CWS v2.4.1 Release Branch (#173)
Browse files Browse the repository at this point in the history
* update configure script for repeat run case

* formatted the backup date YYYY-MM-DD_HH:MM:SS

* removed file copying process from being printed in console

* created backup dir for already run cws

* set backup_ to . dir and maintain root bpmn directory

* included subdirectory for backups and sorted clean folder

* remove commented code

* fixed issue of copied CWS-resources folder to clean_

* changed Re-running CWS statement. Included configuration.properties to clean_ exception to maintain config settings.

* removed dot files from cws root: databaseCreated and installType

* update configure script for repeat run case

* formatted the backup date YYYY-MM-DD_HH:MM:SS

* removed file copying process from being printed in console

* created backup dir for already run cws

* set backup_ to . dir and maintain root bpmn directory

* included subdirectory for backups and sorted clean folder

* remove commented code

* fixed issue of copied CWS-resources folder to clean_

* changed Re-running CWS statement. Included configuration.properties to clean_ exception to maintain config settings.

* removed dot files from cws root: databaseCreated and installType

* now using rsync to copy clean_ --> cws_root

* direct rsync output to /dev/null

* preliminary removing of simple db connection source

* set backup date format to: backup_YYYYMMDD_hhmmss. And remove dot from subdirs of .backups

* changed copy command for BACKUP_DIR

* update root cws remove cmd to, find grep

* updated info about keystore/truststore paths

* Using linux epoch timestamp in name of worker

-

* preliminary removing of simple db connection source

* decreasing starting connections to 1, in pool

* splitting out CWS and Camunda to two different JDBC pools

and also setting removeAbandoned timeout value to be 15 minutes

* Finalize logic for configure script re-runs.

* Update WorkerMonitorBackgroundThread.java

* added http unsecured y/n logic to CWSInstaller

* added unsecured ES prompt in interactive mode

* Switch connection pool from Tomcat to HikariCP

* installer update host variable

* reset validation

* set SSL connection as default

* add https & http endpoint conditions

* init

* reversed branch changes

* ES host https/http condition

* set urlString with full ES endpoint value

* ES authentication curl fixes

* cleared unnecessary indentation in code

* typo fix

* updated procedure

* ES config edited

* init

* added condition for console only to block duplicate

* set condition for console_only number of rows as Max, 1 row

* type fix

* init

* changed name to worker0000

* set default num days to 1

* added workerDeadTimeMillis var

* added threshold for abandoned worker

* use debug log instead of warn log

* add log debug message for detected abandoned workers

* remove FileWriter

* set dev script worker parameter to WORKER_ABANDONED_DAYS

* set log to ERROR

* Added new cmdline task out variable and test model 'test_out_var'

* Better sanitization on worker_abandoned_days input to avoid crash on Integer.parseInt().

* Logging cleanup / overhaul (#81)

* Complete logging overhaul:
 * Upgrade to log4j2 v2.17.1
 * Switch from Tomcat's JULI to log4j2
 * Split logs between catalina.out and cws.log
   * catalina.out is tomcat's stdout stream
   * cws.log is CWS application logs
 * Various updates install scripts to handle/facilitate the above

* * Update logstash configuration for new logging pattern
* Update start_cws.sh to wait for cws.log to be created

* Bug fix: elasticsearch_host (#82)

* bug fix for elasticsearch_host to be set in CWSInstaller

* Updated docker to work with 2.2.0 (#83)

Also updated to version 2.2.0-pre

* fixed Elasticsearch http https validation (#80)

* fixed with interactive prompt for ES auth (#85)

* README: Update logstash and elasticseach versions to 7.16.2 (#84)

* updated logstash and elasticseach versions to 7.16.2

* Update docker for 2.2

* Updated version to 2.2.0-pre.1

* Fixed issue in docker image build script

* FIX: Add rsync linux package to Dockerfile (#90)

* added rsync package to cws image. In Dockerfile

* including which command to Dockerfile

* Updated Version to 2.2.0

* Fix: Modeler Install elements.json (#92)

* init

* add find command to add elements.json

* moved elements.json to /install/modeler

* Reconfig: preserve user's passed props file when used in `/cws` (#93)

* add find and sync command to config properties file

* preserve used configure arg props file inside cws

* sync config file back to cws root

* save config props using config_file_dir

* ignore copying config props if already exists in root

* Feat: Create Elasticsearch `elasticsearch_protocol` Setting (#97)

* check for https or http in es host variable
* added ES protocol and hostname mismatch check to validator
* CWS installer prompt update
* readme: protocol options HTTP or HTTPS
* using elasticsearch_host_init for initial user input

* Add note about JDK8 being required. (#72)

Co-authored-by: Galen Hollins <[email protected]>

* update ES logstash host (#98)

* Update cws-test: Integration and Unit Tests (skipped) (#91)

* modified cws-test UT and IT skipped
* remove ci/aws from cws-os
* Modified ci conf files
* modified ci/ci_local_core.sh
* clean up ci/ci.sh
* add UT/IT tests
* clean up Jenkins
* edit Jenkinsfile
* set cws unit and integration test files to - skip
* update README.md with cws-test test.sh script step
* edit .gitignore for hostname.txt
* update .gitignore

* Allow custom adaptation actions to be added to the actions dropdown (#99)

* Allow custom adaptation actions to be added to the actions dropdown

* wording to make explicit default behavior

* Minor word change

* Added documentation on how to create and invoke actions in an adaptation project

Co-authored-by: zef <[email protected]>

* Updated readme (#101)

* Updated readme

* Fix: Double Log Output (#102)

* set log4j2 additivity to false

* update log4j2 properties

* Fix: SMTP Email Error (#95)

Fixed SMTP Email "From" Error

* Added to explicitly call the history service cleanup (#106)

Co-authored-by: James Wood <[email protected]>

* Added warning to process that complete on different worker (#107)

* CIGitHubActions (#105)

* Create maven.yml

* CI test.sh update

* CI test.sh update

* CI test.sh update

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated workflow.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* Updated yml file.

* update-tests (#103)

* Fixed location of configuration.properties file used in installerTest.sh
* Restored WebTest and subtests
* Fixed integration tests
* Updated InitiatorsTestIT

* fix tasklist start issue, added UUID (#108)

* update-tests-and-CI (#109)

* updated ci file.
* created open-source certs.
* updated yml file.
* set chromedriver to headless.
* upload Jacoco report as an artifact.
* updated CI workflow.

* Docker improvements (#110)

* Updated docker setup
* Separated console and worker.  Now console only
* Works with cws 2.3 now
* Worker-only waits for console to startup before running configure.sh
* Updated worker-ls to default connect to console-db-es-ls-kibana setup
* Updated README.md

* Bumped version to 2.3.0-pre.1

* update-webtests (#111)

* run Google Chrome.
* updated WebTestUtil.
* upload test screenshots for debugging.
* updated webtests.
* Updated Test Logs Page BPMN file.
* All webtests updated and working.
* Updated test_workers_page BPMN file.
* Set workflow to run daily at 5 AM PST.
* Updated WorkersTestIT.
* Send Slack message to CWS Dev Team.
* Create LoadTestIT.
* Update unit tests.
* Update Maven dependencies for asserts.
* Updated workflow file.

* Add ability to invoke adaptation js code to the workers page (#113)

Co-authored-by: zef <[email protected]>

* OpenLDAP: open source docker ldap (#115)

* add openldap Docker setup
* update to cws-opensource-ldap and cws-certs
* test ldap server from inside Docker container
* clean up readme files
* re-title certs readme

* update-unit-tests (#114)

* Update InstallerTest for CI.
* Enable run of workflow using the Actions tab.
* Test sending workflow failure message.
* Add webhook payload variable.
* Test sending workflow failure message.
* Show webhook payload.
* Add workflow job URL to Slack message.
* Run Unit and Integration Tests Separately.
* Add processEnginePlugins property to camunda.cfg.xml
* Restore EmailTaskTest
* Update workflow file.
* Display actual JSON data in history page

* Update cmdline log msgs (#117)

* Cleaned up cmdline log messages
* Changed Cmdline output so need to update test

Co-authored-by: James Wood <[email protected]>

* Bumped version to 2.3.0 Final

* Create LDAP CI test (#116)

* Create ldap.yml
* Create a continuous integration pipeline for CWS using LDAP security and associated tests.
* Test security env variable.
* Configure open source LDAP for CI.
* Adjust wait time for web elements
* Create WebTestLdap
* Publish CWS Docker image
* Adjust WebTestUtil for timeout error
* Run WebTestLdap
* Update WebTestUtil
* Update workflow files
* Update web elements for integration tests

* Create ReadMe for CI (#118)

* Create ReadMe for CI
* Update ReadMe for CI

* init: slack notif success build (#119)

* cron job set to sundays

* Update: Github actions Ldap weekly build (#120)

* ldap set to weekly build on sunday

* Update ldap.yml

Changed from running Sunday to Monday

* History json output (#122)

* Pretty up json output

* Fixed to work in Docker.  Doesn't seem like proper open/close tagging, but it works...

Co-authored-by: James Wood <[email protected]>
Co-authored-by: Josh Haile <[email protected]>

* Update maven.yml

Changed to run on Monday morning

* Restore LoadTestIT (#124)

* Restore LoadTestIT
Update SystemLevelTestIT
Configure CI to allow multiple workers

* Update ReadMe for CI
Change trigger for CD component to tag
Update webtests for multiple workers

* Update ci.sh

* Update WebTestUtil

* Update InstallerTest
Revert CI to one worker for testing

* Revert CI to three workers

* Configure CI to two workers

* Create MariaDB service container

* Revert CI to three workers

* Revert CI to three workers

* Configure CI for two workers
Update ReadMe for CI

* Update workflow files
Update ReadMe for CI

* Update WebTestUtil

* Update ReadMe for CI
Update workflow files

* Update WebTestUtil

* Create advanced-test job for CI
Add workers environmental variable
Update LoadTestIT

* Update camunda.yml

* Update LoadTestIT
Update SystemLevelTestIT
Skip LoadTestIT when running integration tests

* Update camunda.yml

* Update LoadTestIT
Update camunda.yml

* Update LoadTestIT

* Update LoadTestIT

* Upload screenshots always after successful build

* Check for idle browser during testing
Update LoadTestIT

* Check for idle browser during testing
Update camunda.yml

* Update LoadTestIT

* Update ReadMe for CI
Refactor Test Suite

* Refactor Test Suite

* Refactor Test Suite
Update workflow files

* Update Slack message with Camunda workflow results

* Update Slack message with detailed job results

* Update Slack message with Camunda workflow results

* Update Slack message with Camunda workflow results

* Update Slack message with detailed job results

* Show webhook payload
Update workflow files

* Test sending Slack message based on user
Update workflow files

* Test sending Slack message based on user

* Test sending Slack message based on user

* Test sending Slack message based on user

* Test sending Slack message based on dev team list

* Test sending Slack message based on dev team list

* Test sending Slack message based on dev team list

* Test sending Slack message based on dev team list

* Verify if a user belongs to the team

* Verify if a user belongs to the team

* Send Slack message only if workflows are triggered by the dev team

* Update ReadMe to reflect new workflow file (#128)

* IDS-9547: Max Processes Per Worker  (#126)

* add max process per worker config

* modify cws_worker in core db

* modify core sql

* rename new proc variable worker_max_num_running_procs

* MaxNumForProcsOnWorker variable for current Running processes

* remove added totalNumOfProcDefKeysOnWorker var

* clean up workerservice commit

* set cap on queryLimit for claiming a process start

* rename variables

* add upgrade script for DB

* working but needs clean and optimization

* clean up logs

* clean up CWSInstaller

* worker_max_num_running_procs changed to default 16

* variable description update

* removed limitProcDefKey condition in claimWithCounter

* set limit param to null. All enabled procs have opporunity to be claimed. update README

* claim query

* Bumped version to 2.4.0pre.1

* remove log lines (#142)

* IDS-9777: Upgrade schema CWSv2.4 (#129)

* upgrade schema for cws2.4 - script

* script and sql file

* upgrade script for CWSv2.4 database

* edit create_server_dist

* update the readme

* fix upgrade_core_db column name - max_num_running_procs

* fix script - insert default CORE to new column

* clean up script alter command

* update to README file

* change readme versions text

* add warning

* add delete worker rows to mysql commands

* add upgrade script for 2.1 and 2.2

* add 2.2 and 2.3 to dist

* rename

* revert to 1 upgrade script

* fix var (#146)

* Bumped version to 2.4.0

---------

Co-authored-by: Galen A Hollins <[email protected]>
Co-authored-by: Zachary Taylor <[email protected]>
Co-authored-by: Galen Hollins <[email protected]>
Co-authored-by: James Wood <[email protected]>
Co-authored-by: jamesfwood <[email protected]>
Co-authored-by: David Cuthbert <[email protected]>
Co-authored-by: zonagit <[email protected]>
Co-authored-by: zef <[email protected]>
Co-authored-by: RonnyFrayRegato <[email protected]>
  • Loading branch information
10 people authored Aug 22, 2023
1 parent 5f9330e commit 96a27a5
Show file tree
Hide file tree
Showing 134 changed files with 4,576 additions and 1,848 deletions.
127 changes: 127 additions & 0 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# CWS Continuous Integration (CI) Pipeline

The following workflows establish continuous integration and continuous delivery (CI/CD) pipelines for this repository containing the open-source release of the [NASA-AMMOS](https://ammos.nasa.gov/) Common Workflow Service (CWS).

Utilizing GitHub Actions, the workflows build, test, and deliver CWS by configuring steps required to accomplish these tasks, making use of Marketplace Actions in the process. There are currently two CI pipelines: [CWS CI Camunda](https://github.com/NASA-AMMOS/common-workflow-service/actions/workflows/camunda.yml) and [CWS CI LDAP](https://github.com/NASA-AMMOS/common-workflow-service/actions/workflows/ldap.yml).

## CWS CI Camunda Workflow

The CWS CI Camunda workflow is triggered upon any push to the repository, including commits, pull requests, and merges. The workflow is also scheduled to run every Monday at 5 AM PST / 12 PM UTC. It is composed of three jobs: `build-and-test-cws`, `advanced-test`, and `publish-cws-image`. All jobs run on separate GitHub runners with the latest version of Ubuntu.

The `build-and-test-cws` job performs all the necessary preliminary steps required to configure and build CWS with Camunda security. Once these steps are completed, a fully-functioning instance of CWS is built. Following the build, unit and integration tests are run, with resulting artifacts being saved to the workflow run. Afterwards, data from the workflow is sent to the development team.

The `advanced-test` job is fundamentally the same for the build of CWS, except 2 workers being started instead of 1. Following the build, an advanced integration test requiring additional workers is run.

The `publish-cws-image` job is the CD component of the workflow, triggered upon a commit with a tag and dependant upon successful completion of the `build-and-test-cws` and `advanced-test` jobs.

> **Note**
> The current GitHub runner configuration used for the build and test of CWS, as well as the advanced test, supports up to 1 console and 2 workers.
> More than 2 workers may result in test failures due to the stress on the runner.
### build-and-test-cws Job

- [**Services**](https://docs.github.com/en/actions/using-containerized-services/about-service-containers):
- MariaDB
- Image: mariadb:10.3
- Ports: 3306:3306
- [**checkout**](https://github.com/marketplace/actions/checkout): This action checks out the repository under `$GITHUB_WORKSPACE`, so the workflow can access it.
- Set up JDK 8:
- [**setup-java**](https://github.com/marketplace/actions/setup-java-jdk): This action downloads and sets up a requested version of Java
- Current configuration:
- Java-version: 8
- Distribution: Temurin
- Cache: Maven
- **Create open-source certs**:
- Runs the `generate-certs.sh` bash script
- Creates and stores keystore and truststore required to access CWS UI
- **Download Logstash**:
- [**download-file-action**](https://github.com/marketplace/actions/download-file-to-workspace): This action downloads a file from the internet into the workspace
- Downloads Logstash using a URL
- Renames the file as `logstash-7.16.2.zip`
- Stores Logstash in appropriate directory
- **Check for Logstash**:
- List files in the directory where Logstash is expected
- Can be used to verify successful download of Logstash
- **Set up Elasticsearch**:
- Creates and starts a Docker container with Elasticsearch
- Runs in the background (detached mode)
- **Show Docker containers**: Shows all Docker containers and their statuses
- **Build CWS**:
- Builds and runs CWS
- Begins by running the first bash script in the build process: run_ci.sh
- The bash script is passed the `SECURITY` environmental variable to run CWS in a specific security mode
- The CWS security mode for this workflow is `CAMUNDA`
- The bash script is passed the `WORKERS` environmental variable to start the specified number of workers
- Current configuration is set to run 1 console and 1 worker
- **Show CWS Log**:
- List files in the directory where `cws.log` is expected
- The presence of the file verifies successful deployment of the Apache Tomcat server
- **Set up Google Chrome**
- Installs or updates the latest stable version of Google Chrome
- Required for Selenium-based integration tests
- **Run Unit Tests**: Runs the JUnit-based unit tests with Jacoco code coverage
- **Run Integration Tests**: Runs the Selenium-based integration tests with Jacoco code coverage
- **Upload Jacoco report**:
- [**upload-artifact**](https://github.com/marketplace/actions/upload-a-build-artifact): This action uploads artifacts from the workflow
- Uploads the Jacoco code coverage reports as artifacts to the workflow run
- **Upload test screenshots**: Uploads the test screenshots produced during the integration tests as artifacts to the workflow run
- **Send custom JSON data to Slack workflow**:
- [**slack-send**](https://github.com/marketplace/actions/slack-send): This action sends data into a Slack channel
- Utilizes Technique 1 - Slack Workflow Builder
- Sends GitHub Actions workflow data to a Slack channel via a webhook URL
- Requires a Slack workflow using webhooks to be created

### advanced-test Job
- **Build CWS**:
- Builds and runs CWS
- Begins by running the first bash script in the build process: run_ci.sh
- The bash script is passed the `SECURITY` environmental variable to run CWS in a specific security mode
- The CWS security mode for this workflow is `CAMUNDA`
- The bash script is passed the `WORKERS` environmental variable to start the specified number of workers
- Current configuration is set to run 1 console and 2 workers
- **Run Load Integration Test**:
- Runs the Selenium-based LoadTestIT
- Requires a minimum of 2 workers to be started in order to run properly

### publish-cws-image Job
- **Check out the repo**:
- Utilizes the same `checkout` action to check out the repository again
- This is done in a new GitHub runner
- **Set up JDK 8**
- Utilizes the same `setup-java` action to set up Java
- **Log in to Docker Hub**:
- [**Docker Login**](https://github.com/marketplace/actions/docker-login): This action is used to log in against a Docker registry
- **Generate CWS Docker image**
- Runs the `./build.sh` bash script
- Generates and tags a CWS Docker image
- **Re-Tag CWS Docker image for open source**
- Re-tags the CWS Docker image created from previous step
- Necessary to publish image on an open source Docker repository
- **Push CWS Docker image**: Publishes CWS Docker image to open source Docker repository

## CWS CI LDAP Workflow

The CWS CI LDAP workflow is triggered upon any push to the repository, including commits, pull requests, and merges. The workflow is also scheduled to run every Monday at 5 AM PST / 12 PM UTC. It is composed of one job, `build-and-test-cws`, and runs on a GitHub runner with the latest version of Ubuntu.

The `build-and-test-cws` job performs all the necessary preliminary steps required to configure and build CWS with LDAP security. Once these steps are completed, a fully-functioning instance of CWS is built. Following the build, LDAP-specific integration tests are run. Afterwards, data from the workflow is sent to the development team.

The following are key differences in the steps of the `build-and-test-cws` job between the `CWS CI Camunda` and `CWS CI LDAP` workflows.

> **Note**
> The current GitHub runner configuration used for the build and test of CWS supports up to 1 console and 2 workers.
> More than 2 workers may result in test failures due to the stress on the runner.
### build-and-test-cws Job
- **Set up CWS LDAP Server**:
- Creates and starts a Docker container with an LDAP server
- Runs in the background (detached mode)
- **Build CWS**:
- Builds and runs CWS
- Begins by running the first bash script in the build process: run_ci.sh
- The bash script is passed the `SECURITY` environmental variable to run CWS in a specific security mode
- The CWS security mode for this workflow is `LDAP`
- The bash script is passed the `WORKERS` environmental variable to start the specified number of workers
- Current configuration is set to run 1 console and 1 worker
- **Run LDAP Integration Tests**:
- Runs LDAP-specific, Selenium-based integration tests
- Uses a different naming scheme to prevent the tests from running with the standard unit and integration tests
Loading

0 comments on commit 96a27a5

Please sign in to comment.