Merge master into fhir-term-graph (#1853)

* ci: introduce integration tests for fhir-audit

Signed-off-by: Paul Bastide <[email protected]>

* ci: introduce integration tests for fhir-audit

Signed-off-by: Paul Bastide <[email protected]>

* ci: add integration tests for fhir-audit feature and fix one bug with use of/from

Signed-off-by: Paul Bastide <[email protected]>

* fix: pseudo tty

Signed-off-by: Paul Bastide <[email protected]>

* fix: small change to the timeout length to 120s

Signed-off-by: Paul Bastide <[email protected]>

* fix: alternative method for getting the results from the kafka-1 container

Signed-off-by: Paul Bastide <[email protected]>

* fix: alternative method for getting the results from the kafka-1 container

Signed-off-by: Paul Bastide <[email protected]>

* fix: add get_results.sh creation of the directory

Signed-off-by: Paul Bastide <[email protected]>

* fix: update

Signed-off-by: Paul Bastide <[email protected]>

* fix: update

Signed-off-by: Paul Bastide <[email protected]>

* fix: for privileged execution

Signed-off-by: Paul Bastide <[email protected]>

* fix: difference running ci local and remote

Signed-off-by: Paul Bastide <[email protected]>

* fix: audit

Signed-off-by: Paul Bastide <[email protected]>

* fix: audit with docker copy

Signed-off-by: Paul Bastide <[email protected]>

* fix: adding tty support and stdin support

Signed-off-by: Paul Bastide <[email protected]>

* fix: work around tty issue

Signed-off-by: Paul Bastide <[email protected]>

* removing the tty references -it

Signed-off-by: Paul Bastide <[email protected]>

* changed the execution pattern

Signed-off-by: Paul Bastide <[email protected]>

* changed the execution pattern

Signed-off-by: Paul Bastide <[email protected]>

* fix: update to predefine output file

Signed-off-by: Paul Bastide <[email protected]>

* fix: update to predefine output file

Signed-off-by: Paul Bastide <[email protected]>

* fix permissions

Signed-off-by: Paul Bastide <[email protected]>

* fix permissions

Signed-off-by: Paul Bastide <[email protected]>

* fix permissions

Signed-off-by: Paul Bastide <[email protected]>

* fix permissions

Signed-off-by: Paul Bastide <[email protected]>

* fix permissions

Signed-off-by: Paul Bastide <[email protected]>

* fix permissions

Signed-off-by: Paul Bastide <[email protected]>

* issues #1839 and #1743 - support search parameter disambiguation

1. Update ParametersMap to support storing multiple search parameters
with the same code
2. Address #1743 by collecting to a map instead of a list
3. Update SearchUtil.getSearchParameter to lookup the search parameter
by URI from the config if possible (instead of applying the filter to
the full set of built-in parameters).
4. Update the docs to reflect that search parameter filtering now
applies to tenant-specific search parameters as well. This should help
us move toward #1596

Also fixed a bad trace message and did some minor formatting / javadoc.

Signed-off-by: Lee Surprenant <[email protected]>

* ci: work around issue with tty

Signed-off-by: Paul Bastide <[email protected]>

* fix: test that doesn't account for year shifts

Signed-off-by: Paul Bastide <[email protected]>

* fix: test that doesn't account for year shifts

Signed-off-by: Paul Bastide <[email protected]>

* ci: work around issue with tty

Signed-off-by: Paul Bastide <[email protected]>

* remove hardcoded year from SearchLastUpdatedIdTest

Signed-off-by: Lee Surprenant <[email protected]>

* fix: change the integration pattern slightly for tty

Signed-off-by: Paul Bastide <[email protected]>

* fix: change the integration pattern slightly for tty

Signed-off-by: Paul Bastide <[email protected]>

* fix: change the integration pattern slightly for tty

Signed-off-by: Paul Bastide <[email protected]>

* fix: last two tests to update with dynamic year

Signed-off-by: Paul Bastide <[email protected]>

* issues #1839 and #1743 - support search parameter disambiguation

1. Update ParametersMap to support storing multiple search parameters
with the same code
2. Address #1743 by collecting to a map instead of a list
3. Update SearchUtil.getSearchParameter to lookup the search parameter
by URI from the config if possible (instead of applying the filter to
the full set of built-in parameters).
4. Update the docs to reflect that search parameter filtering now
applies to tenant-specific search parameters as well. This should help
us move toward #1596

Also fixed a bad trace message and did some minor formatting / javadoc.

Signed-off-by: Lee Surprenant <[email protected]>

* Apply suggestions from code review

Signed-off-by: Lee Surprenant <[email protected]>

* Update build/audit/README.md

Signed-off-by: Paul Bastide <[email protected]>

Co-authored-by: Lee Surprenant <[email protected]>

* add info on accessing the bulk operation job logs

Signed-off-by: Lee Surprenant <[email protected]>

* Add unit tests for the ParametersMap

Also made a minor change to insertAll so it gets the code from the
existing ParametersMap instead of from the SearchParameters in the map.
Usually these are the same, but they can differ.

Signed-off-by: Lee Surprenant <[email protected]>

* Issue #1849 - AuthZ interceptor validate/convert search requests

Signed-off-by: Mike Schroeder <[email protected]>

* Issue #1849 - address review comments

Signed-off-by: Mike Schroeder <[email protected]>

Co-authored-by: Paul Bastide <[email protected]>
Co-authored-by: Lee Surprenant <[email protected]>
Co-authored-by: Mike Schroeder <[email protected]>
Co-authored-by: Michael W Schroeder <[email protected]>

.github/workflows/audit.yml

@@ -0,0 +1,48 @@
+name: Integration - Audit
+
+on:
+  pull_request:
+    paths-ignore:
+      - 'docs/**'
+      - '.github/workflows/site.yml'
+      - '.github/workflows/release.yml'
+
+jobs:
+  e2e-audit:
+    runs-on: ubuntu-latest
+    if: "!contains(github.event.pull_request.labels.*.name, 'ci-skip')"
+    strategy:
+      matrix:
+        java: [ 'openjdk11' ]
+        audit: [ 'kafka' ]
+      fail-fast: false
+    steps:
+    - name: Checkout source code
+      uses: actions/[email protected]
+    - name: Set up java
+      uses: joschi/[email protected]
+      with:
+        java-version: ${{ matrix.java }}
+    - name: Setup prerequisites
+      env:
+        WORKSPACE: ${{ github.workspace }}
+      run: bash build/audit/bin/setup-prerequisites.sh ${{matrix.audit}}
+    - name: Server Integration Tests - Audit
+      env:
+        WORKSPACE: ${{ github.workspace }}
+      shell: 'script -q -e -c "bash {0}"'
+      run: |
+        bash build/audit/bin/pre-integration-test.sh ${{matrix.audit}}
+        bash build/audit/bin/integration-test.sh ${{matrix.audit}}
+        bash build/audit/bin/post-integration-test.sh ${{matrix.audit}}
+    - name: Gather error logs
+      if: failure()
+      env:
+        WORKSPACE: ${{ github.workspace }}
+      run: bash build/audit/bin/gather-logs.sh ${{matrix.audit}}
+    - name: Upload logs
+      if: always()
+      uses: actions/[email protected]
+      with:
+        name: integration-test-results-${{ matrix.audit }}-${{ matrix.java }}
+        path: build/audit/integration-test-results

build/audit/.gitignore

@@ -0,0 +1,2 @@
+**/integration-test-results
+**/workarea

build/audit/README.md

@@ -0,0 +1,53 @@
+# End-to-End Automation Framework for FHIR Audit Integration Tests
+
+This document outlines the end-to-end audit automation framework. 
+
+The automation runs with these steps: 
+
+- **Checkout source code** - Checks out the git code and populates the `github` environment variables.
+- **Set up java** - Downloads and setup java based on the matrix values `matrix.java`.
+- **Setup prerequisites** - This step builds the required artifacts necessary to test the build with the audit. 
+- **Server Integration Tests - Additional audit Layers** - The step executes the pre-integration-docker, then integration-test-docker and runs the post-integration-docker scripts.
+- **Gather error logs** - This step only runs upon an error condition. 
+- **Upload logs** - The step uploads the results of the integration tests and the error condition logs are posted to the job. 
+
+The GitHub Action is parameterized with a matrix for each new audits. Each additional entry in the array ends up creating multiple automation steps which must complete successfully for the workflow. For instance, kafka is parameterized and executes a job for each audit layer value in the matrix (one java x one audit = 1 workflow job). 
+
+``` yaml
+    strategy:
+    matrix:
+        java: [ 'openjdk11' ]
+        audit: [ 'kafka' ]
+    fail-fast: false
+```
+
+Each audit layer that is tested as part of the framework uses the default build files and the files that match the audit name added to the `audit.yml`.
+
+|Filename|Purpose|
+|----------|----------------|
+|bin/gather-logs.sh|Gathers the logs from the build|
+|bin/integration-test.sh|Run after the tests complete to release resources and package tests results|
+|`<audit>`/integration-test.sh|Overrides bin/integration-test.sh, replacing the prior test behavior.|
+|bin/pre-integration-test.sh|Call the pre-integration-test step for `audit`|
+|`<audit>`/pre-integration-test.sh|Run before integration-test.sh to startup image and services for the integration testing|
+|bin/post-integration-test.sh|Call the post-integration-test step for `audit`|
+|`<audit>`/post-integration-test.sh|Run after integration-test.sh to stop image and services from the integration testing|
+|`<audit>`/Dockerfile|The Docker file used in development and end-to-end tests|
+|`<audit>`/docker-compose.yml|The Docker Compose file used with the end-to-end tests|
+|`<audit>`/resources| Stores files used to support the end-to-end tests. |
+|`<audit>`/README.md|Describes the relevant details for this audit layer.|
+|`<audit>`/.gitignore|Ignores files related to the audit layer's tests|
+|`README.md`|This file describing the framework|
+
+Note, `<audit>` is replaced with your audit layer such as `kafka`. 
+
+Consult the reference implementation (`kafka`) to start a new audit tests. The minimum that must be implemented are the `pre-integration-test.sh`, `post-integration-test.sh`, the `Dockerfile`, `.gitignore` and the `README.md`.
+
+Also, be sure to add an example `fhir-server-config-<audit-name>.json` configuration file to the `fhir-server` project as well. 
+
+## Test the Automation
+
+To test the build, be sure to pre-set `WORKSPACE` with `export WORKSPACE=$(pwd)`.
+You must also start Docker, so the image is built that supports the IBM FHIR Server.
+
+If you have any questions, please reach out on Zulip.

build/audit/bin/gather-logs.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+
+###############################################################################
+# (C) Copyright IBM Corp. 2020
+#
+# SPDX-License-Identifier: Apache-2.0
+###############################################################################
+set -ex
+
+# Gathers the logs
+package_logs(){
+    echo "Gathering logs for [${1}]"
+
+    it_results=${WORKSPACE}/build/audit/integration-test-results
+    if [ ! -d ${it_results} ]
+    then
+        rm -fr ${it_results} 2>/dev/null
+    fi
+
+    mkdir -p ${it_results}
+    mkdir -p ${it_results}/server-logs
+    mkdir -p ${it_results}/fhir-server-test
+
+    # Look for the FHIR Server Container
+    containerId=$(docker ps -a | grep ibm-fhir-server | cut -d ' ' -f 1)
+    if [[ -z "${containerId}" ]]; then
+        echo "Warning: Could not find fhir container!!!"
+    else
+        echo "fhir container id: $containerId"
+        # Grab the container's console log
+        docker logs $containerId  >& ${it_results}/docker-console.txt
+
+        echo "Gathering post-test server logs from docker container: $containerId"
+        docker cp -L $containerId:/logs ${it_results}/server-logs
+    fi
+
+    echo "Gathering integration test output"
+    cp -pr ${WORKSPACE}/fhir-server-test/target/surefire-reports/* ${it_results}/fhir-server-test
+
+    echo "Move the ${1} Elements too the output area'"
+    cp -pr ${WORKSPACE}/build/audit/${1}/workarea/* ${it_results}
+}
+
+###############################################################################
+# Check if the workspace is set.
+if [ -z "${WORKSPACE}" ]
+then 
+    echo "The WORKSPACE value is unset"
+    exit -1
+fi 
+
+# Store the current directory to reset to
+pushd $(pwd) > /dev/null
+
+package_logs "${1}"
+
+# Reset to Original Directory
+popd > /dev/null
+
+# EOF
+###############################################################################

build/audit/bin/integration-test.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+###############################################################################
+# (C) Copyright IBM Corp. 2020
+#
+# SPDX-License-Identifier: Apache-2.0
+###############################################################################
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+run_tests(){
+    # The integration tests may be overriden completely, or fall through to the default. 
+    AUDIT="${1}"
+
+    if [ ! -z "${AUDIT}" ] && [ -f "build/audit/${AUDIT}/integration-test.sh" ]
+    then 
+        echo "Running [${AUDIT}] specific integration tests"
+        bash build/audit/${AUDIT}/integration-test.sh
+    else 
+        # Go to the Default
+        echo "Executing the default integration tests"
+        mvn -B test -DskipTests=false -f fhir-server-test -DskipWebSocketTest=true --no-transfer-progress
+    fi
+}
+
+###############################################################################
+# Check if the workspace is set.
+if [ -z "${WORKSPACE}" ]
+then 
+    echo "The WORKSPACE value is unset"
+    exit -1
+fi 
+
+# Store the current directory to reset to
+pushd $(pwd) > /dev/null
+
+# Change to the AUDIT/bin directory
+cd "${WORKSPACE}"
+
+run_tests "${1}"
+
+# Reset to Original Directory
+popd > /dev/null
+
+# EOF
+###############################################################################

build/audit/bin/post-integration-test.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+###############################################################################
+# (C) Copyright IBM Corp. 2020
+#
+# SPDX-License-Identifier: Apache-2.0
+###############################################################################
+
+# audit_post - executes for each AUDIT post integration steps
+audit_post(){
+    AUDIT="${1}"
+    if [ ! -z "${AUDIT}" ] && [ -f build/audit/${AUDIT}/post-integration-test.sh ]
+    then 
+        echo "Running [${AUDIT}] post-integration-test"
+        bash build/audit/${AUDIT}/post-integration-test.sh
+    fi
+}
+
+###############################################################################
+# Check if the workspace is set.
+if [ -z "${WORKSPACE}" ]
+then 
+    echo "The WORKSPACE value is unset"
+    exit -1
+fi 
+
+# Store the current directory to reset to
+pushd $(pwd) > /dev/null
+
+cd "${WORKSPACE}"
+
+audit_post ${1}
+
+# Reset to Original Directory
+popd > /dev/null
+
+# EOF 
+###############################################################################

build/audit/bin/pre-integration-test.sh

@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+###############################################################################
+# (C) Copyright IBM Corp. 2020
+#
+# SPDX-License-Identifier: Apache-2.0
+###############################################################################
+
+# audit_pre - executes for each audit pre integration steps
+audit_pre(){
+    AUDIT="${1}"
+    if [ ! -z "${AUDIT}" ] && [ -f build/audit/${AUDIT}/pre-integration-test.sh ]
+    then 
+        echo "Running [${AUDIT}] pre-integration-test"
+        bash build/audit/${AUDIT}/pre-integration-test.sh
+    fi
+}
+
+###############################################################################
+# Check if the workspace is set.
+if [ -z "${WORKSPACE}" ]
+then 
+    echo "The WORKSPACE value is unset"
+    exit -1
+fi 
+
+# Store the current directory to reset to
+pushd $(pwd) > /dev/null
+
+# Change to the audit_pre
+cd "${WORKSPACE}"
+
+audit_pre "${1}"
+
+# Reset to Original Directory
+popd > /dev/null
+
+# EOF 
+###############################################################################

build/audit/bin/setup-prerequisites.sh

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+
+###############################################################################
+# (C) Copyright IBM Corp. 2020
+#
+# SPDX-License-Identifier: Apache-2.0
+###############################################################################
+set -ex
+
+# required_build - executes for every build
+required_build(){
+    # Clean up the packages and docker files not on the Mac
+    if [[ "$OSTYPE" != "darwin"* ]]
+    then
+        sudo apt clean
+        docker rmi $(docker image ls -aq)
+        df -h
+    fi
+
+    # build binaries
+    mvn -B install --file fhir-examples --no-transfer-progress
+    mvn -B install --file fhir-parent -DskipTests -P include-fhir-igs,integration --no-transfer-progress
+
+    # Build dockerfile
+    mvn -B dockerfile:build -f fhir-install --no-transfer-progress
+}
+
+# audit_build - executes for each audit type.
+audit_build(){
+    AUDIT="${1}"
+    if [ -f "build/audit/${AUDIT}/setup-prerequisites.sh" ]
+    then
+        echo "Running [${AUDIT}] setting setup prerequisites"
+        bash build/audit/${AUDIT}/setup-prerequisites.sh
+    fi
+}
+
+###############################################################################
+# Store the current directory to reset to
+pushd $(pwd) > /dev/null
+
+if [ -z "${WORKSPACE}" ]
+then 
+    echo "The WORKSPACE value is unset"
+    exit -1
+fi 
+
+# Change to the release directory
+cd "${WORKSPACE}"
+
+required_build
+audit_build "${1}"
+
+# Reset to Original Directory
+popd > /dev/null
+
+# EOF 
+###############################################################################

build/audit/kafka/.fhir.env

@@ -0,0 +1 @@
+# Environment values for FHIR

build/audit/kafka/.kafka-1.env

@@ -0,0 +1,11 @@
+KAFKA_BROKER_ID=1
+KAFKA_ZOOKEEPER_CONNECT=zookeeper-1:22181,zookeeper-2:32181
+KAFKA_ADVERTISED_LISTENERS=SSL://kafka-1:19092
+KAFKA_SSL_KEYSTORE_FILENAME=kafka.broker1.keystore.jks
+KAFKA_SSL_KEYSTORE_CREDENTIALS=broker1_keystore_creds
+KAFKA_SSL_KEY_CREDENTIALS=broker1_sslkey_creds
+KAFKA_SSL_TRUSTSTORE_FILENAME=kafka.broker1.truststore.jks
+KAFKA_SSL_TRUSTSTORE_CREDENTIALS=broker1_truststore_creds
+KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=" "
+KAFKA_SSL_CLIENT_AUTH=requested
+KAFKA_SECURITY_INTER_BROKER_PROTOCOL=SSL 

build/audit/kafka/.kafka-2.env

@@ -0,0 +1,11 @@
+KAFKA_BROKER_ID=2
+KAFKA_ZOOKEEPER_CONNECT=zookeeper-1:22181,zookeeper-2:32181
+KAFKA_ADVERTISED_LISTENERS=SSL://kafka-2:29092
+KAFKA_SSL_KEYSTORE_FILENAME=kafka.broker2.keystore.jks
+KAFKA_SSL_KEYSTORE_CREDENTIALS=broker2_keystore_creds
+KAFKA_SSL_KEY_CREDENTIALS=broker2_sslkey_creds
+KAFKA_SSL_TRUSTSTORE_FILENAME=kafka.broker2.truststore.jks
+KAFKA_SSL_TRUSTSTORE_CREDENTIALS=broker2_truststore_creds
+KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=" "
+KAFKA_SSL_CLIENT_AUTH=requested
+KAFKA_SECURITY_INTER_BROKER_PROTOCOL=SSL 

build/audit/kafka/.zookeeper-1.env

@@ -0,0 +1,6 @@
+ZOOKEEPER_SERVER_ID=1
+ZOOKEEPER_CLIENT_PORT=22181
+ZOOKEEPER_TICK_TIME=2000
+ZOOKEEPER_INIT_LIMIT=5
+ZOOKEEPER_SYNC_LIMIT=2
+ZOOKEEPER_SERVERS=zookeeper-1:22888:23888;zookeeper-2:32888:33888