Automated IT Tests

Automated IT tests

Tests failed? Quick reference

1. run the tests locally on the main branch and check if they fail - if so follow the normal process of debugging/getting help
   1. start the sbt console with sbt
   2. change the project to support-workers with project support-workers
   3. run all the integration tests with it:test
   4. run a specific test file with it:testOnly *<nameOfFile> e.g. it:testOnly *ZuoraITSpec
2. (optional) check how long they have been failing for by checking it-test-failed and it-test-succeeded metrics here
3. check the logs at log group /aws/lambda/support-it-tests-PROD here
4. search for "TestFailed" around the time period they failed or scroll through the logs to see why they didn't run

Background

The IT tests are a set of scalatests run from SBT, which exercise the links between our code and the external DEV dependencies e.g. salesforce, zuora plus some non-prod AWS resources. We run the unit tests as part of a normal build, but running the IT tests is a little more tricky as they need access to the external systems. The goal of any individual development activity is to be aware of issues on as short a cycle as possible. The goal of automation is to provide a guarantee that can be relied on being true across the whole team.

Overview

The tests run after a PR has been merged, and any failures go to the whole team.

The IT tests are automated by

• deploying all the necessary test code and runner in to AWS as a lambda
• using CI/CD to keep the test code up to date
• giving the lambda the appropriate permissions to access what it needs
• running the lambda regularly on a schedule
• sending a cloudwatch alarm email to the team if there are issues

How does the Automated IT test runner work

See original PR https://github.com/guardian/support-frontend/pull/2623

• The support-workers teamcity build runs it:assembly on support-workers, this produces a massive jar including the tests themselves plus the prod code.
• When it runs riffraffUpload the IT jar is uploaded into S3 ready for deployment with the rest of support-workers.
• it-test-runner is built, uploaded and deployed as a separate project with cfn.
• a cloudwatch event rule triggers the lambda once an hour.
• the lambda downloads the jar with the tests, and runs the tests using scalatest Runner
• logging is sent to the Lambda logger which goes to cloudwatch group /aws/lambda/it-test-runner-PROD
• metrics are sent to cloudwatch for tests run, and tests failed
• if tests fail or stop running, an alarm is sent to the reader revenue team.

Developing the tests

You can deploy the stack to CODE if you are trying something out. The alarms won't be created, so there will be no alerting. The tests will run the CODE version of the IT tests, so be sure to deploy support-workers CODE (at least the tests jar) if necessary. Best to delete the stack afterwards to avoid them running hourly.

Limitations/Future work

• The tests run after merge and notify everyone, it would be more ideal to run on a branch build and notify on the PR as a required status, to avoid a window where the rest of the team may be affected.
• scalatest Runner is designed for command line use, so it calls System.exit meaning the lambda technically "fails"
• only support-workers IT tests are run at present
• tests must end with *Spec to be found due to the size of the jar
• if either the runner or the test jar are not updated, the tests may carry on running and passing.