Anatomy of an integration test #5082
Conversation
newhoggy
force-pushed
the
newhoggy/anatomy-of-an-integration-test
branch
from
fc1b58c to
0bcc2de
Compare
| import Testnet.Util.Process | ||
| import Testnet.Util.Runtime | ||
|
|
||
| hprop_leadershipSchedule :: Property |
There was a problem hiding this comment.
Integration tests use hedgehog. They are not property tests even thought the type says they are. We are merely using hedgehog to get nice a test failure report which includes annotated source code.
| import Testnet.Util.Runtime | ||
|
|
||
| hprop_leadershipSchedule :: Property | ||
| hprop_leadershipSchedule = H.integrationRetryWorkspace 2 "babbage-leadership-schedule" $ \tempAbsBasePath' -> do |
There was a problem hiding this comment.
This line describes how an integration test is introduced.
Integration tests start with a function beginning with the prefix integration.
"babbage-leadership-schedule" is the name of the integration test.
The Workspace suffix indicates that we want a workspace created for our integration test. A workspace is a temporary directory in which all the temporary files can reside. This includes configuration files, logs, socket files, etc. The location of this temporary directory will be bound to tempAbsBasePath'.
The Retry infix indicates that this integration test is flaky and may need to be retried in case of failure. The 2 indicates the integration test may be retried an additional 2 times.
If all the retries failed, you will see this:
forAll0 =
All 2 attempts failed
forAll288 =
Retry attempt 2 of 2
forAll564 =
Retry attempt 1 of 2
forAll859 =
Retry attempt 0 of 2
|
|
||
| hprop_leadershipSchedule :: Property | ||
| hprop_leadershipSchedule = H.integrationRetryWorkspace 2 "babbage-leadership-schedule" $ \tempAbsBasePath' -> do | ||
| H.note_ SYS.os |
There was a problem hiding this comment.
note_ adds a custom annotation to the failure report. Here is are annotating the OS. The annotation looks like this:
49 ┃ H.note_ SYS.os
┃ │ darwin
┃ │ darwin
┃ │ darwin
This is output three times because the test is retried twice.
Without the retry and in typical test failures it would look like this:
49 ┃ H.note_ SYS.os
┃ │ darwin
For rest of the comments, I have set the retry count to 0 to avoid illustrating with duplicate outputs.
| hprop_leadershipSchedule :: Property | ||
| hprop_leadershipSchedule = H.integrationRetryWorkspace 2 "babbage-leadership-schedule" $ \tempAbsBasePath' -> do | ||
| H.note_ SYS.os | ||
| base <- H.note =<< H.noteIO . IO.canonicalizePath =<< H.getProjectBase |
There was a problem hiding this comment.
getProjectBase gives you the directory of the Github repository.
We need this directory because we use some configuration files that are found in the repository.
There is a bug on this line. Both note and noteIO are used, so we get a duplication annotation.
50 ┃ base <- H.note =<< H.noteIO . IO.canonicalizePath =<< H.getProjectBase
┃ │ /Users/jky/wrk/iohk/cardano-node
┃ │ /Users/jky/wrk/iohk/cardano-node
The difference between note and note_ is that the note_ returns () whereas note returns the argument. ie. note_ === void . note.
The difference between note and noteIO is that the former takes a String argument and the latter takes an IO String argument.
One notable thing is that the hedgehog support functions are all exception-safe in the sense that if an exception is thrown, a useful annotation is applied to the failure report showing the exception in the context of the source code.
We do not get this exception safety if we run IO actions through liftIO. An exception in such cases will produce a failure report with no source code or annotations.
Exceptions from pure values can also be problematic.
For example, this will similarly produce an anaemic failure report:
let !x = error "exception from pure value"
Therefore it is always advisable to use the note family of functions for computed values, including pure ones.
| hprop_leadershipSchedule = H.integrationRetryWorkspace 2 "babbage-leadership-schedule" $ \tempAbsBasePath' -> do | ||
| H.note_ SYS.os | ||
| base <- H.note =<< H.noteIO . IO.canonicalizePath =<< H.getProjectBase | ||
| configurationTemplate <- H.noteShow $ base </> "configuration/defaults/byron-mainnet/configuration.yaml" |
There was a problem hiding this comment.
configuration/defaults/byron-mainnet/configuration.yaml is the configuration file we alluded to earlier that is found in the git repository.
51 ┃ configurationTemplate <- H.noteShow $ base </> "configuration/defaults/byron-mainnet/configuration.yaml"
┃ │ "/Users/jky/wrk/iohk/cardano-node/configuration/defaults/byron-mainnet/configuration.yaml"
We use noteShow here which is the same as note except that the argument can be any value that has a Show argument.
We could have actually just used note because the argument is a String.
| base <- H.note =<< H.noteIO . IO.canonicalizePath =<< H.getProjectBase | ||
| configurationTemplate <- H.noteShow $ base </> "configuration/defaults/byron-mainnet/configuration.yaml" | ||
| conf@Conf { tempBaseAbsPath, tempAbsPath } <- H.noteShowM $ | ||
| mkConf (ProjectBase base) (YamlFilePath configurationTemplate) tempAbsBasePath' Nothing |
There was a problem hiding this comment.
mkConf creates a conf value that is used to set up a testnet.
noteShowM is like noteShowIO except for any m with the following constraint: (MonadTest m, MonadCatch m)
52 ┃ conf@Conf { tempBaseAbsPath, tempAbsPath } <- H.noteShowM $
53 ┃ mkConf (ProjectBase base) (YamlFilePath configurationTemplate) tempAbsBasePath' Nothing
┃ │ Conf {tempAbsPath = "/private/tmp/nix-shell.0QhPpd/babbage-leadership-schedule-0-test-55125045e4f9b46e", tempRelPath = "babbage-leadership-schedule-0-test-55125045e4f9b46e", tempBaseAbsPath = "/private/tmp/nix-shell.0QhPpd", logDir = "/private/tmp/nix-shell.0QhPpd/babbage-leadership-schedule-0-test-55125045e4f9b46e/logs", base = "/Users/jky/wrk/iohk/cardano-node", socketDir = "babbage-leadership-schedule-0-test-55125045e4f9b46e/socket", configurationTemplate = "/Users/jky/wrk/iohk/cardano-node/configuration/defaults/byron-mainnet/configuration.yaml", testnetMagic = 1397}
| conf@Conf { tempBaseAbsPath, tempAbsPath } <- H.noteShowM $ | ||
| mkConf (ProjectBase base) (YamlFilePath configurationTemplate) tempAbsBasePath' Nothing | ||
|
|
||
| work <- H.note $ tempAbsPath </> "work" |
There was a problem hiding this comment.
It is customary to create a work directory under our workspace to contain any temporary files we create that are specific to our test. This will make it easier for testers to find them.
55 ┃ work <- H.note $ tempAbsPath </> "work"
┃ │ /private/tmp/nix-shell.0QhPpd/babbage-leadership-schedule-0-test-55125045e4f9b46e/work
| mkConf (ProjectBase base) (YamlFilePath configurationTemplate) tempAbsBasePath' Nothing | ||
|
|
||
| work <- H.note $ tempAbsPath </> "work" | ||
| H.createDirectoryIfMissing work |
There was a problem hiding this comment.
There are some convenience functions provided by hedgehog-extras. These functions correspond to the ones found in standard Haskell libraries.
createDirectoryIfMissing for example is the hedgehog-extras version of the one in System.Directory. The difference is that this one is "exception-safe" and also annotates the output.
createDirectoryIfMissing is implemented like this:
createDirectoryIfMissing :: (MonadTest m, MonadIO m, HasCallStack) => FilePath -> m ()
createDirectoryIfMissing filePath = GHC.withFrozenCallStack $ do
H.annotate $ "Creating directory if missing: " <> filePath
H.evalIO $ IO.createDirectoryIfMissing True filePath
annotate is the same as note. evalIO calls the IO action in an "exception-safe" manner. It is the same as noteIO except it doesn't annotate. noteIO is implemented in terms of evalIO.
The call to withFrozenCallStack ensures that annotations are attached to the caller, not here. For conveniences functions like these, it is advised to always do this because we care about the annotations in the context of the failing test, not the convenience function. To make this work we also need to use the HasCallStack constraint.
This is the output of that line:
56 ┃ H.createDirectoryIfMissing work
┃ │ Creating directory if missing: /private/tmp/nix-shell.0QhPpd/babbage-leadership-schedule-0-test-55125045e4f9b46e/work
| , poolNodes | ||
| -- , wallets | ||
| -- , delegators | ||
| } <- testnet testnetOptions conf |
There was a problem hiding this comment.
This is the code that starts the testnet. We have the opportunity to configure the testnet before we start it.
Once the testnet has completed started up, this function will return and give us a testnet runtime.
The testnet runtime gives us information about the testnet that has started. For example the location of configuration, what testnet magic was used. What the runnings are and what wallets have been created.
| -- , delegators | ||
| } <- testnet testnetOptions conf | ||
|
|
||
| poolNode1 <- H.headM poolNodes |
There was a problem hiding this comment.
Instead of using head we use headM. This is important because head is a partial function that can fail with an exception, which makes head not exception safe.
This line selects the first node in the list.
We will late connect to this node when running cardano-cli commands.
|
|
||
| poolNode1 <- H.headM poolNodes | ||
|
|
||
| env <- H.evalIO getEnvironment |
There was a problem hiding this comment.
We can use evalIO or eval for code that isn't exception safe. If you need to do this because hedgehog-extras doesn't export the convenience function you need, consider making a contribution to hedgehog-extras
|
|
||
| env <- H.evalIO getEnvironment | ||
|
|
||
| poolSprocket1 <- H.noteShow $ nodeSprocket $ poolRuntime poolNode1 |
There was a problem hiding this comment.
A sprocket is an abstraction over the relevant IPC:
- Socket for Linux and MacOS
- Named Pipe for Windows
Each of these have their on OS imposed naming restrictions. It is these restrictions that necessitate the abstraction to ensure we abide by them when pass them to the cardano-cli.
| -- successfully start that process. | ||
| <> env | ||
| , H.execConfigCwd = Last $ Just tempBaseAbsPath | ||
| } |
There was a problem hiding this comment.
We construct an execution config that describes how we want to run cardano-cli.
We want to current directory to be set to tempBaseAbsPath. This is because we will use a relative path to the node socket on POSIX systems due to some OSes having restrictions to the length of the socket filename. The use of relative path allows us to avoid hitting that restriction.
We also set the CARDANO_NODE_SOCKET_PATH environment variable for cardano-cli. sprocketArgumentName formats the name of the sprocket in a way that cardano-cli understands for all supported OSes.
| , H.execConfigCwd = Last $ Just tempBaseAbsPath | ||
| } | ||
|
|
||
| tipDeadline <- H.noteShowM $ DTC.addUTCTime 210 <$> H.noteShowIO DTC.getCurrentTime |
There was a problem hiding this comment.
We get the current time and add 210 seconds to it.
tipDeadline will serve as the deadline for a successful run of the query tip command which we will run later.
|
|
||
| tipDeadline <- H.noteShowM $ DTC.addUTCTime 210 <$> H.noteShowIO DTC.getCurrentTime | ||
|
|
||
| H.byDeadlineM 10 tipDeadline "Wait for two epochs" $ do |
There was a problem hiding this comment.
byDeadlineM is a combinator that allows us to run some code repeatedly until it either succeeds or the deadline expires.
10 is the poll time. The action will be invoked every 10 seconds.
We do this for these reasons:
- Although the testnet is "running", nodes in the testnet may not yet be accepting connections. Invocation of the
query tipcommand may fail because the node we are trying to connect to may not yet be ready. - We use the
query tipcommand to assert progress. In this case we require that the current epoch is greater than2. It make take some amount of time to get to this point. We poll every10seconds until this happens. - There must be a deadline because when we assert progress, that progress may never happen and we need to abort when progress is unlikely or else the test would run forever.
There was a problem hiding this comment.
The requirement for epoch greater than 2 is asserted later on (line 102)
| [ "query", "tip" | ||
| , "--testnet-magic", show @Int testnetMagic | ||
| , "--out-file", work </> "current-tip.json" | ||
| ] |
There was a problem hiding this comment.
We invoke the query tip command. Reminder that temporary and output files should be written to the work directory.
| , "--out-file", work </> "current-tip.json" | ||
| ] | ||
|
|
||
| tipJson <- H.leftFailM . H.readJsonFile $ work </> "current-tip.json" |
There was a problem hiding this comment.
Read the output JSON file to an aeson Value. Because decode of the JSON file can fail, this action returns an Either. leftFailM discards the Left in an exception safe way so tipJson as the type Value.
| ] | ||
|
|
||
| tipJson <- H.leftFailM . H.readJsonFile $ work </> "current-tip.json" | ||
| tip <- H.noteShowM $ H.jsonErrorFail $ J.fromJSON @QueryTipLocalStateOutput tipJson |
There was a problem hiding this comment.
We then decode the Value to the type we want: QueryTipLocalStateOutput.
| tip <- H.noteShowM $ H.jsonErrorFail $ J.fromJSON @QueryTipLocalStateOutput tipJson | ||
|
|
||
| currEpoch <- case mEpoch tip of | ||
| Nothing -> H.failMessage callStack "cardano-cli query tip returned Nothing for EpochNo" |
There was a problem hiding this comment.
If the value has no epoch, we fail with a user-friendly message.
| Just currEpoch -> return currEpoch | ||
|
|
||
| H.note_ $ "Current Epoch: " <> show currEpoch | ||
| H.assert $ currEpoch > 2 |
There was a problem hiding this comment.
assert can be used to make test assertions.
|
|
||
| let poolVrfSkey = poolNodeKeysVrfSkey $ poolKeys poolNode1 | ||
|
|
||
| id do |
There was a problem hiding this comment.
id do is how we introduce a smaller scope within a test for local bindings.
| , "--vrf-signing-key-file", poolVrfSkey | ||
| , "--out-file", scheduleFile | ||
| , "--current" | ||
| ] |
There was a problem hiding this comment.
Run the leadership-schedule command.
|
|
||
| expectedLeadershipSlotNumbers <- H.noteShowM $ fmap (fmap slotNumber) $ H.leftFail $ J.parseEither (J.parseJSON @[LeadershipSlot]) scheduleJson | ||
|
|
||
| maxSlotExpected <- H.noteShow $ maximum expectedLeadershipSlotNumbers |
There was a problem hiding this comment.
maximum is a partial pure function. This code is exception-safe because we use noteShow.
This PR is not meant to merged. It is used to illustrate some interesting aspects of integration tests.
The test under consideration is
cardano-testnet-tests:Spec.Babbage.leadership-schedule.The test is run like this:
A test report for a test failure using a manually injected failure is shown in this gist:
https://gist.github.com/newhoggy/16b9d3e0b5239cc19f6e0fc59044bb1c
To view the source code of the test with comments click through to this commit: 0bcc2de
Please do not resolve comments in this PR as they are for documentation purposes.