Pre/Post setup Hurl scripts based on directory structure

[lambrospetrou]

Hey folks, while building Skybear.NET and expanding my own Hurl test suites for different APIs, I usually have some common steps to run before the actual test suite.

In traditional test frameworks based on code we have the before/beforeAll/after/afterAll hooks (or variations of these) to run some code pre/post the actual test.

I am thinking on providing the same functionality on the Skybear.NET platform using the directory structure of the input source files, and this can probably be part of the Hurl CLI tool itself in the future.

For example, assume the following files (output of the tree command):

├── _hurl_after
│   ├── 01-ensure-deleted-resources.hurl
│   └── 02-delete-user.hurl
├── _hurl_before
│   └── 01-auth-create-user.hurl
├── api1.hurl
└── api2.hurl

3 directories, 5 files

The above structure would lead to the following (imaginary) execution:

1. All files under _hurl_before in parallel (01-auth-create-user.hurl).
2. All files in current directory in parallel ({api1,api2}.hurl).
3. All files under _hurl_after in parallel (01-ensure-deleted-resources.hurl, 02-delete-user.hurl).

The same pattern can be applied in nested directories, with the following properties:

• The special _hurl_before, _hurl_after directories cannot have nested directories.
• The nested directories execute after their parent directories complete.

Parallel vs Sequential

An extra feature would be to make the before/after scripts to run in order based on alphabetical filename sorting, to allow dependent scripts to run in order.

The main test files would follow the existing execution mode whatever the user specified (based on --jobs, --test option, etc).

Just use Hurl on your own

It's a fair point that adding complexity in Hurl itself for this could be unjustified.

I already plan to build a tiny Go CLI wrapping the hurl ... invocations while tree-walking the filesystem and simulating the above behavior, but it would be nice having it in Hurl.

Open Question

The main issue I found with the above approach is that usually in the before/beforeAll steps you usually create a resource and then use that throughout the whole test suite.

This is NOT a problem if your resources can be queried by a deterministic name provided by the tests themselves, and the above approach would be awesome for those cases already. If the generation of resources leads to random IDs though we have the issue explained in this section.

As discussed in #3455 it is not currently possible natively with Hurl to pass information among files.

I can solve this on the SkybearNET platform easily by providing a local KV database/store that only lives throughout an execution and you can POST values to it from one file, and GET them later from a different file.

Another maybe cleaner option is a file-based approach where all the responses from the setup scripts are available under a specific endpoint so that followup scripts can GET that special URL and get the response they need, e.g. extracting the resource ID created during setup.

This would break the compatibility guarantee I have though for the platform since running those scripts locally would behave differently. I don't want and will not do that!

So, this is something else that would need to be integrated with Hurl itself or done in a non-breaking approach, but I am still thinking of a nice way to do it.

Feedback

What do others think about such functionality? Did you need it before? Would this be helpful?

[lambrospetrou]

A possible solution for the open question that could work natively with Hurl itself is to provide a function (since these are being introduced now easier) that can return the body/header of a previously persisted response that had explicit output: <somename> option defined.

This way, for the requests that we care about propagating information across files, we could put explicit output names, and then fetch those from other files, similar to uploading those output files, but using them with the upcoming Hurl expressions functionality.

Example:

// _hurl_before/create-user.hurl
POST /api/users
[Options]
output: create-user.json
HTTP 200

// api-test.hurl
GET /api/users/{{ userId }}
[Options]
userId: {{ getOutput "create-user-json" jsonpath "$.id" }}
HTTP 200

Syntax may vary, just an example of the interaction😅

---

Actually, this functionality could be useful on its own, regardless of the original feature described above, even in the existing Hurl execution mode where we can properly name our files to execute in specific order (when serially) and use outputs from previous files in our expressions.