Scenario testing - Restarts and upgrades #3269
Comments
|
@jamshale are there instructions for running the e2e tests? |
|
In You can use other pytest commands to run an individual test. |
|
ok thanks, I'll add this to the README |
|
Probably more useful to be able to test this with a back-end postgres database rather than having to worry about import/export for sqlite wallets (and then the tests could run in a separate docker container, and just point to the external database). |
|
I agree. For this scenario testing against a postgres db would be easier and better. The sqlite testing was much more difficult with restarts in acapy-tools. Also would be good to have postgres getting tested more here in general. |
|
@jamshale I'm getting errors running the tests. I ran I updated the project to use After running Any thoughts? Looks like a certificate error and then the |
|
(the above is in the (most of the tests fail) |
|
It doesn't clean up very well when there's a fail. Sometimes you need to manually delete the containers or the network. I'm not sure about the certificate error though. I'm not getting any and the tests are passing for me. You might be able to switch the genesis file. I'm just trying that now. |
|
What's running at |
|
The acapy-tools tests use the indicio indy test ledger. I guess that's where it's trying to register a did. I changed the scenario tests in acapy to use Not sure why your getting certificate problems though. |
|
Looks like all that is hard-coded into the unit tests. I'll update to use env vars (if available) and also it looks like some tests have a postgres dependency. Eventually would be nice to run the tests in docker (I always have to mess around when running locally to get all the proper versions, like python 3.10 which isn't a default on my local ...) |
|
Ya. These test in I got around the hardcoded stuff in the library for the ledger here https://github.com/openwallet-foundation/acapy/blob/main/scenarios/examples/presenting_revoked_credential/example.py#L50. |
|
I notice that the test container logic starts up a bunch of services within the python code. Typically in other projects we do this within a docker-compose file, and then pass parameters in (overrides) via environment variables. I'd like to convert this project to use a similar structure - docker compose and then build a docker image to run the unit tests. Before I dive into the code, does this make sense? It should make it easier to use these components within other repo's (like aca-py). |
|
Yes. That would be good. In the scenario tests in acapy we use docker compose files. It's really just about how to persist the storage containers for the length of the test and restart the agents with a different image and configs. The acapy-tools e2e tests do this so I thought we could use a similar design. |
I'll work on updating the acapy-tools tests to use docker and think about how to do this for the aca-py scenario tests ... |
|
The reason the tests start the container in python is because that seemed like the best way to control starting and stopping containers. I also generally prefer to keep it simple and just use docker compose wherever possible but, at the time, I was not finding good solutions that permitted both simple docker compose and controlling startup parameters and starting and stopping the containers. So docker compose got nixed. |
FYI managed to fix the certificate issue with this: https://stackoverflow.com/questions/52805115/certificate-verify-failed-unable-to-get-local-issuer-certificate |
|
@jamshale I think the scenario testing approach will have to change to support acapy restarts (especially if we are shutting down one acapy version and then starting up another). Right now the scenario test all start their agents via docker compose, so there is no mechanism for the test (which runs in one container) to restart the agent (which runs in a different container). (The test creates an instance of an One possibility is to change the Another possibility is to ditch the use of docker compose to start the agents and just manage all the agents within python code, similar to how the One wrinkle is the requirement to support different versions of acapy - i.e. startup an older version of acapy, run some scenarios, shut down the agent and then start up a new agent (with a newer acapy version) pointing to the same wallet. (Or running a wallet upgrade in between.) |
|
What if it was separate from the scenario tests? Like, it was a different set of tests called migration tests. Maybe it could be in the same poetry project but started as a different poetry script? I think the suite of tests runs the tests in the examples folder sequentially. If the migration tests were in another folder, it might be easier to have different setup/structure. |
I'll setup a separate test ... One option for the "acapy version" issue is to have two Controller containers based on two different acapy versions (for example the latest acapy release and the local acapy source code) and then starting an agent in one container, executing some protocols, shutting down the agent, and then starting a new agent from the other Controller container (pointing to the same wallet). The upgrade (if we're running a wallet upgrade) could be run directly from the example code. |
|
@jamshale @swcurran @dbluhm ... After a bit of beating my head against a wall I've come to the conclusion that the way we're testing the aca-py tools vs the aca-py scenario tests are basically incompatible:
The python docker library use by aca-py tools tests can't be run within a docker container (can't connect to docker on the host) so it can't be used within the scenario tests, the way they are currently structured. A couple of options:
Either of these would be fairly complicated so not sure it would be worth the effort, vs just using the aca-py tools repo to do the upgrade testing. Thoughts? |
|
Ok. That's a bit of a bummer. I didn't expect it to be this complicated. Maybe it would be easier to just have a test in acapy-tools that tests the upgrade to latest acapy with the scenarios we think are important. I still think it would be very useful for migration stuff like anoncreds upgrade and the upcoming DIDInfo work. We could have a github action that clones acapy-tools and runs the test on a release like we are doing with OATH right now. |
|
I'm not sure the python docker library will work under github actions (it needs a docker instance to talk to) |
|
Ok. Maybe we can leave this for now. I didn't intend for it to be a lot of work. Maybe we can think of another way to test these scenarios. |
|
It's possible to share the docker socket with a container to enable it to use docker on the host machine. This usually involves adding a volume to the container. A quick ChatGPT generated example: version: "3.8"
services:
docker-client:
image: docker:latest
container_name: docker_client
privileged: true
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./workspace:/workspace
environment:
- DOCKER_HOST=unix:///var/run/docker.sock
entrypoint: ["sh", "-c", "while true; do sleep 3600; done"]
# Example Usage:
# To run a Docker command inside the container:
# docker exec -it docker_client docker ps |
I ran a quick test and this worked! I'll put together a simple example and open a PR. |
|
I'm working on a simple test scenario with an agent restart (including updating the agent from the previous release to the latest code). The stop/start is working but the networking config on the new agent isn't working yet :-S I've created a draft PR in case anyone wants to take a look. Note that the new agent is started by docker directly rather than docker-compose, so docker-compose doesn't know anything about it. This will complicate the test cleanup a bit ... |
|
Explicitly creating a docker network might help with that. There's a lot of gotchas in docker networking though -- hopefully we can work through this one |
You can use indy-test-automation as a reference, it sets up a docker network in several places (GHA workflows and scripts) for the purpose of setup and testing of a network. Search for |
|
Turns out I didn't need to create a network just explicitly connect to the network docker compose already created ... The restart is now working (shut down alice aca-py I'll work on including an upgrade step in between the alice shutdown and restart. |
|
Good news (or bad news) - I've implemented a fairly robust anoncreds upgrade test and I'm now getting an error in the upgrade step. I'll dig into this but it looks like an aca-py issue. Alice has issued 6 credentials to 3 holders (2 each) and revoked 3 credentials (1 each). I set the revocation registry size to 5 so a new registry should have been created during this scenario. See PR #3410 |
|
OK. Thanks for testing this. I wasn't super confident about some of this more complicated revocation stuff. There's some subtle differences between revocation with anoncreds. Maybe we can keep the failing test as a separate branch, and I can try and look at it soon. |
|
Doing a very quick review of the aca-py upgrade code, I suspect the issue is with the record for the "used up" revocation registry (used for the first 5 credentials). |
|
Found the issue! |
|
This is awesome — nice work! Testing is really helpful! Having a good ways to test — amazing. |
In the acapy tools repo there is some really useful testing using docker containers that allows testing of upgrade scripts and restarts. https://github.com/hyperledger/aries-acapy-tools/tree/main/askar_tools/tests/e2e.
It should be possible to support this in the scenarios directory from acapy itself. It would be awesome to be able to test a supported upgrade path from the last LTS version and test a few different restarts with changed configurations such as making public dids and using seeds etc.
The text was updated successfully, but these errors were encountered: