feat: migrating localnet to latest indexer v3.x images

[aorumbayev]

Proposed Changes

• Adds support and migrates to -> official indexer v3 image (latest tag as of today)
• Adds required dependency on conduit which forces using extra algod follower container that does slow down execution of certain operations as well as boot up and shut down time of compose environment

Related improvements

• Adding pytest-xdist to speed up test execution

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/algokit
__init__.py	15	7	53%	6–13, 17–24, 32–34
__main__.py	2	2	0%	1–3
src/algokit/cli
completions.py	108	2	98%	83, 98
deploy.py	72	7	90%	44, 46, 89–91, 155, 179
dispenser.py	118	1	99%	75
doctor.py	48	3	94%	142–144
explore.py	50	12	76%	34–39, 41–46
generate.py	63	1	98%	118
goal.py	39	1	97%	57
init.py	190	16	92%	273–274, 324, 327–329, 340, 384, 410, 450, 459–461, 464–469, 482
localnet.py	93	3	97%	162, 183–184
src/algokit/cli/common
utils.py	26	2	92%	120, 123
src/algokit/cli/tasks
assets.py	82	13	84%	65–66, 72, 74–75, 105, 119, 125–126, 132, 134, 136–137
ipfs.py	54	11	80%	54, 84, 89–91, 96, 98–99, 109–111
mint.py	66	4	94%	48, 70, 91, 247
send_transaction.py	65	10	85%	52–53, 57, 89, 158, 170–174
sign_transaction.py	59	8	86%	21, 28–30, 71–72, 109, 123
transfer.py	37	3	92%	25, 89, 116
utils.py	99	45	55%	26–34, 40–43, 75–76, 100–101, 125–133, 152–162, 209, 258–259, 279–290, 297–299
vanity_address.py	56	10	82%	41, 45–48, 112, 114, 121–123
wallet.py	79	4	95%	21, 66, 136, 162
src/algokit/core
bootstrap.py	161	24	85%	103–104, 126, 149, 214, 217, 223–237, 246–251
conf.py	54	8	85%	10, 24, 28, 36, 38, 71–73
deploy.py	69	11	84%	61–64, 73–75, 79, 84, 91–93
dispenser.py	202	26	87%	91, 123–124, 141–149, 191–192, 198–200, 218–219, 259–260, 318, 332–334, 345–346, 356, 369, 384
doctor.py	65	7	89%	67–69, 92–94, 134
generate.py	41	2	95%	68, 86
goal.py	60	3	95%	30–31, 41
log_handlers.py	68	7	90%	50–51, 63, 112–116, 125
proc.py	45	1	98%	98
sandbox.py	199	16	92%	70, 116–123, 134, 429, 445, 460–462, 478
typed_client_generation.py	80	5	94%	55–57, 70, 75
utils.py	18	2	89%	27–28
version_prompt.py	73	8	89%	27–28, 40, 59–62, 80, 109
src/algokit/core/tasks
ipfs.py	57	4	93%	133, 137, 139, 145
nfd.py	49	13	73%	25, 31, 34–41, 70–72, 99–101
vanity_address.py	90	34	62%	49–50, 54, 59–75, 92–108, 128–131
wallet.py	71	5	93%	37, 129, 155–157
src/algokit/core/tasks/mint
mint.py	78	10	87%	123–133, 187
models.py	90	11	88%	50, 52, 57, 71–74, 85–88
TOTAL	3185	362	89%

Tests	Skipped	Failures	Errors	Time
382	0 💤	0 ❌	0 🔥	14.056s ⏱️

[aorumbayev]

@winder a few further questions to clarify if possible (cc @robdmoore @neilcampbell )

I was able to execute the algod container with 2 networks (relay participation and one follower nodes). It wasn't clear that network configurations are passed differently that config.json which is why i was confused initially. If you execute the compose on the contents of the attached archive you will see that it seems to be working fine. Conduit is able to chat with 8081 port on algod (which is the port for the follower, while relay is at 8080). There is a slight millisecond delay that still happens with this setup where fresh deployment of an app takes certain amount of time before it pops up on indexer search application by account endpoint response -> will require adding some tweaks to our fixtures on utils ts and py (cc @neilcampbell @robdmoore )

1. The configuration attached is generated from the compose file here - for the sake of simplicity for yourself to check it out. Despite setup working as expected (conduit being able to chat with second node within single algod container and it running with DEV_MODE env var), i am seeing the following error logs reported by follower:

2023-11-15 00:35:48 {"Context":"sync","file":"service.go","function":"github.com/algorand/go-algorand/catchup.(*Service).fetchAndWrite","level":"info","line":312,"msg":"fetchAndWrite(14): Aborted while waiting for lookback block to ledger after failing once : wsFetcherClient(127.0.0.1:44360).requestBlock(14): Request failed: requested block is not available","name":"0.0.0.0:10000","time":"2023-11-14T23:35:48.635482Z"}

Which i assume implies that follower node is unable to fetch the data from relay on port 10000 ? The port 10000 is set as a GOSSIP_PORT which i set similarly to your gist at https://gist.github.com/winder/8e167d94817370c8ed2e8579432f9003. Removing this value completely fails on algokit_algod | Error starting deployed network: open /algod/data/algod-listen.net: no such file or directory. What can be root cause for these errors or its just a warning that can be ignored in localnet scenarios (assuming that it seemingly doesn't affect anything)?

2. Do i need to specify GENESIS_ADDRESS and PEER_ADDRESS env vars on an algod container if both follower and relay are running in a single container? In the gist setup sample https://gist.github.com/winder/8e167d94817370c8ed2e8579432f9003 those are passed explicitly to the follower container but i am unsure whether i need to set them or not in the case of having follower inside first container. Perhaps this is also somehow causing the errors from question 1?
   Archive.zip

If you need any extra context/input please let me know.

[winder]

What behavior do you see that is undesirable?

The log you shared is an info level event, so not an error.

[aorumbayev]

@winder aside from compose working the only side effect observed was on our algokit utils unit tests in few places where we get to test app update calls we perform 2 consecutive deploys of a contract. In the case with current v2 indexer, changes are propagated almost instantly so its possible to perform lookup_account_application_by_creator right after app is created (expecting that it will find a freshly minted app id). In case of running against this compose there is still a slight millisecond delay which causes certain tests to fail (since they were assuming that lookup_account_application_by_creator would immediately return app id of a fresh app after txn is submitted and confirmed by algod). But i assume that with v3 indexer (and this compose setup) its expected behaviour given that there is an extra hop for data with extra follower and conduit as middleware. If you can confirm whether its reasonable to expect very short delays for data to propagate from algod to indexer in such localnet scenarios, even when follower node is within the same container then all good.

Other than that would still appreciate a reply for question 2. Just want to ensure i'm not passing any unnecessary environment variables or configurations that can cause unforeseen side effects.

After that will publish the pr and we will do more extensive smoke tests + will require changes in unit tests to account for increased delays on certain tests (that were previously not accounting for the fact that it may take extra time for indexer to sync with changes from algod). CC @robdmoore @neilcampbell