feat: GitHub Codespaces support in LocalNet command group

[aorumbayev]

Spike following the https://github.com/orgs/algorandfoundation/projects/1/views/1?filterQuery=aorum&pane=issue&itemId=56476748

Proposed Changes

This document serves as an informal Architecture Decision Record (ADR). Proceedings from the PR discussion will be converted to an ADR markdown file later.

Motivation

A significant number of users attending bootcamps encountered scenarios where their devices had hardware incompatibilities with Docker (primarily certain Windows 10 variations). The proposal aims to provide a convenient built-in experience that leverages GitHub CLI capabilities to provision a dummy Codespace instance (using the Algokit base template as a target) to port forward localnet ports to the user's machine.

Proposal

The idea is to provide a new command under localnet group to cover this scenario for users experiencing issues with docker. We can rely on GitHub CLI to leverage codespaces and their port-forwarding capabilities, allowing the Algokit CLI to lazily load the dependency if not present. The current proof of concept assumes winget for Windows and uses a 'webi' executable on Unix devices.

On the first login, the user is redirected to GitHub CLI's interactive authentication flow. Afterward, any existing Codespaces with the 'sandbox' prefix are automatically deleted. Finally, a fresh Codespace is created, and after a 60-second sleep (a holistic estimate for base machine quota, roughly half for container boot, and the remaining quarters for pipx algokit install and starting the localnet), the indexer, algod, and kmd ports from the Codespace are redirected to the user's machine. The user can then open a new tab and interact with the localnet as if it were running locally (similar to the Docker-based experience).

Pros

• One of the simplest ways to solve the main requirement of providing alternative mechanisms for users to access localnet if Docker is not an option. It's important to maintain the Docker-based approach as the recommended option, given that Docker still offers advantages in offline local development scenarios and abstract away heaps of 'native' complexity. However, a simple, interactive experience within the localnet command group that provides a free quota-based temporary on-demand instance can offer a flexible alternative solution for any edge case where Docker is unavailable for local development.
• Embraces the notion of dev containers, which are actively propagated by official Algokit templates. Provides a marginally similar experience to using the Docker-based localnet.
• GitHub CLI is available on a massive array of architectures and operating systems, both as native binaries and via major package managers, allowing conditional handling of main variations by picking the appropriate distribution channel to obtain the artifact.

Cons and Minor Notes

• Cannot work offline.
• Similar to the Docker-based approach, for simplicity, we assume that only one Codespace-based localnet instance can be up and running (otherwise, default ports will clash). An advanced developer can always tailor the ports as needed, so it's more of an inconvenience for beginners.

Next steps

• - address comments, gather feedback on the proposal. Which parts of the initial poc can be further refined/improved?
• - if this proposal or a modified version based on final discussion in the pr are accepted, the pr implementation is to be refined, covered with snapshot tests and documented.

How to play around with the poc ?
pipx install git+https://github.com/algorandfoundation/algokit-cli@feat/codespaces-localnet --force then algokit localnet codespace start

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
src/algokit
__init__.py	15	7	53%	6–13, 17–24, 32–34
__main__.py	4	4	0%	1–7
src/algokit/cli
__init__.py	47	3	94%	31–34
codespace.py	50	9	82%	28, 114, 137, 150–155
completions.py	108	9	92%	63–64, 83, 93–99
dispenser.py	121	1	99%	77
doctor.py	53	3	94%	146–148
explore.py	50	12	76%	34–39, 41–46
generate.py	70	3	96%	76–77, 155
goal.py	47	1	98%	79
init.py	309	24	92%	493–494, 499–500, 503, 524, 527–529, 540, 544, 602, 628, 657, 690, 699–701, 704–709, 722, 741, 753–754
localnet.py	152	32	79%	65, 86–113, 133–137, 170, 182, 197–207, 220, 271, 292–293
task.py	34	3	91%	25–28
src/algokit/cli/common
utils.py	26	2	92%	120, 123
src/algokit/cli/project
bootstrap.py	32	1	97%	33
deploy.py	99	20	80%	47, 49, 101, 124, 146–148, 227, 234, 248–256, 259–268
link.py	89	16	82%	60, 65–66, 101–105, 115–120, 148–149, 218–219, 223
list.py	33	5	85%	21–23, 51–56
run.py	46	3	93%	38, 71, 160
src/algokit/cli/tasks
analyze.py	81	1	99%	81
assets.py	82	13	84%	65–66, 72, 74–75, 105, 119, 125–126, 132, 134, 136–137
ipfs.py	51	8	84%	52, 80, 92, 94–95, 105–107
mint.py	66	4	94%	48, 70, 91, 250
send_transaction.py	65	10	85%	52–53, 57, 89, 158, 170–174
sign_transaction.py	59	8	86%	21, 28–30, 71–72, 109, 123
transfer.py	39	3	92%	26, 90, 117
utils.py	99	45	55%	26–34, 40–43, 75–76, 100–101, 125–133, 152–162, 209, 258–259, 279–290, 297–299
vanity_address.py	56	10	82%	41, 45–48, 112, 114, 121–123
wallet.py	79	4	95%	21, 66, 136, 162
src/algokit/core
codespace.py	175	68	61%	34–37, 41–44, 48–71, 111–112, 125–133, 191, 200–202, 210, 216–217, 229–236, 251–298, 311–313, 338–344, 348, 395
conf.py	57	9	84%	12, 24, 28, 36, 38, 73–75, 80
dispenser.py	202	26	87%	91, 123–124, 141–149, 191–192, 198–200, 218–219, 259–260, 318, 332–334, 345–346, 356, 369, 384
doctor.py	65	7	89%	67–69, 92–94, 134
generate.py	48	3	94%	44, 81, 99
goal.py	65	4	94%	21, 36–37, 47
init.py	67	10	85%	53, 57–62, 70, 81, 88, 108–109
log_handlers.py	68	7	90%	50–51, 63, 112–116, 125
proc.py	45	1	98%	99
sandbox.py	253	23	91%	32, 84–87, 92, 96–98, 145, 192–199, 210, 502, 518, 543, 551
typed_client_generation.py	170	20	88%	62–64, 103–108, 132, 135–138, 156, 159–162, 229, 232–235
utils.py	146	38	74%	50–51, 57–66, 122–128, 152, 155, 161–174, 195–197, 226–229, 251
src/algokit/core/compilers
python.py	28	5	82%	19–20, 25, 49–50
src/algokit/core/config_commands
container_engine.py	41	21	49%	24, 29–31, 47–76
version_prompt.py	92	14	85%	37–38, 68, 87–90, 108, 118–125, 148
src/algokit/core/project
__init__.py	53	3	94%	50, 86, 145
bootstrap.py	120	8	93%	47, 126–127, 149, 176, 207–209
deploy.py	54	9	83%	61–64, 73–75, 79, 84
run.py	125	15	88%	83, 88, 97–98, 133–134, 138–139, 143, 147, 261–269, 284
src/algokit/core/tasks
analyze.py	93	3	97%	105–112, 187
ipfs.py	63	7	89%	58–64, 140, 144, 146, 152
nfd.py	49	13	73%	25, 31, 34–41, 70–72, 99–101
vanity_address.py	90	34	62%	49–50, 54, 59–75, 92–108, 128–131
wallet.py	71	5	93%	37, 129, 155–157
src/algokit/core/tasks/mint
mint.py	78	10	87%	123–133, 187
models.py	90	11	88%	50, 52, 57, 71–74, 85–88
TOTAL	4622	638	86%

Tests	Skipped	Failures	Errors	Time
487	0 💤	0 ❌	0 🔥	24.500s ⏱️

[ShoGinn]

@aorumbayev This is an epic feature; I've been trying to implement an offline container-agnostic approach since, as you stated, using Docker can be problematic.

My approach is similar to yours in that we need a local network engine. Thank you for not destroying the local network design. I'm adding this to my workboard, but this might be a better stopgap.

[ShoGinn]

@aorumbayev I could not get the pull to work without docker installed... I'm guessing its in the localnet group of the cli erroring out.

[neilcampbell]

@ShoGinn Nice find, you're spot on. The presence of docker is checked when the localnet command group is loaded. At the moment this PR is a quick spike to see what is possible, so a bit rough around the edges.

[aorumbayev]

@ShoGinn this is still a preview mvp, but i patched the issue you had with docker, feel free to retry again

[ShoGinn]

@ShoGinn this is still a preview mvp, but i patched the issue you had with docker, feel free to retry again

I actually tried it when I saw you made some commits. It works great, and I'm excited to see how it progresses.

This solution is much cleaner than local docker/podman and enables a solid dev environment.

I think the CLI is going to be the biggest work. My recommendation is that rather than having dappflow or other commands check for the engine, have it check for the URL: port, but that's just a recommendation.

[aorumbayev]

@neilcampbell @robdmoore @Loedn , ill still keep pr description as adr up until this pr is closed/merged/resolved.
For now assume this comment is the description (since initially it was a draft mvp, latest commits polish up the experience and add docs and tests)

Proposed changes

• algokit localnet codespace command - a single command that takes care of one full lifecycle from creation to termination -(user can opt out from auto cleanup with force flag) of a github codespace where upon localnet being booted inside the container - algokit cli will rely on gh cli to port forward the kmd algod and indexer to local network. See docs for detailed breakdown of behaviour and/or pipx install the branch to try it out yourself
• minor refinements, improving the start_with_animation spinner to gracefully handle keyboard interrupts.
• initial batch of snap shot tests (TODO: more to be added tomorrow)
• initial documentation

(cc @ShoGinn , feel free to check it out comment too, first hand user feedback is always welcome! :-) )

[ShoGinn]

@aorumbayev I did some tests with the latest code; the codespace will start without start (I'm not sure if that is intended). algokit localnet codespace I expected the cli to provide some instruction without start.

Also, because the cli is always looking for docker, it won't work for algokit localnet explore etc

I would recommend that localnet include codespace or docker for commands, dig deeper for things like logs, etc., and make the explore and console less restrictive, just naturally failing or checking ports rather than files.

[aorumbayev]

@ShoGinn hey thank you for checking it out!

RE: starting without start
This is indeed intended behaviour, that was changed in this commit. The reasoning was due to the fact that the command is mainly designed to work with on-demand codespaces that it creates (unless a -n name to a specific instance is provided in such cases it will try to just reconnect to an existing instance). There were no direct use cases for stop, reset and reset --update and having them integrated under docker based localnet command groups can put a lot more coupling with gh cli and we still want to maintain a fairly light set of features that cli wraps for the user (because gh cli by itself is very flexible and user friendly out of the box). The initial draft you tried was still an early mvp, so starting from last 2-3 commits you can expect a lot less breaking changes until this pr is finalized. One way to look at the new behaviour (dropping explicit codespace start) is similar to how kubectl behaves when it comes to port forwarding.

RE: explore
Good point! I will make sure it can work with both codespace and docker based modes. (will let you know once commit is in place)

RE: 'I would recommend that localnet include codespace or docker for commands'
I got the second half of this but regarding this specific bit, can you expand a little bit more? What are you referring to when you are mentioning localnet? A command group within cli or the group of containers representing the localnet ?

For extra reference while this pr is open you can access the new docs covering this command from here https://github.com/algorandfoundation/algokit-cli/blob/feat/codespaces-localnet/docs/features/localnet.md#github-codespaces-based-localnet

[ShoGinn]

@ShoGinn hey thank you for checking it out!

RE: starting without start This is indeed intended behaviour changes in this commit. The reasoning was due to the fact that the command is mainly designed to work with on-demand codespaces that it creates (unless a -n name to a specific instance is provided in such cases it will try to just reconnect to an existing instance). There were no use cases for stop, reset and reset --update and having them integrated under docker based localnet command groups can put a lot more coupling with gh cli and we still want to maintain a fairly light set of features that cli wraps for the user (because gh cli by itself is very flexible and user friendly out of the box). The initial draft you tried was still an early mvp, so i'd say you can expect a lot less breaking changes until this pr is finalized.

Makes sense!

RE: explore Good point! I will make sure it can work with both codespace and docker based modes. (will let you know once commit is in place)

Can't wait!

RE: 'I would recommend that localnet include codespace or docker for commands' I got the second half of this but regarding this specific bit, can you expand a little bit more? What are you referring to when you are mentioning localnet? A command group within cli or the group of containers representing the localnet ?

Sorry this was a fragmented thought! I just meant in the cli make commands specific to the docker portion a sub-command in the cli.

  logs       See the output of the Docker containers
  reset      Reset the AlgoKit LocalNet.
  start      Start the AlgoKit LocalNet.
  status     Check the status of the AlgoKit LocalNet.
  stop       Stop the AlgoKit LocalNet.

All of these commands should be under algokit localnet docker

that way when implemented you know what features are what.

algokit localnet
Usage: algokit localnet [OPTIONS] COMMAND [ARGS]...

Options:
  -h, --help  Show this message and exit.

Commands:
  codespace  Manage the AlgoKit LocalNet in GitHub Codespaces.
  docker     Manage the Algokit LocalNet in local Docker environment
  console    Run the Algorand goal CLI against the AlgoKit LocalNet via a Bash console so you can execute multiple
             goal commands and/or interact with a filesystem.
  explore    Explore the AlgoKit LocalNet using Dappflow

[aorumbayev]

@ShoGinn i see, i do agree that it does make the command group semantics a bit cleaner (given that this pr is now introducing an alternative mechanism to accessing the localnet).

1. Your option -> one way this could be done without breaking changes is applying the same process we did to algokit bootstrap and deploy commands at the root command level in latest v2 release. They are still there yet they are hidden from help and upon invoking you see a deprecation warning. However, that would still require an eventual change in majority of algokit repos to switch to new syntax (and any algokit based project relying on it). This is a bit of a broader suggestion, i wonder what are your opinions on this @robdmoore @Loedn @neilcampbell ?)

2. Alternative, moving the codespace under task command group. While having codespaces under localnet makes sense (and codespaces approach being part of a solution to a formal outcome related to localnet), we also have a separate command group category called tasks which we use exclusively for extra utils OR integrations with external tools (see algokit task analyze for instance). As an alternative we can also move this codespace command under algokit task command group. That wouldn't require a breaking change in a large majority of existing algokit repos and projects used to invoking localnet directly without extra docker command group. Curious on opinions for this option as well.

[neilcampbell]

Your option -> one way this could be done without breaking changes is applying the same process we did to algokit bootstrap and deploy commands at the root command level in latest v2 release. They are still there yet they are hidden from help and upon invoking you see a deprecation warning. However, that would still require an eventual change in majority of algokit repos to switch to new syntax (and any algokit based project relying on it). This is a bit of a broader suggestion, i wonder what are your opinions on this @robdmoore @Loedn @neilcampbell ?)

It's definitely an option. I do quite like that we are somewhat decoupled from Docker with the command semantics, as I guess it's still possible LocalNet could be an entirely native setup.

Alternative, moving the codespace under task command group. While having codespaces under localnet makes sense (and codespaces approach being part of a solution to a formal outcome related to localnet), we also have a separate command group category called tasks which we use exclusively for extra utils OR integrations with external tools (see algokit task analyze for instance). As an alternative we can also move this codespace command under algokit task command group. That wouldn't require a breaking change in a large majority of existing algokit repos and projects used to invoking localnet directly without extra docker command group. Curious on opinions for this option as well.

A potential issue with this approach is discoverability. It's probably not the first place someone will look to run a hosted LocalNet. What would the command be in this case algokit task codespace localnet? It does allow for the extensibility of adding other potential codespace features in the future.

I think there are pros and cons of the approaches. Let's chat to work them out and make a decision.

[neilcampbell]

I think we should put some info to justify why we run an interactive session. This should be a nice segue into some brief content a the free and linking to the GH docs to show the current free limits.

I think it would also be good to link to the docs for setting up spending limits, to ensure users have the info needed to ensure they don't get unexpected bills. I believe by default the spending limits are $0, however I think this is still good to include. https://docs.github.com/en/billing/managing-billing-for-github-codespaces/about-billing-for-github-codespaces#setting-a-spending-limit

[neilcampbell]

What was the outcome with regards to inactivity timeouts? Does port forwarding prevent the inactivity timeout from ever triggering.

[Loedn]

I would keep the localnet functionality outside of the task namespace, so that the end user can still type algokit localnet start, I would probably go down the avenue of having an algokit localnet config command where the end user decides which localnet to trigger by default with the options being: standard, custom, codespaces.
Thoughts?

[aorumbayev]

@ShoGinn sorry it took a while to get back to this since we were focused on other higher priority items.

Feel free to track the progress on this in the newly created github issue here, #504. You can assume the codespaces to be available on algokit cli as soon as that issue is closed.

[aorumbayev]

@neilcampbell @robdmoore podman is rather tricky to install on github actions, ubuntu registry has an outdated version and running on m1 macs require a lot of extra setup that will slow down the build. Should we address the ci bit for it separately? Otherwise ready for re-review:

1. Podman integration is in place so using algokit localnet config will allow user to use the preferred container registry
2. Further refined the compose integration

[aorumbayev]

Fyi @robdmoore @neilcampbell this for now is manually set on the repo but I mentioned to @Loedn that if there will be a github app available on foundation's org account we can use that later on to securely auth requests to trigger these checks from a single pipeline that we can place on the https://github.com/algorandfoundation/algokit repo (without the need to deal with PATs, but obv this would not be the only advantage, can simplify cross repo communication by a great extent, we can also properly use branch protection rules that way). @Loedn for the time being though this will be set as a manual cron only covering the cli repo, as soon as the bot is available for use to auth just let me know

[aorumbayev]

@neilcampbell switched to webi installer on windows too and the official installer recommends restarting the terminal session so ive matched the cli behaviour with that, if gh is not installed it will exit upon auto installation and user will need to re run the codespace command upon restart of the terminal (will update the docs on this tomorrow).
Other than that added extra idle timeout parameter, just port forwarding without doing anything extra is considered idle as well so user can set up up to 240 minutes of active state before codespace is shut down, each time there is any sort of interaction with the forwarded localnet the timer will be reset.

[neilcampbell]

I don't believe that powershell --version is available.

[neilcampbell]

I also get the following error when the webi installer runs. It still installs though. I'm running PowerShell 7.3.6.

Set-ExecutionPolicy : The 'Set-ExecutionPolicy' command was found in the module 'Microsoft.PowerShell.Security', but the module could not be loaded. For more information, run 'Import-Module

[aorumbayev]

@neilcampbell should be resolved now, powershell is checked via different command that is more versatile.

The Set-ExecutionPolicy on the other hand is a bit tricky to replicate, whats the output of Get-ExecutionPolicy -List on your side when you run it in powershell on windows (preferably same environment as the one where the error occurred during install).

[neilcampbell]

@aorumbayev Output is this:
Scope ExecutionPolicy

---

MachinePolicy Undefined
UserPolicy Undefined
Process Undefined
CurrentUser Undefined
LocalMachine RemoteSigned

[neilcampbell]

@aorumbayev Forgot to mention I was running algokit -v localnet codespace.
When running without -v it works fine without any error being printed, so we can ignore this.

[aorumbayev]

@ShoGinn codespaces support is out and available under this release version https://github.com/algorandfoundation/algokit-cli/releases/tag/v2.1.0