Built-in restore functionality

[lawndoc]

Currently, stack-back only provides backup functionality, and restores have to happen manually with the restic CLI. We will be adding the functionality to restore backups to stack-back in a future release.

This discussion is for external users and collaborators to provide input on what the restore functionality should look like in stack-back.

[faekiva]

Potentially it could just store a config file on disk, representing the current snapshot versions as seen in rcb snapshots, like

version: 0.0.1

restore_all: false
items:
  - path: /volumes
    id: 19928e1c
    restore: false
  - path: /databases/mysql/all_databases.sql
    id: 7a642f37
    restore: false

The container could update the ids automatically as the backups occurred. On container startup, if any of the individual restore keys or restore-all is set to true, the appropriate path(s) are restored, the keys setting to false as they're restored.

[lawndoc]

This is a bit more complex than I was expecting. I worry that keeping a config file in sync with the current state would introduce risk for something to go wrong. Permission issues could prevent the file from being written to, and you'd also have the recursive problem where the config file to restore the backup would also need to be backed up.

What do you think about just adding a single command like rcb restore abcd1234 that stops all containers, restores the specified snapshot, and starts the containers back up?

[faekiva]

I think it makes sense to have both available, shouldn't be too much extra effort to do the rcb command once the config version is done, particularly since we can pretty much just expose the restic restore command since it's all in one snapshot. The config format changes work for me, it'd be a "nice to have" to allow adding path specific restores via the flat file but since that could pretty easily be added in a non-breaking way I'm happy to put that as a potential future enhancement to keep the code size small.

In terms of the config, when I'm looking at the code for the application I'm not seeing any dependencies for the application itself at first glance, just the requirements.txt for the test. Since we'd be allowing users to modify the config, I think it'd be worth having a robust YAML library to do (de)serialization if we go for YAML as the config language. I'd personally pick PyYAML since it seems like the defacto standard, but I'm open to other thoughts.

We could also do it with JSON instead of YAML to avoid the library, but I think YAML's nicer, and theoretically the ability to use anchors and things could be useful for users down the line.

If we end up going the dependency route, do you have a preference as to how they're managed? I have a weak preference for using something like poetry or uv instead of managing venvs manually, but ultimately it's your call.

[faekiva]

I discovered requests and docker libraries in use and now I'm extra wondering how dependencies are being managed :)

[lawndoc]

I like your idea of eventually having the option to restore an individual volume/database. I'm actually new to restic, so I wasn't sure yet if there was a way to pick specific files or directories out of a snapshot. Either way, I do think we should focus on just having full snapshot restore functionality for now, since that will be the most common and stable way to restore a compose project. Let's add more granular restore capability in a future update.

I'm fine with adding PyYAML as a dependency, I like YAML too.

As far as where the Python dependencies are currently defined, docker is specified in setup.py, which actually pulls in requests as a dependency when restic-compose-backup is packaged at build time. If you have a compelling argument for changing how dependencies are managed, I'll hear you out. Otherwise I kind of feel like "it ain't broke so don't fix it" :)

[faekiva]

As far as where the Python dependencies are currently defined, docker is specified in setup.py, which actually pulls in requests as a dependency when restic-compose-backup is packaged at build time.

That makes sense! It might be worth adding requests as a dependency in setup.py anyways, just to protect from the application relying on a docker package implementation detail (ie: using requests) function.

If you have a compelling argument for changing how dependencies are managed, I'll hear you out. Otherwise I kind of feel like "it ain't broke so don't fix it" :)

Honestly it's largely a developer experience thing for me. For VS Code intellisense to work properly I needed to make a virtual env that had all of the dependencies in it, I think technically I could do it with setup.py but I wasn't able to find a great guide for doing it, since I don't want to have the actual cli installed in the venv, just the dependencies. I'm also personally running mypy to make sure I don't do anything stupid with a type mismatch, which means I have dev dependencies of types-docker and types-requests to make sure that I can easily track what the data that's being returned looks like.

It's also nice using uv specifically that for the virtual env I don't need to worry about installing the right python version that the application uses with a separate application, since it integrates with a .python-version file in the repo. (it'd also likely be worth pinning the python version even if we don't use that route, right now the Dockerfile is just using the latest version which means the code might break randomly).

It's also broadly recommended to use a pyproject.toml by python and using something like uv or poetry helps manage it.

Because of all of that, I have do have the relevant files for it in a basic version of it set up on my fork right now for my own use regardless, and I think it'd be useful to have it upstream and the build system updated to use the same tooling. That being said, I can definitely just rebase my changes after I make them onto a branch that doesn't have those things on it so it can be merged upstream, so it's not a huge deal if you decide to keep things as is! I know that my workflow is not everyone's workflow :)

[lawndoc]

to protect from the application relying on a docker package implementation detail (ie: using requests) function

I think you make a good point here. I'm fine with adding requests as a dependency explicitly.

It's also broadly recommended to use a pyproject.toml by python

This is already something I was considering updating, so this change is fine with me too. I also think pinning the Python version is a good idea.

For dependency management, I've tended to stick to the standard Python tooling (pip + venv). I think you're a more experience Python dev than I am, though, so if you think it's worth tweaking the repo to work better with mypy and uv then I trust you.

Long term, I'd like to have dependency management automated with GitHub Actions or Dependabot (with integration tests). As long as there isn't anything added that is going to make that challenging, I don't have a problem with it.

[lawndoc]

Just wanted to call out a few options in the restic restore command that might be relevant for the rcb restore functionality as defaults or passthrough arguments.

--overwrite if-changed will only restore files that haven't changed (based on file size + modification timestamp). --overwrite always is default in restic, but that seems a bit wasteful to me. It's probably kept to always for stability reasons, but IMO if-changed seems safe enough. Either way, we could pass through this argument from the rcb restore wrapper to restic restore to let the user choose this behavior.

--delete will remove files from the target directory that do not exist in the snapshot during restore. The default behavior in restic is to not delete extra files.

--dry-run --verbose=2 shows which files would be restored or deleted when using --overwrite if-changed or --delete

[lawndoc]

@faekiva I just looked at your fork and it looks like you already handle passthrough args. Disregard this message :)