Add remote borg backup support

[timdiels]

This adds support for backing up directly to a remote borg repo. (I tried reaching out earlier)

Why?

I'd like to backup/restore via the UI, resist ransomware and store the backups remotely without storing a local (compressed) copy of the nextcloud data.

Alternatives (https://github.com/nextcloud/all-in-one?tab=readme-ov-file#are-remote-borg-backups-supported):

• Mount a network file system: Ransomware can encrypt the backups on the network storage.
• rclone: Ransomware and stores a local copy of the borg repo
• borg backup the local repo to borgbase: Stores a local compressed copy of the nextcloud data as at least 1 borg backup
• create your own solution: backups cannot be managed through the UI and others will have to reinvent my work

How to use it

Instead of entering a local repo path (which remains supported), you can choose to instead enter a remote borg repo url. (These screenshots are from a disaster recovery but it's analogous, just ignore the passphrase):

The first you try to initialise the repo, the backup container will create an ssh key and foolishly try to init the borg repo resulting in error because you have to authorise the ssh key it generated first. So, e.g. I copy paste the public key shown here to my borgbase repo:

Then I can try again as instructed and it should work. The following backup info is shown:

Changes

• Support for storing backups directly at a remote borg repo (Replace commands to not assume the repo is local, e.g. use BORG_REPO env var, ...)
• Exclude files from backups if we don't plan on restoring them anyway, such as the audit log
• Restore with borg extract because, at least with a remote repo, borg mount is very slow (20 seconds vs 24 minutes restore for pretty much an empty nextcloud install).

FYI Oddly the original code never restores host-mounts.

Testing

I hacked the code til I could run local deployments of nextcloud-aio without any builds and manually tested:

• Entering invalid stuff into forms I changed
• Remote borg repo init
• Creating and restoring a backup. I created some files in the master container volume; some in an ignored directory to make sure those are not deleted, some in a directory that is part of the backup, after the backup made some changes and made sure that once restored, the ignored files were left alone, new (non-ignored) files were deleted and changed files restored. I can't really open the user UI with my local test setup.
• Checking a backup
• Setting up a new nextcloud from backup

TODO

• I did at one point test local backups, but I should redo that.
• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great
• Update the readme, but I first want to hear your thoughts before putting more time in it

[szaimen]

Hi, first of all thank you for your contribution!

I've thought a bit about this the last week and came to the conclusion that I do not want to maintain this since it requires an additional server for me for testing (with a not so easy to reproduce setup) and has the potential of many users needing help getting this to work (and thus only a limited usecase). So I would still only test local backup also in the future.

However I would be fine with merging this if you @timdiels would step in as the maintainer of this feature. That means if bug reports or any questions regarding this feature come in, I would ask you for help on the topic. If that is fine for you, I would continue with the review.

FYI Oddly the original code never restores host-mounts.

Yes, this is expected and documented.

TODO

• I did at one point test local backups, but I should redo that.

Sounds good!

• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Yeah, we could create a dev instance from this if we proceed...

• Update the readme, but I first want to hear your thoughts before putting more time in it

Yes

[timdiels]

Hi,

However I would be fine with merging this if you @timdiels would step in as the maintainer of this feature. That means if bug reports or any questions regarding this feature come in, I would ask you for help on the topic. If that is fine for you, I would continue with the review.

I'm willing to maintain it, mostly on weekends, for as long as I'm a nextcloud user (which I have been for a couple of years so far).

• I did at one point test local backups, but I should redo that.

I will do it on the dev build after I've made the change you suggested.

• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Is there an easier way to develop than the hacks I did (in a different branch) for local testing? Not sure how you normally test AIO? If it's always via push to github, wait for build and deploy, I will probably keep my debug branch for later.

Note to self:

• Update readme
• Local repo should use borg mount and rsync
• Test local and remote on dev build

[szaimen]

I'm willing to maintain it, mostly on weekends, for as long as I'm a nextcloud user (which I have been for a couple of years so far).

Cool, then we can go ahead with this PR. I've invited you to the repo for easier collaboration :)

• I did at one point test local backups, but I should redo that.

I will do it on the dev build after I've made the change you suggested.

Great :)

• If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Is there an easier way to develop than the hacks I did (in a different branch) for local testing? Not sure how you normally test AIO? If it's always via push to github, wait for build and deploy, I will probably keep my debug branch for later.

Usually it is indeed always via push to github, wait for build and deploy. So probably it is easier for you to keep your debug branch 👍

[gregjohnsonsaltaire]

Very much appreciate the PR guys!
I'm happy to help test informally ... I have 3 nextcloud-aio sites of 20GB, 40GB & 350GB doing borg backups to Hetzner Storageboxes that I'm prepared to convert to the PR ... Regards

[timdiels]

I tried out local backups on my machine and made all the requested changes I believe. Could we get a a dev build going?

[szaimen]

@timdiels one last task is then to resolve the conflicts after handling the comments.

Afterwards I will create a final image from the branch for a last testing round.

[timdiels]

@szaimen I believe the only remaining comment was to keep the excludes in sync. I've merged main into it. Could you create the new image?

[szaimen]

I believe the only remaining comment was to keep the excludes in sync.

Yes

I've merged main into it. Could you create the new image?

I created new images 👍

[szaimen]

@timdiels just a few suggestions to not print out borg info to not spam the logs...

[timdiels]

I retested the remote backup, still works.

[szaimen]

LGTM. Thank you very much! 🥳🎉🚀

[szaimen]

This is now released with v10.0.0 Beta. Testing and feedback is welcome! See https://github.com/nextcloud/all-in-one#how-to-switch-the-channel

[KoMa1012]

Is there a way to back up to remote and local? Unfortunately I can't find any way to "reset" my old backup option (different path) somehow, even after following: 5593 (except step 5, since I'm already on 10.0.0).

Since I'm not sure how it's working currently I'll just give you how I did backups until now, once a day to local and once a week to remote. In general I never expect to use the remote only in natural disaster situations, and then I can accept loosing the last week. If this option isn't available, this could be a nice thing to add.