[Suggestion] More advanced backup / trimming options

[DanTheMan827]

I see that backup events have been added, but I have a couple suggestions to improve it further

• Only backup on interval if there has been or is an active player since the last interval backup (why backup when there's no way a change would've been made?)
• Have the option to trim event backups separately (to limit space usage in case someone logs in and out rapidly)
  • Further expanding on this scenario, if a backup is being made because of the last logout and another backup is in progress for the same reason, that should probably be aborted, the partial file deleted, and a new one made (maybe?)

Even though no one may have played on the server in a few days, backups are made like clockwork at the interval configured, this is fine for the most part but it's a waste of space to backup files that haven't changed.

[Kaiede]

The first item is something I can put on the backlog for sure. Good opportunity to hook into the ‘list’ command as well to ensure that we know how many players are logged in when the service starts (just in case the two containers are not started at the same time like it currently assumes).

The second is something the service handles by throttling the backups taken, rather than trimming after the fact. Trimming after the fact would be pretty wasteful of resources (disk and CPU in particular). This is the purpose of the minInterval setting. It ensures only one backup is made during that minimum interval from either interval or event-based backups, but it doesn’t affect dailies. Under that sort of model, I do the following on my own server:

schedule:
  daily: 1:30 # Ensure we get at least one snapshot before the nightly upload to B2
  minInterval: 3h # Ensure we get no more than 8 backups a day from events
  onPlayerLogout: true # Take opportunistic backups when players log out

Another example using interval:

schedule:
  interval: 12h # Make sure we have at least 2 backups a day when there isn’t much activity
  minInterval: 3h # Ensure we get no more than 8 backups a day by waiting 3h since the last backup to start a new one.
  onPlayerLogout: true # Take opportunistic backups when players log out

Does this not work for the scenario you envision? If there’s something missing, lets identify what it is so I can track that in a separate issue (to not overload this one).

if a backup is being made because of the last logout and another backup is in progress for the same reason, that should probably be aborted, the partial file deleted, and a new one made (maybe?)

Good catch, because the code now uses Swift Concurrency, I don’t have the benefit of serial queues anymore which would prevent multiple backups from running simultaneously. But there’s other mechanisms I can use to block that. I’ve opened #39 to track work to fix that.

That said, if a backup is already in progress, I believe it’s better to simply complete the old one and abort the new one. The server has already frozen the filesystem state at that point, so aborting, resuming the save, telling the server to flush again, is a rather messy way to ensure you capture the most recent snapshot. I’d much rather err on the side of simpler logic (for reliability) than try to get too clever here, if that makes sense.

[DanTheMan827]

I suppose that makes sense.

I was just thinking to grab the newest snapshot when the last user logs out even those snapshots may be made faster than the interval, but I suppose I may just be over cautious because I lost a world

Basically, every time the last person logs out, I would like a backup, but then again my world isn't multiple GB in size... yet... so I could see where more frequent backups could cause issues.

It's not a huge thing, just an idea.

[Kaiede]

Don’t get me wrong, I’m more trying to understand the need better in the hopes of identifying the right balance to strike, if the existing feature set is not full enough to describe what you want. I just want to strike that balance between simplicity and fullness that keeps the service reliable and reduces complexity and edge cases that need to be thought about in the logic that can make the service harder to maintain as a single developer with maybe a couple hours a week to devote to this.

The current model is based around the thinking of “How much play time would I be comfortable losing if I had to rollback?” And minInterval is a direct representation of that. But it sounds like what you want here is more “Use events to keep N very recent backups, use interval/dailies to have backstops in the case of world corruption in all the event backups”? Do I have that right?

I think both approaches have their own merit. The minInterval approach ensures that these extra backups are spread throughout the day, minimizing the odds that they are all corrupted due to a flurry of activity. The cost is that you always lose some multiple of minInterval of time in your rollback, but that can still be much lower than interval in many cases. The benefit of your suggestion if I understand it correctly is that you can minimize the amount of play time lost to the rollback, so long as you catch it early enough (i.e. corruption doesn’t affect all N event backups). Otherwise, you are having to rollback some multiple of interval in the bad case.

[DanTheMan827]

Well ideally I don't want it running backups too frequently when I'm actively playing because it lags the server, but then I would like to have it immediately do a backup when I logout so that my most recent changes get backed up

I'm not so concerned about CPU resources to back up the world on logout, but I also don't want dozens of backups made by the events.

I might have an interval backup every day if it also ran on logout of all players regardless of the minInterval setting and trimmed those event backups down to the most recent one separately from interval backups

Maybe I'm just that one user that has a desire for this, maybe not...

[Kaiede]

I might have an interval backup every day if it also ran on logout of all players regardless of the minInterval setting and trimmed those event backups down to the most recent one separately from interval backups.

This is enough to properly define the work item.

But I guess I’m still a little confused as to the why aspect of it. Why trimming vs throttling backups fired from player events, in your mind?

[Kaiede]

To spec something like this out, things that need to happen:

• Need some mechanism for knowing what the trigger of a backup was.
• Need the ability to split out the trimming buckets based on the trigger.
• Need a configuration option that splits out the trim buckets based on the trigger.

So the config file would have a new option:

trim:
  trimEventBackupsSeparately: true

And the easiest ways to track the trigger of a backup is either with a journal that is written out as JSON, or by modifying the file names like is done for the partial backups. Although the journal is probably the easier way to go, as it can be a simple [String:String] dictionary.

This assumes that all event backups during the last N days want to be kept (this can be dozens in some cases though). Having to do a live trim would be extra functionality. @DanTheMan827, is this sufficient for your needs? My concern here is that this would be pretty heavy on disk space if event backups don't get trimmed for a couple days.

[Kaiede]

Options for trimming event and interval backups separately

[Kaiede]

This has gotten a bit stale, and unfortunately I’m still not convinced that we need the ability to trim the two buckets separately to account for aggressive backup schedules, nor am I sure what problem it is actually solving.

I’ve opened #94 to describe the changes I’d like to make to 1.3.x to improve backup scheduling before moving onto the next larger piece of work, which will include the backup journal and some ways to have more frequent backups based on user activity. So for now I’m closing this one, but mostly to say that if we want to go over this again, we should probably do it fresh.