Can you run backup and forget --prune in parallel (simultaneously)

[sjiep]

I'm in the process of automating my backup workflow with systemd services and would like to know if running rustic backup and rustic forget --prune at the same time with two different services is problematic. As an example

• run rustic backup daily (00:00) with persistent=true (making sure it will run on the next boot if the computer was shutdown)
• run rustic forget --prune weekly (also 00:00), also with persistent=true

Now, on the day that rustic forget --prune runs, the rustic backup command runs simultaneously. (One could argue that I can make two different start times for the services, but I don't have fixed schedules for when my PC is switched on, so they could always potentially run at the same time on "next boot".

For this case where they run in parallel, I'm wondering if rustic handles this correctly. I tried to find the answer in the docs, but come to a bit of conflicting conclusion.

On the one hand, at this page it states that

Pruning snapshots can be a time-consuming process, depending on the number of snapshots and data to process. During a prune operation, the repository is locked and backups cannot be completed. Please plan your pruning so that there’s time to complete it and it doesn’t interfere with regular backup runs.

which makes me believe you shouldn't run them in parallel. On the other hand, in the FAQ # Are all operations lock free? it mentions the following which seems to indicate that they can.

Yes, all operations are designed lock-free. This means all commands can run parallel. This is especially true for multiple backup runs and backup runs parallel to prune. However, make sure that each individual backup run won’t take longer as the --keep-delete option (default: 23h) if you run prune parallel to backup - and that you don’t use --instant-delete.

Any insight on this situation would be greatly appreciated!

[lucaspar]

should be fine

This two-phase deletion is needed for rustic to work lock-free: If a backup runs parallel to a prune run (or forget --prune), it could be that prune decides that some blobs can be removed, but the parallel backup uses these blobs for the newly generated snapshot.

https://rustic.cli.rs/docs/FAQ.html?highlight=parallel#how-does-the-lock-free-prune-work

That said, I'd still just schedule the weekly prune for e.g. 00:30.

[sjiep]

Thanks for your answer @lucaspar.

I would agree, that the easiest would indeed be to just schedule them on different times. However, my pc is not switched on all the time, and the time that is is switched on is also not following a consistent pattern.

For this reason I wanted to rely on the systemd persistent=true option, so that it will run on the next boot which again will run them at the same time.

But you are right. The next FAQ section seems to indicate that it indeed should be fine. Having said that it still seems in conflict with what is mentioned in the intro of the "Removing backup snapshots" section, https://rustic.cli.rs/docs/commands/forget/intro.html

[lucaspar]

I see, I wasn't familiar with that systemd flag. Another possibility is to run the same custom script every day and put the prune branching in it, so they run sequentially once a week:

#!/usr/bin/env bash

set -euo pipefail

function is_saturday() {                                                                                                               
    [[ $(date +%a) = "Sat" ]]
}

function main() {

    rustic backup
  
    if is_saturday; then
        rustic forget --prune
    else
        echo "Skipping prune operation"
    fi

}

main "$@"

Naturally, this assumes you'll boot the machine every Saturday, but you can replace the logic with something more robust if that's not true.

[sjiep]

Thanks, that actually makes sense, I'll make a slightly more robust version of it 👍🏻

[aawsome]

Hi @sjiep!
I removed that wrong part in the docu mentioning that prune does lock the repository. It does not, and you can run backup parallel to forget --prune.
Besides this, there are reasons to try to prevent a parallel run and do exclusive pruning. If you want to do so, you need some external locking mechanism, e.g. within systemd.

[nickchomey]

Besides this, there are reasons to try to prevent a parallel run and do exclusive pruning. If you want to do so, you need some external locking mechanism, e.g. within systemd.

To clarify, are you saying

"no, prune doesn't lock the repo. But you still should try to avoid pruning and running a backup at the same time" ?

If so, why? What might happen if you do so? And what sort of external locking mechanism could you use (eg how would you use systemd here)?

[aawsome]

No I am saying that rustic is designed to run backup and prune in parallel by default.

Reasons to prevent this anyways could be:

• Trying to avoid stress by parallel operations on the backend
• Use of rustic in combination with restic on the same repository
• Difficulties to guarantee that a backup does not run longer than the --keep-delete option of prune (which is 23h by default)
• Two-phase pruning is not wanted, instead --instant-delete is used
• massive parallel runs (like multiple backup and prune runs) could run into edge cases which might be not as well tested as standard cases. In these edge cases the chance of undetected bugs is higher than bugs in more often used cases

If you run backup and prune on the same PC, you could use systemd e.g. to run a prune (or forget --prune) which depends on a finished backup. But there are several possibilities and I must say I am no systemd expert..

[sjiep]

Thanks for your clarification @aawsome!