Skip to content

Commit 55652d4

Browse files
sahlbergclaude
andcommitted
Add --era: find changed data with dm-era instead of checksums
For keeping a copy that nobody else changes up to date (e.g. backups of VM images) when the source filesystem sits on a Linux dm-era device. Instead of reading both files and computing checksums, the generator sends an "era request" -- a sum head with no strong sums, followed by the destination file's user.rsync.era xattr ("<era> <time> <era-device-id>") -- and the sender looks up which blocks of the source were written in that era or later (era.c, using era_invalidate on a metadata snapshot). It answers with block references for unchanged blocks and literal data for the rest, so only changed source data is read and the receiver neither reads nor rewrites unchanged blocks. The whole-file checksum covers only the literal data, on both sides. With every file in --era mode the sender also sends the era the destination will be current with: the era started by the snapshot used for it, or for a normal transfer the newest logged era. The receiver removes the xattr before changing a file and sets it again once the checksum matches, so an interrupted file has no era and gets a normal update next time. Each file is therefore only re-sent the changes made since it was itself last updated. Each era device has a log of the eras our snapshots started and when (/var/lib/rsync/era/<device-id>, also the lock serializing runs); a file's era is only accepted if the log has it with the same time, so era numbers that start over after the era metadata is recreated cannot match. -X neither copies nor removes the xattr. A run takes at most 8 snapshots per device; after that a file reuses the bitmap of the newest older era, a superset. A file without an era, or whose era is from another device or not in the log, uses normal sums. If the sender cannot handle a requested file (not on dm-era, modified during the snapshot, ...) it declines with one byte; the receiver leaves the file untouched and asks for a redo, which uses the normal algorithm and never sends another era request. A missing era_invalidate on the sender, or a destination without user xattrs, is a hard error. --era is only built for standard Linux (not Android or other systems) and is needed on both sides. It implies --inplace and --no-whole-file, conflicts with --whole-file, --sparse, --append, --write-devices and batch mode, and requires protocol 30. Signed-off-by: Ronnie Sahlberg <ronniesahlberg@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent ea64520 commit 55652d4

12 files changed

Lines changed: 1243 additions & 20 deletions

File tree

‎Makefile.in‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ OBJS1_NO_MAIN=flist.o rsync.o generator.o receiver.o cleanup.o sender.o exclude.
4949
OBJS1=$(OBJS1_NO_MAIN) main.o
5050
OBJS2=options.o io.o compat.o hlink.o token.o uidlist.o socket.o hashtable.o \
5151
usage.o fileio.o batch.o clientname.o chmod.o acls.o xattrs.o
52-
OBJS3=progress.o pipe.o niceness.o @MD5_ASM@ @ROLL_SIMD@ @ROLL_ASM@
52+
OBJS3=progress.o pipe.o niceness.o era.o @MD5_ASM@ @ROLL_SIMD@ @ROLL_ASM@
5353
DAEMON_OBJ = params.o loadparm.o clientserver.o access.o connection.o authenticate.o
5454
popt_OBJS= popt/popt.o popt/poptconfig.o \
5555
popt/popthelp.o popt/poptparse.o popt/poptint.o

‎cleanup.c‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,11 @@ NORETURN void _exit_cleanup(int code, const char *file, int line)
140140
who_am_i(), code, src_file(file), line);
141141
}
142142

143+
#include "case_N.h"
144+
switch_step++;
145+
146+
era_cleanup(); /* drop a held dm-era metadata snapshot */
147+
143148
#include "case_N.h"
144149
switch_step++;
145150

‎compat.c‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ extern int use_qsort;
3232
extern int allow_inc_recurse;
3333
extern int preallocate_files;
3434
extern int append_mode;
35+
extern int era_mode;
3536
extern int fuzzy_basis;
3637
extern int read_batch;
3738
extern int write_batch;
@@ -679,6 +680,13 @@ void setup_protocol(int f_out,int f_in)
679680
protocol_version);
680681
exit_cleanup(RERR_PROTOCOL);
681682
}
683+
if (era_mode) {
684+
rprintf(FERROR,
685+
"--era requires protocol 30 or higher"
686+
" (negotiated %d).\n",
687+
protocol_version);
688+
exit_cleanup(RERR_PROTOCOL);
689+
}
682690
}
683691

684692
if (delete_mode && !(delete_before+delete_during+delete_after)) {

0 commit comments

Comments
 (0)