Skip to content

Commit 7998ccf

Browse files
receiver: add --delay-symlinks (#1124)
* Add --delay-symlinks to create symlinks after all files are in place The generator creates a symlink as soon as it reaches it in the file list, while regular files arrive later through the sender and receiver (and, with --delay-updates, are only renamed into place at the end). For the rest of the transfer a new symlink can point at a file that isn't there yet, so anything reading the destination during the sync -- e.g. a loader resolving libfoo.so -> libfoo.so.2 -- can fail. The new --delay-symlinks option makes the generator defer new and changed symlinks and create them once the receiver reports that every file, including any --delay-updates renames, is in place. Each link is itemized and logged after it has been created, before the generator ends the delay-updates phase, and the deletion stats are sent after any directory that a deferred link replaces. The option implies --no-inc-recursive on the receiving side and is passed to a remote receiver, which must also support it. It is an error to combine it with --remove-source-files, --hard-links, --compare-dest, --copy-dest, or --link-dest, or to use it with a protocol older than 29. support/rrsync accepts the option and refuses it under -no-overwrite, as it does --delay-updates. Add delay-symlinks_test.py, which holds a push partway through the link's target to check that new and changed links aren't created early, and rrsync-no-overwrite-delay-symlinks_test.py. * Document --delay-symlinks limits and test how it is passed to the server The man page now says the option narrows the inconsistency window rather than making the update atomic, that an interrupted transfer leaves new symlinks missing and changed ones at their old targets, and that --delete-before can also remove an old target early. Add delay-symlinks-remote_test.py. Through a remote shell that records the server command, it checks that the option is passed to a remote receiver on a push but not to a remote sender on a pull, and that incremental recursion is off in both directions. It also checks that --protocol=28 fails with the protocol error, and that a push to an older rsync (old_versions/rsync_3.4.1) is refused before anything is transferred while a pull from it works. * testsuite: seed delay-symlinks fixture with rsync * NEWS: document --delay-symlinks --------- Co-authored-by: Zen Dodd <mail@steadytao.com>
1 parent e6120a6 commit 7998ccf

10 files changed

Lines changed: 699 additions & 7 deletions

‎NEWS.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,8 @@
99
`B/s`. Example output includes `46.7M`, `48.8Mi`, `782,448B` and
1010
`113,295B/s`. File-list, created-directory and skipped-deletion counts also
1111
use thousands separators.
12+
- Added `--delay-symlinks` to defer creating new and changed symlinks until
13+
other transferred files are in place.
1214

1315
------------------------------------------------------------------------------
1416

‎compat.c‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ extern int fuzzy_basis;
3636
extern int read_batch;
3737
extern int write_batch;
3838
extern int delay_updates;
39+
extern int delay_symlinks;
3940
extern int checksum_seed;
4041
extern int basis_dir_cnt;
4142
extern int prune_empty_dirs;
@@ -173,7 +174,7 @@ void set_allow_inc_recurse(void)
173174
allow_inc_recurse = 0;
174175
else if (!am_sender
175176
&& (delete_before || delete_after
176-
|| delay_updates || prune_empty_dirs))
177+
|| delay_updates || delay_symlinks || prune_empty_dirs))
177178
allow_inc_recurse = 0;
178179
else if (am_server && strchr(client_info, 'i') == NULL)
179180
allow_inc_recurse = 0;
@@ -719,6 +720,14 @@ void setup_protocol(int f_out,int f_in)
719720
protocol_version);
720721
exit_cleanup(RERR_PROTOCOL);
721722
}
723+
724+
if (delay_symlinks) {
725+
rprintf(FERROR,
726+
"--delay-symlinks requires protocol 29 or higher"
727+
" (negotiated %d).\n",
728+
protocol_version);
729+
exit_cleanup(RERR_PROTOCOL);
730+
}
722731
} else if (protocol_version >= 30) {
723732
if (am_server) {
724733
compat_flags = allow_inc_recurse ? CF_INC_RECURSE : 0;

‎generator.c‎

Lines changed: 89 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ extern int msgdone_cnt;
5858
extern int ignore_errors;
5959
extern int remove_source_files;
6060
extern int delay_updates;
61+
extern int delay_symlinks;
6162
extern int update_only;
6263
extern int human_readable;
6364
extern int ignore_existing;
@@ -117,14 +118,48 @@ static int need_retouch_dir_times;
117118
static int need_retouch_dir_perms;
118119
static const char *solo_file = NULL;
119120

121+
/* With --delay-symlinks, new and changed symlinks are deferred here and created
122+
* only after the receiver has put every file into place (including any
123+
* --delay-updates renames). Each entry keeps what recv_generator() needs to
124+
* itemize and log the link once it has been created. */
125+
struct deferred_symlink {
126+
struct file_struct *file;
127+
int ndx;
128+
char *fname;
129+
int itemizing;
130+
enum logcode code;
131+
};
132+
static int defer_symlinks;
133+
static item_list deferred_symlinks = EMPTY_ITEM_LIST;
134+
135+
#ifdef SUPPORT_LINKS
136+
static void defer_symlink(struct file_struct *file, int ndx, const char *fname,
137+
int itemizing, enum logcode code)
138+
{
139+
struct deferred_symlink *ds
140+
= EXPAND_ITEM_LIST(&deferred_symlinks, struct deferred_symlink, 100);
141+
142+
ds->file = file;
143+
ds->ndx = ndx;
144+
if (!(ds->fname = strdup(fname)))
145+
out_of_memory("defer_symlink");
146+
ds->itemizing = itemizing;
147+
ds->code = code;
148+
}
149+
#endif
150+
120151
/* Forward declarations. */
121152
#ifdef SUPPORT_HARD_LINKS
122153
static void handle_skipped_hlink(struct file_struct *file, int itemizing,
123154
enum logcode code, int f_out);
124155
#endif
125156

126-
#define EARLY_DELAY_DONE_MSG() (!delay_updates)
127-
#define EARLY_DELETE_DONE_MSG() (!(delete_during == 2 || delete_after))
157+
/* Deferred symlinks are created and itemized in the delay-updates phase, so
158+
* that phase must not be ended early either, and the deletion stats must wait
159+
* for any directory that a deferred symlink replaces. */
160+
#define EARLY_DELAY_DONE_MSG() (!delay_updates && !defer_symlinks)
161+
#define EARLY_DELETE_DONE_MSG() \
162+
(!(delete_during == 2 || delete_after || defer_symlinks))
128163

129164
static int start_delete_delay_temp(void)
130165
{
@@ -2001,6 +2036,10 @@ static void recv_generator(char *fname, struct file_struct *file, int ndx,
20012036
fnamecmp = fnamecmpbuf;
20022037
}
20032038
}
2039+
if (defer_symlinks) {
2040+
defer_symlink(file, ndx, fname, itemizing, code);
2041+
goto cleanup;
2042+
}
20042043
if (atomic_create(file, fname, sl, NULL, MAKEDEV(0, 0), &sx, statret == 0 ? DEL_FOR_SYMLINK : 0)) {
20052044
set_file_attrs(fname, file, NULL, NULL, 0);
20062045
if (itemizing) {
@@ -2715,6 +2754,46 @@ void check_for_finished_files(int itemizing, enum logcode code, int check_redo)
27152754
}
27162755
}
27172756

2757+
/* Create the symlinks deferred by recv_generator(), then itemize and log each
2758+
* one as recv_generator() would have. We re-stat each destination because the
2759+
* transfer may have changed what's there. A link that can't be created is
2760+
* reported as an error and not itemized. */
2761+
static void create_deferred_symlinks(void)
2762+
{
2763+
#ifdef SUPPORT_LINKS
2764+
struct deferred_symlink *ds = deferred_symlinks.items;
2765+
size_t i;
2766+
2767+
for (i = 0; i < deferred_symlinks.count; i++) {
2768+
struct file_struct *file = ds[i].file;
2769+
/* atomic_create() can delete a directory in the way, which builds
2770+
* its entries' paths in this buffer, so it must be MAXPATHLEN. */
2771+
char fname[MAXPATHLEN];
2772+
stat_x sx;
2773+
int statret;
2774+
2775+
strlcpy(fname, ds[i].fname, sizeof fname);
2776+
free(ds[i].fname);
2777+
init_stat_x(&sx);
2778+
statret = gen_entry_stat(fname, file, &sx.st, 0);
2779+
if (atomic_create(file, fname, F_SYMLINK(file), NULL, MAKEDEV(0, 0),
2780+
&sx, statret == 0 ? DEL_FOR_SYMLINK : 0)) {
2781+
set_file_attrs(fname, file, NULL, NULL, 0);
2782+
if (ds[i].itemizing) {
2783+
if (statret == 0 && !S_ISLNK(sx.st.st_mode))
2784+
statret = -1;
2785+
itemize(fname, file, ds[i].ndx, statret, &sx,
2786+
ITEM_LOCAL_CHANGE|ITEM_REPORT_CHANGE, 0, NULL);
2787+
}
2788+
if (ds[i].code != FNONE && INFO_GTE(NAME, 1))
2789+
rprintf(ds[i].code, "%s -> %s\n", fname, F_SYMLINK(file));
2790+
}
2791+
free_stat_x(&sx);
2792+
}
2793+
deferred_symlinks.count = 0;
2794+
#endif
2795+
}
2796+
27182797
void generate_files(int f_out, const char *local_name)
27192798
{
27202799
int i, ndx, next_loopchk = 0;
@@ -2748,6 +2827,8 @@ void generate_files(int f_out, const char *local_name)
27482827
symlink_timeset_failed_flags = ITEM_REPORT_TIME
27492828
| (protocol_version >= 30 || !am_server ? ITEM_REPORT_TIMEFAIL : 0);
27502829
implied_dirs_are_missing = relative_paths && !implied_dirs && protocol_version < 30;
2830+
/* A dry run creates nothing, so its symlinks take the normal path. */
2831+
defer_symlinks = delay_symlinks && !dry_run;
27512832

27522833
if (DEBUG_GTE(GENR, 1))
27532834
rprintf(FINFO, "generator starting pid=%d\n", (int)getpid());
@@ -2881,6 +2962,12 @@ void generate_files(int f_out, const char *local_name)
28812962
wait_for_receiver();
28822963
}
28832964

2965+
/* The receiver has now put every file into place, including any
2966+
* --delay-updates renames, and the sender is still reading the itemized
2967+
* output, so the deferred symlinks can be created and reported. */
2968+
if (defer_symlinks)
2969+
create_deferred_symlinks();
2970+
28842971
if (protocol_version >= 29) {
28852972
phase++;
28862973
if (DEBUG_GTE(GENR, 1))

‎options.c‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,7 @@ int blocking_io = -1;
151151
int checksum_seed = 0;
152152
int inplace = 0;
153153
int delay_updates = 0;
154+
int delay_symlinks = 0;
154155
int32 block_size = 0;
155156
time_t stop_at_utime = 0;
156157
char *skip_compress = NULL;
@@ -790,6 +791,8 @@ static struct poptOption long_options[] = {
790791
{"partial-dir", 0, POPT_ARG_STRING, &partial_dir, 0, 0, 0 },
791792
{"delay-updates", 0, POPT_ARG_VAL, &delay_updates, 1, 0, 0 },
792793
{"no-delay-updates", 0, POPT_ARG_VAL, &delay_updates, 0, 0, 0 },
794+
{"delay-symlinks", 0, POPT_ARG_VAL, &delay_symlinks, 1, 0, 0 },
795+
{"no-delay-symlinks",0, POPT_ARG_VAL, &delay_symlinks, 0, 0, 0 },
793796
{"prune-empty-dirs",'m', POPT_ARG_VAL, &prune_empty_dirs, 1, 0, 0 },
794797
{"no-prune-empty-dirs",0,POPT_ARG_VAL, &prune_empty_dirs, 0, 0, 0 },
795798
{"no-m", 0, POPT_ARG_VAL, &prune_empty_dirs, 0, 0, 0 },
@@ -2292,6 +2295,22 @@ int parse_arguments(int *argc_p, const char ***argv_p)
22922295
remove_source_files == 1 ? "source" : "sent");
22932296
goto cleanup;
22942297
}
2298+
if (delay_symlinks && remove_source_files) {
2299+
snprintf(err_buf, sizeof err_buf,
2300+
"--delay-symlinks cannot be used with --remove-%s-files\n",
2301+
remove_source_files == 1 ? "source" : "sent");
2302+
goto cleanup;
2303+
}
2304+
if (delay_symlinks && preserve_hard_links) {
2305+
snprintf(err_buf, sizeof err_buf,
2306+
"--delay-symlinks cannot be used with --hard-links\n");
2307+
goto cleanup;
2308+
}
2309+
if (delay_symlinks && basis_dir_cnt) {
2310+
snprintf(err_buf, sizeof err_buf,
2311+
"--delay-symlinks cannot be used with %s\n", alt_dest_opt(0));
2312+
goto cleanup;
2313+
}
22952314
if (batch_name && strlen(batch_name) > MAX_BATCH_NAME_LEN) {
22962315
snprintf(err_buf, sizeof err_buf,
22972316
"the batch-file name must be %d characters or less.\n",
@@ -3069,6 +3088,9 @@ void server_options(char **args, int *argc_p)
30693088
} else if (keep_partial && am_sender)
30703089
args[ac++] = "--partial";
30713090

3091+
if (delay_symlinks && am_sender)
3092+
args[ac++] = "--delay-symlinks";
3093+
30723094
if (ignore_errors)
30733095
args[ac++] = "--ignore-errors";
30743096

‎rsync.1.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -621,6 +621,7 @@ has its own detailed description later in this manpage.
621621
--partial keep partially transferred files
622622
--partial-dir=DIR put a partially transferred file into DIR
623623
--delay-updates put all updated files into place at end
624+
--delay-symlinks create symlinks after all files are in place
624625
--prune-empty-dirs, -m prune empty directory chains from file-list
625626
--numeric-ids don't map uid/gid values by user/group name
626627
--usermap=STRING custom username mapping
@@ -1012,6 +1013,7 @@ sign) if you want the local shell to expand it.
10121013
- [`--delete-after`](#opt)
10131014
- [`--prune-empty-dirs`](#opt)
10141015
- [`--delay-updates`](#opt)
1016+
- [`--delay-symlinks`](#opt)
10151017

10161018
In order to be compatible with incremental recursion,
10171019
[`--delete-during`](#opt) is the default delete mode for [`--delete`](#opt).
@@ -3745,6 +3747,34 @@ sign) if you want the local shell to expand it.
37453747
update algorithm that is even closer to atomic (it uses [`--link-dest`](#opt)
37463748
and a parallel hierarchy of files).
37473749

3750+
0. `--delay-symlinks`
3751+
3752+
This option tells the receiving rsync to hold back the creation of new and
3753+
changed symlinks until every other file in the transfer has been put into
3754+
place, including the renames done by [`--delay-updates`](#opt). Without
3755+
it, a symlink is created as soon as rsync reaches it in the file list, so
3756+
for the rest of the transfer it can point at a file that hasn't arrived
3757+
yet.
3758+
3759+
This narrows the window in which the destination is inconsistent; it does
3760+
not make the update atomic. A transfer that is interrupted before the end
3761+
leaves new symlinks missing and changed symlinks pointing at their old
3762+
targets.
3763+
3764+
If a symlink is moved to a new target and the old target is deleted, also
3765+
use [`--delete-delay`](#opt) or [`--delete-after`](#opt), since
3766+
[`--delete-before`](#opt) and the default [`--delete-during`](#opt) can
3767+
remove the old target while the old symlink still points at it.
3768+
3769+
This option implies [`--no-inc-recursive`](#opt) since it needs the full
3770+
file list in memory in order to be able to iterate over it at the end.
3771+
3772+
Conflicts with [`--remove-source-files`](#opt), [`--hard-links`](#opt),
3773+
[`--compare-dest`](#opt), [`--copy-dest`](#opt), and [`--link-dest`](#opt).
3774+
This option is incompatible with rsync versions prior to 2.6.4 (March
3775+
2005), and when the receiving side is remote, the remote rsync must also
3776+
support this option.
3777+
37483778
0. `--prune-empty-dirs`, `-m`
37493779

37503780
This option tells the receiving rsync to get rid of empty directories from

‎support/rrsync‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ long_opts = {
5252
'copy-unsafe-links': 0,
5353
'daemon': -1,
5454
'debug': -1,
55+
'delay-symlinks': 0,
5556
'delay-updates': 0,
5657
'delete': 0,
5758
'delete-after': 0,
@@ -519,7 +520,11 @@ def main():
519520
# backup onto a name that already exists deletes what is there
520521
# (backup.c make_backup()), and deletion backs files up as well
521522
# (delete.c), so an unrelated --delete can land on a protected name.
523+
# --delay-symlinks creates its symlinks at the end of the transfer and
524+
# replaces whatever has appeared at that name since the
525+
# --ignore-existing check.
522526
long_opts['log-file'] = long_opts['partial-dir'] = long_opts['delay-updates'] = -1
527+
long_opts['delay-symlinks'] = -1
523528
long_opts['backup-dir'] = -1
524529
short_disabled += 'b' # must precede the short_no_arg_re build below
525530

‎support/rrsync.1.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -98,11 +98,13 @@ The remainder of this manpage is dedicated to using the rrsync script.
9898
9999
Because `--ignore-existing` protects only the file being transferred, this
100100
also refuses the options that can reach a *different* existing file in the
101-
restricted dir: `--log-file`, `--partial-dir`, `--delay-updates`, and
102-
backup mode (`-b`, `--backup-dir`, whose published backup replaces whatever
101+
restricted dir: `--log-file`, `--partial-dir`, `--delay-updates`,
102+
`--delay-symlinks` (which creates its symlinks at the end of the transfer,
103+
replacing whatever has appeared at that name since the check), and backup
104+
mode (`-b`, `--backup-dir`, whose published backup replaces whatever
103105
already occupies the backup name). Resumable uploads with an explicit
104-
`--partial-dir`, `--delay-updates`, and server-side logging are therefore
105-
unavailable under this option.
106+
`--partial-dir`, `--delay-updates`, `--delay-symlinks`, and server-side
107+
logging are therefore unavailable under this option.
106108
107109
0. `-help`, `-h`
108110

0 commit comments

Comments
 (0)