3.4.0 -H flag broken

[cschuber]

Consider the following rsync -aH:

bob# rsync -aH bob:/opt/src/git-doc .
ABORTING due to invalid path from sender: git-doc/.vale/vale.tmpl
rsync: connection unexpectedly closed (322182 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(232) [receiver=3.4.0]
rsync error: requested action not supported (code 4) at flist.c(2693) [generator=3.4.0]
rsync: [generator] write error: Broken pipe (32)
bob#

Now consider this:

bob# rsync -a bob:/opt/src/git-doc .
bob#

3.4.0 has broken preserve hard links.

[rosorio]

I was able to reproduce an issue with the -H flag using a single a single src in a directory

mkdir src
rsync -aH in bob:
Internal hashtable error: illegal key supplied!
rsync error: errors with program diagnostics (code 13) at hashtable.c(88) [generator=3.4.0]

Here is the debug log : outlog.txt

With in and in2 two regular directories

% rsync -a in  bob:
%
% rsync -aH in in2 localhost:/home/rodrigo/rsynctest/out
%

[ncopa]

I was able to reproduce an issue with the -H flag using a single a single src in a directory

Awesome! should be added to the testsuite

[ncopa]

I git bisected it to 688f5c3

ncopa-desktop:~/src/rsync ((no branch, bisect started on master))$ git bisect bad
688f5c379a433038bde36897a156d589be373a98 is the first bad commit
commit 688f5c379a433038bde36897a156d589be373a98 (HEAD)
Author: Wayne Davison <[email protected]>
Date:   Thu Nov 14 15:46:50 2024 -0800

    Refuse a duplicate dirlist.

 flist.c | 9 +++++++++
 rsync.h | 1 +
 2 files changed, 10 insertions(+)

[ncopa]

I added an abort() to get a backtrace of where it happens:

diff --git a/hashtable.c b/hashtable.c
index 2cc4e550..eb40ac4a 100644
--- a/hashtable.c
+++ b/hashtable.c
@@ -85,6 +85,7 @@ void *hashtable_find(struct hashtable *tbl, int64 key, void *data_when_new)
 
        if (key64 ? key == 0 : (int32)key == 0) {
                rprintf(FERROR, "Internal hashtable error: illegal key supplied!\n");
+               abort();
                exit_cleanup(RERR_MESSAGEIO);
        }
 

The backtrace:

Core was generated by `/home/ncopa/src/rsync/rsync -aH /home/ncopa/src/rsync/testtmp/preserve-symlink/'.
Program terminated with signal SIGABRT, Aborted.
#0  __restore_sigs (set=set@entry=0x7ffda4875690) at ./arch/x86_64/syscall_arch.h:40

warning: 40	./arch/x86_64/syscall_arch.h: No such file or directory
(gdb) bt
#0  __restore_sigs (set=set@entry=0x7ffda4875690) at ./arch/x86_64/syscall_arch.h:40
#1  0x00007fbe0f860e1b in raise (sig=sig@entry=6) at src/signal/raise.c:11
#2  0x00007fbe0f8449ad in abort () at src/exit/abort.c:11
#3  0x0000562c7eedfae8 in hashtable_find (tbl=0x7fbe0f6dd840, key=key@entry=0, 
    data_when_new=data_when_new@entry=0x562c7ef04b5e) at hashtable.c:88
#4  0x0000562c7eed9736 in match_gnums (ndx_list=0x7fbe0f6e81c0, ndx_count=<optimized out>) at hlink.c:126
#5  match_hard_links (flist=flist@entry=0x7fbe0f6fea10) at hlink.c:202
#6  0x0000562c7eec2768 in do_recv (f_in=<optimized out>, f_in@entry=0, f_out=f_out@entry=1, 
    local_name=local_name@entry=0x0) at main.c:1110
#7  0x0000562c7eec2eb6 in do_server_recv (f_in=0, f_out=1, argc=<optimized out>, argv=<optimized out>) at main.c:1227
#8  start_server (f_in=f_in@entry=0, f_out=f_out@entry=1, argc=<optimized out>, argv=<optimized out>) at main.c:1261
#9  0x0000562c7eec3025 in child_main (argc=<optimized out>, argv=<optimized out>) at main.c:1234
#10 0x0000562c7eee9b24 in local_child (argc=2, argv=argv@entry=0x7ffda4875ab0, f_in=f_in@entry=0x7ffda4875a10, 
    f_out=f_out@entry=0x7ffda4875a14, child_main=child_main@entry=0x562c7eec3010 <child_main>) at pipe.c:166
#11 0x0000562c7eea1780 in do_cmd (cmd=<optimized out>, machine=<optimized out>, user=0x0, remote_argv=<optimized out>, 
    remote_argc=<optimized out>, f_in_p=0x7ffda4875a10, f_out_p=0x7ffda4875a14) at main.c:646
#12 start_client (argc=<optimized out>, argv=0x7fbe0f7e7b30) at main.c:1577
#13 main (argc=<optimized out>, argv=<optimized out>) at main.c:1848
(gdb) 

[carnil]

#697 for the same issue.

[ncopa]

Alright, I found why this happens.

#define FLAG_HLINKED (1<<5)	/* receiver/generator (checked on all types) */
#define FLAG_GOT_DIR_FLIST (1<<5)/* sender/receiver/generator - dir_flist only */

When the FLAG_GOT_DIR_FLIST is set, it is later interpreted as FLAG_HLINKED being set, due to they use same bit. (number 5).

This fixes it:

diff --git a/rsync.h b/rsync.h
index 9be1297b..479ac484 100644
--- a/rsync.h
+++ b/rsync.h
@@ -84,7 +84,6 @@
 #define FLAG_DUPLICATE (1<<4)  /* sender */
 #define FLAG_MISSING_DIR (1<<4)        /* generator */
 #define FLAG_HLINKED (1<<5)    /* receiver/generator (checked on all types) */
-#define FLAG_GOT_DIR_FLIST (1<<5)/* sender/receiver/generator - dir_flist only */
 #define FLAG_HLINK_FIRST (1<<6)        /* receiver/generator (w/FLAG_HLINKED) */
 #define FLAG_IMPLIED_DIR (1<<6)        /* sender/receiver/generator (dirs only) */
 #define FLAG_HLINK_LAST (1<<7) /* receiver/generator */
@@ -93,6 +92,7 @@
 #define FLAG_SKIP_GROUP (1<<10)        /* receiver/generator */
 #define FLAG_TIME_FAILED (1<<11)/* generator */
 #define FLAG_MOD_NSEC (1<<12)  /* sender/receiver/generator */
+#define FLAG_GOT_DIR_FLIST (1<<13)/* sender/receiver/generator - dir_flist only */
 
 /* These flags are passed to functions but not stored. */
 

[rosorio]

Good job !
When I was looking at the code this was my first guess based on the commit you pointed.

[mikegilchrist]

Glad to learn I'm not the only one who ran into this issue. Is there a simple workaround for this?

[setharnold]

Are these bits sent over the wire? Or are these entirely internal to a given process (or family of processes on a single machine)?

Thanks

[cschuber]

Thank for the fix and thank you for the quick 3.4.1 release.

[kpinc]

Same issue as #697 (and #713).