send -Dc causes corruption when received

[implr]

System information

Type	Version/Name
Distribution Name	Gentoo
Linux Kernel	4.19.16-gentoo
Architecture	amd64
ZFS Version	v0.7.12-r1-gentoo
SPL Version	v0.7.12-r1-gentoo

Describe the problem you're observing

This is quite likely related to #7703, but in my case happens only when -c is also enabled and I should be able to share more information.
Transferring data between pools with send -RDc (and perhaps other options, doesn't affect the outcome) causes some files on the receiving side to be corrupted. The corruption looks like portions of the file were skipped or replaced with a few unrelated bytes.

Describe how to reproduce the problem

# dd if=/dev/zero of=/mnt/stuff/testpool bs=4k count=1500k
# zpool create -o altroot=/tmp/foo testpool /mnt/stuff/testpool
# zfs send -RDLecpv  zfs/backup/boston/ROOT@arg-190214-l | zfs recv -usFe testpool
full send of zfs/backup/boston/ROOT@dataset-reorg-1 estimated size is 40.6K
send from @dataset-reorg-1 to zfs/backup/boston/ROOT@dataset-reorg-2 estimated size is 624B
send from @dataset-reorg-2 to zfs/backup/boston/ROOT@arg-190214-l estimated size is 624B
full send of zfs/backup/boston/ROOT/gentoo@dataset-reorg-1 estimated size is 40.6K
send from @dataset-reorg-1 to zfs/backup/boston/ROOT/gentoo@dataset-reorg-2 estimated size is 3.91G
send from @dataset-reorg-2 to zfs/backup/boston/ROOT/gentoo@arg-190214-l estimated size is 1000M
total estimated size is 4.88G

Correct, source file:

# sha256sum /zfs/backup/boston/.zfs/snapshot/arg-190214-l/usr/include/postgresql-11/server/pg_config.h
4c9206b0c06d63f2365a883fe4b7c773d285ee0a449404f640b0999ecd9c77b6  /zfs/backup/boston/.zfs/snapshot/arg-190214-l/usr/include/postgresql-11/server/pg_config.h

Broken:

# sha256sum /tmp/foo/zfs/backup/boston/.zfs/snapshot/arg-190214-l/usr/include/postgresql-11/server/pg_config.h
42ba36cff44297667b1157a3c2fbfeed5baf3177281240ac2732d95b63b35d00  /tmp/foo/zfs/backup/boston/.zfs/snapshot/arg-190214-l/usr/include/postgresql-11/server/pg_config.h

# xxd /tmp/foo/zfs/backup/boston/.zfs/snapshot/arg-190214-l/usr/include/postgresql-11/server/pg_config.h | head -n 20
00000000: 0000 2d81 f81d 2f2a 2073 7263 2f69 6e63  ..-.../* src/inc
00000010: 6c75 6465 2f70 675f 636f 6e66 6967 2e68  lude/pg_config.h
00000020: 2e20 2047 656e 6572 6174 6564 2066 726f  .  Generated fro
00000030: 6d20 1d00 6269 6e20 6279 202c 009f 7572  m ..bin by ,..ur
00000040: 652e 2020 2a2f 0a4d 0008 2e69 6e50 0006  e.  */.M...inP..
00000050: 3e00 024e 00a2 6175 746f 6865 6164 6572  >..N..autoheader
00000060: 4f00 0050 00f0 0c44 6566 696e 6520 746f  O..P...Define to
00000070: 2074 6865 2074 7970 6520 6f66 2061 7267   the type of arg
00000080: 2031 0900 8027 6163 6365 7074 277e 0022   1...'accept'~."
00000090: 2364 2c00 ff05 4143 4345 5054 5f54 5950  #d,...ACCEPT_TYP
000000a0: 455f 4152 4731 2069 6e74 4c00 0c1f 324c  E_ARG1 intL...2L
000000b0: 0014 ff04 3220 7374 7275 6374 2073 6f63  ....2 struct soc
000000c0: 6b61 6464 7220 2a5a 000c 1f33 5a00 1411  kaddr *Z...3Z...
000000d0: 3353 004f 6c65 6e5f ac00 0165 7265 7475  3S.Olen_...eretu
000000e0: 726e ff00 0ff6 000d 6c52 4554 5552 4ef8  rn......lRETURN.
000000f0: 00f0 1e69 6620 6275 696c 6469 6e67 2075  ...if building u
00000100: 6e69 7665 7273 616c 2028 696e 7465 726e  niversal (intern
00000110: 616c 2068 656c 7065 7220 6d61 6372 6f29  al helper macro)
00000120: 5101 f013 2f2a 2023 756e 6465 6620 4143  Q.../* #undef AC
00000130: 5f41 5050 4c45 5f55 4e49 5645 5253 414c  _APPLE_UNIVERSAL

RDLecpv were the options I used when I first discovered the problem, but -RDc alone is sufficient. Removing either -c or -D fixes the problem - no corruption when received.
It seems like the corruption happens on send, and is not just a fault with the original pool:

zfs send -Rcv  zfs/backup/boston/ROOT@arg-190214-l | zfs recv -vusFe testpool
zfs send -RDcv testpool/ROOT@arg-190214-l  | zfs recv -usFe testpool2

testpool is not corrupted, but testpool2 is (exactly the same way as above)

I cannot share this dataset as is, since it is a complete rootfs backup, including SSL keys. I will try removing all the sensitive data, take another snapshot and see if sending that snapshot alone still triggers the problem. If it does, I can share that send stream.

Include any warning/errors/backtraces from the system logs

Nothing in dmesg.

[bunder2015]

This feature may be removed in a future release, see #7887

[ahrens]

The send --dedup flag has been removed by #10212. New releases (2.0+) will ignore the --dedup flag to zfs send, so you won't experience this problem. It's unfortunate that we didn't get to a root cause of this :(