File size multiplying with zfs send | zfs receive

[jimsalterjrs]

System information

Distribution Name | Ubuntu
Distribution Version | 18.04
Linux Kernel | 4.15.0-29-generic #31-Ubuntu SMP
Architecture | x86_64
ZFS Version | 0.7.5-1ubuntu15
SPL Version | 0.7.5-1ubuntu1

Describe the problem you're observing

Source dataset on Ubuntu 14.04 / 0.6.5.11-1~trusty host is replicated to this host using zfs send | zfs receive. Apparent size of the single file this dataset contains (as observed with ls -lh, or du --apparent-size) does not change, but used size of file (as observed with zfs list, or du -h) roughly doubles on the target as compared to the host.

This appears to be a function of the source being ashift=12 and the target being ashift=13; replicating from the same source to another host with same versions of Ubuntu and ZFS, but an ashift=12 pool, does not result in the file size multiplication.

Scp or other non-ZFS-replication methods of copying the file from source to target also does not result in file size multiplication. It only happens with replication using zfs send | zfs receive.

Source and target both have compress=lz4; source is recordsize=16K and target is recordsize=64K.

Describe how to reproduce the problem

I'm not sure how to coherently reproduce the problem; other datasets on the same source replicated to the same host do not appear to suffer from the same issue. A 3.1T .qcow2 file in one dataset ended up being 6.2T on the target (and filling the pool), but a 580G .qcow2 file in another dataset was still only 580G when replicated to the target.

[jimsalterjrs]

The problem is also exhibited in a much smaller dataset, making it easier to work with:

root@hs0:/data/shadowcopies/brute# syncoid data/shadowcopies/brute root@new-prod0:data/shadowcopies/brute
INFO: Sending oldest full snapshot data/shadowcopies/brute@syncoid_hs0_2019-01-16:17:00:23 (~ 86.3 GB) to new target filesystem:
91.8GiB 0:11:10 [ 140MiB/s] [=================================] 106%            
INFO: Updating new target filesystem with incremental data/shadowcopies/brute@syncoid_hs0_2019-01-16:17:00:23 ... syncoid_hs0_2019-01-19:16:16:01 (~ 4 KB):
2.74KiB 0:00:00 [46.5KiB/s] [======================>           ] 68%         

root@hs0:/data/shadowcopies/brute# scp brute-shadowcopies.qcow2 root@new-prod0:/data/test/
brute-shadowcopies.qcow2                      100%   90GB 111.2MB/s   13:52

So, we've moved the single-file contents of this dataset from one host to another by both replication, and a simple scp. Both methods agree we transferred roughly 90GB of data. And du --apparent-size agrees that both methods resulted in 91G of data:

root@meca-prod0:/data/test# du -h --apparent-size /data/test
91G     /data/test
root@meca-prod0:/data/test# du -h --apparent-size /data/shadowcopies/brute
91G     /data/shadowcopies/brute

But when we drop the --apparent-size, we see the file size multiplication:

root@new-prod0:/data/test# du -h /data/test
69G     /data/test
root@new-prod0:/data/test# du -h /data/shadowcopies/brute
155G    /data/shadowcopies/brute

We also see it with zfs list:

root@meca-prod0:/data/test# zfs list | head -n 1 ; zfs list | egrep 'brute|test'
NAME                         USED  AVAIL  REFER  MOUNTPOINT
data/shadowcopies/brute      154G  6.22T   154G  /data/shadowcopies/brute
data/test                   68.5G  6.22T  68.5G  /data/test

[jimsalterjrs]

Note that snapshots are not the issue here; USED and REFER are identical. The only snapshot present on this dataset is the one created by syncoid to facilitate replication.

[jimsalterjrs]

Why does this occur with replication, but not with a straight copy - and why only on some datasets? There's another dataset on the same source that's 580G on both source and target. Sorry for opening a not-bug; this is coming as a surprise to me, I've replicated from ashift=12 to ashift=13 hosts many times and not seen this before.

[jimsalterjrs]

Example, from same source and target, also a dataset with a single .qcow2 file in it:

root@old-prod0:~# zfs list data/images/mail
NAME               USED  AVAIL  REFER  MOUNTPOINT
data/images/mail   568G   885G   560G  /data/images/mail

root@new-prod0:/data/test# zfs list data/images/mail
NAME               USED  AVAIL  REFER  MOUNTPOINT
data/images/mail   572G  6.22T   563G  /data/images/mail

[jimsalterjrs]

OK, thank you; respecting your desire to close here. =)