OpenZFS send/recv patches

[behlendorf]

No description provided.

[behlendorf]

@kernelOfTruth can you please review and test. This is a rebased version of #4396 with all the functional/rsend tests enabled and passing with a couple exceptions. rsend_008_pos and rsend_009_pos are disabled, they also failing the automated OpenZFS testing not an issue with this patch.

@ahrens rsend_020_pos hits the ASSERT in the drr->drr_type != DRR_BEGIN case in module/zfs/libzfs_sendrecv.c:dump_record() However, when commenting out the ASSERT the test appears to pass. Do you know if this is a known issue? Does the OpenZFS automated testing run with all the ASSERTs enabled?

[ahrens]

rsend_020_pos hits the ASSERT in the drr->drr_type != DRR_BEGIN case in module/zfs/libzfs_sendrecv.c:dump_record() However, when commenting out the ASSERT the test appears to pass. Do you know if this is a known issue?

@behlendorf Assuming you are talking about:

        ASSERT(ZIO_CHECKSUM_IS_ZERO(&drr->drr_u.
            drr_checksum.drr_checksum));

I have not seen that fail. This could indicate a problem with zeroing out the record to begin with, or that the drr_checksum overlaps with fields in the other union members (which is what this assert is trying to check).

Does the OpenZFS automated testing run with all the ASSERTs enabled?

Yes.

[behlendorf]

@ahrens OK thanks, I'll run it down.

[behlendorf]

I've added commit 30ffc01 to this PR which addresses the last remaining issue. 30ffc01 adds a new ZFS_IOC_RECV_NEW ioctl, similar to ZFS_IOC_SEND_NEW, which is used for a resumable zfs receive. This way the zfs_cmd_t structure doesn't need to be modified as was done in the original patch which breaks user/kernel compatibility. We can not change these legacy interfaces.

@ahrens can you comment on if I should submit a version of this change upstream. You've already made the change to zfs_cmd_t so I wouldn't include that part. But it also adds a libzfs core function called lzc_receive_one() which zfs_receive_one() has been updated to use in addition to the new non-legacy ZFS_IOC_RECV_NEW ioctl. I could submit a patch which includes just those portions.

[ahrens]

@behlendorf It looks like the difference between lzc_receive_with_header in illumos and lzc_receive_one in this patch is the addition of the following arguments:

int cleanup_fd, uint64_t *read_bytes,
 +    uint64_t *errflags, uint64_t *action_handle, nvlist_t **errors

Is there a description of the new functionality of these new arguments? Assuming it's generally useful, I'd be happy to have that upstream. Also tagging @pcd1193182 for comment.

[behlendorf]

@ahrens great, I'll definitely make another pass to improve the documentation for this function. None of these additional arguments add new functionality they solely provide access to the four existing return values of ZFS_IOC_RECV. This way zfs_receive_one() can properly use libzfs_core and get out of the business of constructing its own zfs_cmd_t.

• "read_bytes" -> number of bytes read
  

• "error_flags" -> zprop_errflags_t
  

• "action_handle" -> handle for this guid/ds mapping
  

• "errors" -> error for each unapplied received property (nvlist)
  

Let me polish the patch a little bit more and I'll submit a version to OpenZFS.

[pcd1193182]

Seems a bit odd that this is the only section in the man page that wraps to 80 characters. Is that deliberate?

[behlendorf]

Will fix.

[dpquigl]

What can we do to get this PR merged. It is the base for being able to port over compressed send/recv so we can continue on our porting of ADB.

[behlendorf]

@dpquigl I had hoped to merge this on Friday but ran in to one last issue. The resumable send/recv tests fail when the new AVX2 optimized fletcher4 checksums are enabled. I was able to reliable reproduce this issue using the rsend_019_pos test case but haven't yet had the time to fully investigate. @dpquigl if either you or @jxiong have time and can get the to root cause this is ready to me merged.

[behlendorf]

@ironMann @jxiong can you please review commit 832fe89 from the stack. The fletcher 4 failures I was observing with the patch stack was caused by the updated send/recv code calling fletcher_4_native() with a buffer which was not a multiple of 8 bytes. Rather than attempt to handle this in any of the optimized implementation I've added a check to fallback to the scalar routines for this specific rare case.

[behlendorf]

Merged as:

5c27b29 Merge branch 'illumos-2605'
0dab2e8 Vectorized fletcher_4 must be 128-bit aligned
d1d19c7 OpenZFS 6876 - Stack corruption after importing a pool with a too-long n
eca7b76 OpenZFS 6314 - buffer overflow in dsl_dataset_name
43e52ed Implement zfs_ioc_recv_new() for OpenZFS 2605
8c62a0d OpenZFS 6562 - Refquota on receive doesn't account for overage
671c935 OpenZFS 4986 - receiving replication stream fails if any snapshot exceed
f8866f8 OpenZFS 6738 - zfs send stream padding needs documentation
b607405 OpenZFS 6536 - zfs send: want a way to disable setting of DRR_FLAG_FREER
e6d3a84 OpenZFS 6393 - zfs receive a full send as a clone
fd41e93 OpenZFS 6051 - lzc_receive: allow the caller to read the begin record
47dfff3 OpenZFS 2605, 6980, 6902