Illumos 2605 want to resume interrupted zfs send #4213

kernelOfTruth · 2016-01-13T14:32:40Z

Reviewed by: George Wilson [email protected]
Reviewed by: Paul Dagnelie [email protected]
Reviewed by: Richard Elling [email protected]
Reviewed by: Xin Li [email protected]
Reviewed by: Arne Jansen [email protected]
Approved by: Dan McDonald [email protected]

References:
https://www.illumos.org/issues/2605
illumos/illumos-gate@9c3fd12

Porting notes:

The tests from Illumos in usr/src/test/zfs-tests were left out
Further the following files which don't exist in ZoL:
usr/src/cmd/truss/expound.c
usr/src/lib/libzfs/common/mapfile-vers
usr/src/lib/libzfs_core/common/mapfile-vers
usr/src/pkg/manifests/system-test-zfstest.mf

From a readability point of view the changes in [module/zfs/dmu_traverse.c]
from zfsonlinux@ecfb0b5 Fix misuse of input argument in traverse_visitbp
appear to be the optimum - also there's no functional change,
when going with upstream the compiler would likely complain about ISO C90 errors - so only switching
cdnp = buf->b_data;
to
child_dnp = buf->b_data;

diverged code base from Illumos:

[cmd/zstreamdump/zstreamdump.c]
zfsonlinux@37f8a88 Illumos 5746 - more checksumming in zfs send

[include/sys/zfs_ioctl.h]
zfsonlinux@0cee240 Speed up 'zfs list -t snapshot -o name -s name'

[module/zfs/zfs_ioctl.c]
zfsonlinux@13fe019 Illumos 3464 zfs synctask code needs restructuring

[module/zfs/dmu_send.c]
zfsonlinux@b58986e Use large stacks when available
zfsonlinux@044baf0 Use taskq for dump_bytes()
ISO C90 - mixed declarations and code: struct send_block_record *to_data;
struct receive_ign_obj_node *n;
account for 'missing braces around initializer' in 'struct send_thread_arg to_arg ='
ISO C90 - mixed declarations and code: void *payload = NULL;
nvlist_t *nvl = fnvlist_alloc();
uint64_t one = 1; uint64_t zero = 1;
char recvname[ZFS_MAXNAMELEN];
uint64_t val;
uint32_t payloadlen = drc->drc_drr_begin->drr_payloadlen;
size_t payload_len = 0;

[module/zfs/dsl_dataset.c]
Illumos 4185 add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R
hasn't landed yet, thus
' #include <sys/zio_checksum.h>
is still missing,

' #include <sys/dmu_send.h>
however is needed

further
replaced
fletcher_4_native(compressed, compressed_size, NULL, &cksum);
with
fletcher_4_native(compressed, compressed_size, &cksum);

ctx_template argument was removed (purpose: the template will be reused to make initialization more efficient)

C99 or C11 mode: declare 'int i' at the beginning of the function.

ISO C90 - mixed declarations and code: char buf[256];
zio_cksum_t cksum;
char *propval = kmem_asprintf("%u-%llx-%llx-%s",
char recvname[ZFS_MAXNAMELEN];
dsl_dataset_t *recv_ds;

[libzfs_sendrecv.c]
' #include <sha2.h>
isn't listed in the header list since sha2, sha256 is implemented differently in ZoL.

replace
fletcher_4_native(compressed, len, NULL, &cksum);
with
fletcher_4_native(compressed, len, &cksum);

declare 'int nread, i;' at the beginning of the function

zfsonlinux@b8864a2 Fix gcc cast warnings

[module/zcommon/zfs_prop.c]
zfsonlinux@11b9ec2 Add full SELinux support

[module/zfs/dmu_traverse.c]
zfsonlinux@ecfb0b5 Fix misuse of input argument in traverse_visitbp
zfsonlinux@79c76d5 Change KM_PUSHPAGE -> KM_SLEEP
zfsonlinux@a168788 Reduce stack for traverse_visitbp() recursion
zfsonlinux@b8d06fc Switch KM_SLEEP to KM_PUSHPAGE
zfsonlinux@47050a8 Fix stack traverse_impl()
ISO C90: place 'uint64_t obj, error;' at the beginning, further replace 'err' with 'error'

[man/man8/zfs.8]
Part of the sections for 'zfs receive' and 'zfs send' was rewritten and reordered to approximate upstream.

Ported-by: kernelOfTruth [email protected]

kernelOfTruth · 2016-01-13T14:33:10Z

There are 0 errors left,

the other ones are caused by ISO C90 warnings/mixed declaration

edit:

looks like the patch utility this time really messed up 😀

kernelOfTruth · 2016-01-13T15:29:40Z

/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dmu_send.c: In function ‘dmu_send_impl’:
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dmu_send.c:822:2: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
  void *payload = NULL;
  ^
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dmu_send.c:832:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
   nvlist_t *nvl = fnvlist_alloc();
   ^
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dmu_send.c: In function ‘dmu_recv_begin_sync’:
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dmu_send.c:1452:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
   uint64_t one = 1;
   ^
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dmu_send.c: In function ‘dmu_recv_resume_begin_check’:
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dmu_send.c:1522:2: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
  char recvname[ZFS_MAXNAMELEN];
  ^
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dmu_send.c:1545:2: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
  uint64_t val;
  ^
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dmu_send.c: In function ‘dmu_recv_stream’:
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dmu_send.c:2768:2: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
  uint32_t payloadlen = drc->drc_drr_begin->drr_payloadlen;
  ^
  CC [M]  /var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dnode.o
  CC [M]  /var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dnode_sync.o
  CC [M]  /var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dsl_dataset.o
  CC [M]  /var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dsl_deadlist.o
  CC [M]  /var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dsl_deleg.o
  CC [M]  /var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dsl_bookmark.o
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dsl_dataset.c: In function ‘get_receive_resume_stats’:
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dsl_dataset.c:1743:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
   char buf[256];
   ^
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dsl_dataset.c:1759:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
   zio_cksum_t cksum;
   ^
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dsl_dataset.c:1767:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
   char *propval = kmem_asprintf("%u-%llx-%llx-%s",
   ^
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dsl_dataset.c: In function ‘dsl_dataset_stats’:
/var/tmp/portage/sys-fs/zfs-kmod-9999-r1/work/zfs-kmod-9999/module/zfs/dsl_dataset.c:1865:3: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
   char recvname[ZFS_MAXNAMELEN];
   ^

kernelOfTruth · 2016-01-13T16:32:38Z

Fascinating that the declaration error for fletcher_4_native in lib/libzfs/libzfs_sendrecv.c wasn't detected on my local compilation ...

anyway - I'm out for now ...

kernelOfTruth · 2016-01-14T13:04:06Z

looks like some of the buildbots sorely need the latest master changes from
89666a8 Increase default user space stack size

edit:
rebased

Reviewed by: George Wilson <[email protected]> Reviewed by: Paul Dagnelie <[email protected]> Reviewed by: Richard Elling <[email protected]> Reviewed by: Xin Li <[email protected]> Reviewed by: Arne Jansen <[email protected]> Approved by: Dan McDonald <[email protected]> References: https://www.illumos.org/issues/2605 illumos/illumos-gate@9c3fd12 Porting notes: The tests from Illumos in usr/src/test/zfs-tests were left out Further the following files which don't exist in ZoL: usr/src/cmd/truss/expound.c usr/src/lib/libzfs/common/mapfile-vers usr/src/lib/libzfs_core/common/mapfile-vers usr/src/pkg/manifests/system-test-zfstest.mf From a readability point of view the changes in [module/zfs/dmu_traverse.c] from openzfs@ecfb0b5 Fix misuse of input argument in traverse_visitbp appear to be the optimum - also there's no functional change, when going with upstream the compiler would likely complain about ISO C90 errors - so only switching cdnp = buf->b_data; to child_dnp = buf->b_data; diverged code base from Illumos: [cmd/zstreamdump/zstreamdump.c] openzfs@37f8a88 Illumos 5746 - more checksumming in zfs send [include/sys/zfs_ioctl.h] openzfs@0cee240 Speed up 'zfs list -t snapshot -o name -s name' [module/zfs/zfs_ioctl.c] openzfs@13fe019 Illumos 3464 zfs synctask code needs restructuring [module/zfs/dmu_send.c] openzfs@b58986e Use large stacks when available openzfs@044baf0 Use taskq for dump_bytes() ISO C90 - mixed declarations and code: struct send_block_record *to_data; struct receive_ign_obj_node *n; account for 'missing braces around initializer' in 'struct send_thread_arg to_arg =' ISO C90 - mixed declarations and code: void *payload = NULL; nvlist_t *nvl = fnvlist_alloc(); uint64_t one = 1; uint64_t zero = 1; char recvname[ZFS_MAXNAMELEN]; uint64_t val; uint32_t payloadlen = drc->drc_drr_begin->drr_payloadlen; size_t payload_len = 0; [module/zfs/dsl_dataset.c] Illumos 4185 add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R hasn't landed yet, thus ' #include <sys/zio_checksum.h> is still missing, ' #include <sys/dmu_send.h> however is needed further replaced fletcher_4_native(compressed, compressed_size, NULL, &cksum); with fletcher_4_native(compressed, compressed_size, &cksum); ctx_template argument was removed (purpose: the template will be reused to make initialization more efficient) C99 or C11 mode: declare 'int i' at the beginning of the function. ISO C90 - mixed declarations and code: char buf[256]; zio_cksum_t cksum; char *propval = kmem_asprintf("%u-%llx-%llx-%s", char recvname[ZFS_MAXNAMELEN]; dsl_dataset_t *recv_ds; [libzfs_sendrecv.c] ' #include <sha2.h> isn't listed in the header list since sha2, sha256 is implemented differently in ZoL. replace fletcher_4_native(compressed, len, NULL, &cksum); with fletcher_4_native(compressed, len, &cksum); declare 'int nread, i;' at the beginning of the function openzfs@b8864a2 Fix gcc cast warnings [module/zcommon/zfs_prop.c] openzfs@11b9ec2 Add full SELinux support [module/zfs/dmu_traverse.c] openzfs@ecfb0b5 Fix misuse of input argument in traverse_visitbp openzfs@79c76d5 Change KM_PUSHPAGE -> KM_SLEEP openzfs@a168788 Reduce stack for traverse_visitbp() recursion openzfs@b8d06fc Switch KM_SLEEP to KM_PUSHPAGE openzfs@47050a8 Fix stack traverse_impl() ISO C90: place 'uint64_t obj, error;' at the beginning, further replace 'err' with 'error' [man/man8/zfs.8] Part of the sections for 'zfs receive' and 'zfs send' was rewritten and reordered to approximate upstream. Ported-by: kernelOfTruth [email protected]

Update arc_c under a mutex from openzfs#4197 ABD2 openzfs#3441 ABD: linear/scatter dual typed buffer for ARC (ver 2) 4950 openzfs#4207 Illumos openzfs#4950 files sometimes can't be removed from a full filesystem 2605 openzfs#4213 Illumos 2605 want to resume interrupted zfs send

kernelOfTruth · 2016-03-09T11:43:05Z

superseded by: #4396

kernelOfTruth mentioned this pull request Jan 13, 2016

[WIP] Illumos 2605 want to resume interrupted zfs send #4176

Closed

kernelOfTruth force-pushed the zfs_master_12.01.2016_2+2605 branch 4 times, most recently from d0da080 to 1211e35 Compare January 13, 2016 15:27

kernelOfTruth force-pushed the zfs_master_12.01.2016_2+2605 branch 2 times, most recently from cf4a5b5 to 9dc93e5 Compare January 13, 2016 16:28

kernelOfTruth force-pushed the zfs_master_12.01.2016_2+2605 branch 2 times, most recently from 7868ebd to d040536 Compare January 13, 2016 21:58

kernelOfTruth mentioned this pull request Jan 13, 2016

want resumable zfs send/recv #3896

Closed

kernelOfTruth force-pushed the zfs_master_12.01.2016_2+2605 branch from d040536 to fb0dafe Compare January 14, 2016 01:43

kernelOfTruth force-pushed the zfs_master_12.01.2016_2+2605 branch from fb0dafe to f3d2c1b Compare January 14, 2016 13:16

kernelOfTruth force-pushed the zfs_master_12.01.2016_2+2605 branch from f3d2c1b to 8d0c7dd Compare January 14, 2016 20:37

kernelOfTruth mentioned this pull request Jan 16, 2016

[dedicated box testing] ABD2 + Illumos #4950 + Illumos #2605 (master January 16th, 2015) #4235

Closed

kernelOfTruth mentioned this pull request Jan 17, 2016

[dedicated box testing] ABD2 + Illumos #4950 + Illumos #2605 (master January 16th, 2015) #4236

Closed

kernelOfTruth closed this Mar 9, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Illumos 2605 want to resume interrupted zfs send #4213

Illumos 2605 want to resume interrupted zfs send #4213

kernelOfTruth commented Jan 13, 2016

kernelOfTruth commented Jan 13, 2016

kernelOfTruth commented Jan 13, 2016

kernelOfTruth commented Jan 13, 2016

kernelOfTruth commented Jan 14, 2016

kernelOfTruth commented Mar 9, 2016

Illumos 2605 want to resume interrupted zfs send #4213

Illumos 2605 want to resume interrupted zfs send #4213

Conversation

kernelOfTruth commented Jan 13, 2016

kernelOfTruth commented Jan 13, 2016

kernelOfTruth commented Jan 13, 2016

kernelOfTruth commented Jan 13, 2016

kernelOfTruth commented Jan 14, 2016

kernelOfTruth commented Mar 9, 2016