[WIP] Illumos 2605 want to resume interrupted zfs send #4176
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reviewed by: Christopher Siden <[email protected]> Reviewed by: George Wilson <[email protected]> Reviewed by: Matthew Ahrens <[email protected]> Reviewed by: Bayard Bell <[email protected]> Reviewed by: Richard PALO <[email protected]> Reviewed by: Steven Hartland <[email protected]> Approved by: Rich Lowe <[email protected]> References: illumos/illumos-gate@3092556 https://www.illumos.org/issues/5745 Porting notes: fix the missing braces around initializer error: zfs_cmd_t zc = {"\0"}; account for the error "error: too many arguments for format [-Werror=format-extra-args]" in zfs_do_set Declare at the top: zfs_prop_t prop; nvpair_t *elem; nvpair_t *next; int i; Additionally initialize: int added_resv = 0; zfs_prop_t prop = 0; zc->zc_nvlist_conf = '\0'; zc->zc_nvlist_src = '\0'; zc->zc_nvlist_dst = '\0'; Ported-by: kernelOfTruth [email protected]
Reviewed by: Christopher Siden <[email protected]> Reviewed by: George Wilson <[email protected]> Reviewed by: Bayard Bell <[email protected]> Approved by: Albert Lee <[email protected]> References: illumos/illumos-gate@98110f0 https://www.illumos.org/issues/5746 openzfs#905 (Add "zstreamdump" integrity check program openzfs#905) openzfs@b79fc3f (Add zstreamdump(8) command to examine ZFS send streams.) Porting notes: Some of those changes were somewhat hard to track due to: openzfs@2024041 Remove superfluous statement openzfs@044baf0 Use taskq for dump_bytes() openzfs@88904bb Illumos 5162 - zfs recv should use loaned arc buffer to avoid copy kmem_alloc calls were changed to vmem_alloc in accordance with openzfs@77aef6f (Use vmem_alloc() for nvlists) and the kmem-rework account for ISO C90 warnings (-Werror=declaration-after-statement) account for error: format '%llx' expects argument of type 'long long unsigned int', but argument X has type 'uint64_t' [-Werror=format=] arc_buf_t *abuf; dmu_buf_t *bonus; zio_cksum_t cksum_orig; zio_cksum_t *cksump; Fix whitespace change (void) fprintf(stderr, "ERROR; failed to allocate %u bytes\n", (unsigned)size); to (void) fprintf(stderr, "ERROR; failed to allocate %zu bytes\n", size); and to account for long unsigned int & match upstream Ported-by: kernelOfTruth [email protected]
Illumos 5925 zfs receive -o origin= Reviewed by: Prakash Surya <[email protected]> Reviewed by: Matthew Ahrens <[email protected]> depends on openzfs#3574 Illumos 5745 zfs set allows only one dataset property to be set at a time depends on openzfs#3611 Illumos 5746 more checksumming in zfs send diverged code base from Illumos: [lib/libzfs/libzfs_sendrecv.c] b8864a2 Fix gcc cast warnings 325f023 Add linux kernel device support 5c3f61e Increase Linux pipe buffer size on 'zfs receive' [module/zfs/zfs_vnops.c] 3558fd7 Prototype/structure update for Linux c12e3a5 Restructure zfs_readdir() to fix regressions [module/zfs/zvol.c] function @zvol_map_block(spa_t *spa, zilog_t *zilog, const blkptr_t *bp, is non-existent in ZoL [module/zfs/dmu.c] in function dmu_prefetch(objset_t *os, uint64_t object, uint64_t offset, uint64_t len) int i is initialized before the following code block (c90 vs. c99) [module/zfs/Makefile.in + lib/libzpool/Makefile.am] 47a4a6f Support parallel build trees (VPATH builds) [module/zfs/dbuf.c] fc5bb51 Fix stack dbuf_hold_impl() 9b67f60 Illumos 4757, 4913 {4757 ZFS embedded-data block pointers ("zero block compression") , 4913 zfs release should not be subject to space checks} {reference} 34229a2 Reduce stack usage for recursive traverse_visitbp() [module/zfs/dmu_send.c] b58986e Use large stacks when available 241b541 Illumos 5959 - clean up per-dataset feature count code {reference} 77aef6f Use vmem_alloc() for nvlists 00b4602 Add linux kernel memory support [module/zfs/zvol.c] 9965059 Prefetch start and end of volumes [module/zfs/dmu_send.c, C90 warnings - previous commits, code thus less clear to read] Illumos 5746 more checksumming in zfs send [module/zfs/dbuf.c, ISO C90 - mixed declarations and code] arc_flags_t aflags = uint64_t nextblkid = dpa->dpa_zb.zb_blkid >> dmu_buf_impl_t *db = dbuf_find(dn->dn_objset, dn->dn_object, zio_t *pio = zio_root(dmu_objset_spa(dn->dn_objset), NULL, NULL, blkptr_t *bp = ((blkptr_t *)abuf->b_data) + dbuf_prefetch_arg_t *dpa = kmem_zalloc(sizeof (*dpa), KM_SLEEP); dsl_dataset_t *ds = dn->dn_objset->os_dsl_dataset; [module/zfs/dmu_send.c, ISO C90 - mixed declarations and code] dnode_phys_t *blk = abuf->b_data; uint64_t dnobj = zb->zb_blkid * (blksz >> DNODE_SHIFT); error: ‘for’ loop initial declarations are only allowed in C99 or C11 mode: for (struct receive_ign_obj_node *n = struct send_block_record *to_data; struct receive_ign_obj_node *n; struct receive_ign_obj_node *last_object; uint64_t last_objnum = (last_object != NULL ? FIXME: man/man8/zfs.8 FIXME: different manpage format, currently I don't "get it" yet Ported-by: kernelOfTruth [email protected]
Reviewed by: Adam Leventhal <[email protected]> Reviewed by: George Wilson <[email protected]> Reviewed by: Paul Dagnelie <[email protected]> Approved by: Gordon Ross <[email protected]> diverged code base from Illumos: [module/zfs/dbuf.c] 5f6d0b6 Handle block pointers with a corrupt logical size [module/zfs/dmu_zfetch.c] c65aa5b Fix gcc missing parenthesis warnings 428870f Update core ZFS code from build 121 to build 141. 79c76d5 Change KM_PUSHPAGE -> KM_SLEEP b8d06fc Switch KM_SLEEP to KM_PUSHPAGE account in module/zfs/dmu_zfetch.c for ISO C90 - mixed declarations and code - warnings References: https://www.illumos.org/issues/5987 zfs prefetch code needs work illumos/illumos-gate@cf6106c 5987 zfs prefetch code needs work Porting notes: Module parameters (new/changed): [in module/zfs/dmu_zfetch.c] replaced zfetch_block_cap with zfetch_max_distance (Max bytes to prefetch per stream (default 8MB; 8 * 1024 * 1024)) changed zfs_prefetch_disable from int to boolean_t (B_False) ; typedef boolean_t bool; in lib/libspl/include/sys/types.h Ported-by: kernelOfTruth [email protected]
Reviewed by: Adam Leventhal <[email protected]> Reviewed by: Matt Amdur <[email protected]> Reviewed by: Saso Kiselkov <[email protected]> Reviewed by: Boris Protopopov <[email protected]> Reviewed by: Richard Lowe <[email protected]> Approved by: Dan McDonald <[email protected]> diverged code base from Illumos: [include/sys/fs/zfs.h] openzfs@f67d709 Create an 'overlay' property openzfs@11b9ec2 Add full SELinux support References: https://www.illumos.org/issues/4929 illumos/illumos-gate@b461c74 Ported-by: kernelOfTruth [email protected]
Reviewed by: Matthew Ahrens <[email protected]> Reviewed by: Prakash Surya <[email protected]> Approved by: Dan McDonald <[email protected]> References: https://www.illumos.org/issues/6171 dsl_prop_unregister() slows down dataset eviction. illumos/illumos-gate@03bad06#diff-832418f8bef771ab321cc8bca7235e4dL297 Porting notes: Notable changes from upstream are caused by the following commits: openzfs@3558fd7 Prototype/structure update for Linux openzfs@2cf7f52 Linux compat 2.6.39: mount_nodev() openzfs@13fe019 Illumos openzfs#3464 Fix compilation error: dsl_prop.c:1211:15: error: ‘dsl_prop_unregister’ undeclared here (not in a function) EXPORT_SYMBOL(dsl_prop_unregister); Replacing EXPORT_SYMBOL(dsl_prop_unregister); with EXPORT_SYMBOL(dsl_prop_unregister_all); Fixing 'space or tab at end of line' in include/sys/dsl_dataset.h Conflicts during rebase on December 9th 2015 in include/sys/dsl_dataset.h due to openzfs@241b541 (Illumos 5959 - clean up per-dataset feature count code) Ported-by: kernelOfTruth [email protected]
Reviewed by: George Wilson <[email protected]> Reviewed by: Paul Dagnelie <[email protected]> Reviewed by: Richard Elling <[email protected]> Reviewed by: Xin Li <[email protected]> Reviewed by: Arne Jansen <[email protected]> Approved by: Dan McDonald <[email protected]> References: https://www.illumos.org/issues/2605 illumos/illumos-gate@9c3fd12 Porting notes: The tests from Illumos in usr/src/test/zfs-tests were left out Further the following files which don't exist in ZoL: usr/src/cmd/truss/expound.c usr/src/lib/libzfs/common/mapfile-vers usr/src/lib/libzfs_core/common/mapfile-vers usr/src/pkg/manifests/system-test-zfstest.mf From a readability point of view the changes in [module/zfs/dmu_traverse.c] from openzfs@ecfb0b5 Fix misuse of input argument in traverse_visitbp appear to be the optimum - also there's no functional change, when going with upstream the compiler would likely complain about ISO C90 errors - so only switching cdnp = buf->b_data; to child_dnp = buf->b_data; diverged code base from Illumos: [cmd/zstreamdump/zstreamdump.c] openzfs@37f8a88 Illumos 5746 - more checksumming in zfs send [include/sys/zfs_ioctl.h] openzfs@0cee240 Speed up 'zfs list -t snapshot -o name -s name' [module/zfs/zfs_ioctl.c] openzfs@13fe019 Illumos 3464 zfs synctask code needs restructuring [module/zfs/dmu_send.c] openzfs@b58986e Use large stacks when available openzfs@044baf0 Use taskq for dump_bytes() ISO C90 - mixed declarations and code: struct send_block_record *to_data; ISO C90 - mixed declarations and code: struct receive_ign_obj_node *n; [module/zfs/dsl_dataset.c] Illumos 4185 add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R hasn't landed yet, thus ' #include <sys/dmu_send.h> ' #include <sys/zio_checksum.h> are still missing [libzfs_sendrecv.c] ' #include <sha2.h> isn't listed in the header list since sha2, sha256 is implemented differently in ZoL openzfs@b8864a2 Fix gcc cast warnings [module/zcommon/zfs_prop.c] openzfs@11b9ec2 Add full SELinux support [module/zfs/dmu_traverse.c] openzfs@ecfb0b5 Fix misuse of input argument in traverse_visitbp openzfs@79c76d5 Change KM_PUSHPAGE -> KM_SLEEP openzfs@a168788 Reduce stack for traverse_visitbp() recursion openzfs@b8d06fc Switch KM_SLEEP to KM_PUSHPAGE openzfs@47050a8 Fix stack traverse_impl() FIXME: man/man8/zfs.8 man/man8/zfs.8.rej Ported-by: kernelOfTruth [email protected]
|
@kernelOfTruth all the dependent patches in this stack have now been merged to master, that should make it easier to finish the porting work here. Be aware that both kernel.org builders are failing for unrelated reasons. Now that Linux 4.4 was released the upstream kernel devs has changed some APIs on us again and we need to update the code to handle it. Thank you for opening all these PRs with illumos commits it helps enormously! |
|
@behlendorf Thanks a lot 👍 Several of these commits would affect me in one way or another - so I simply had to port them ;) Also ZFS currently is my Filesystem of choice so I'm personally interested that it works as well as it can for my daily usage. Will work on 2605 again once I'm not occupied that more by other projects - but first I'll rebase it on current tree |
|
superseded by: #4213 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Illumos 2605 want to resume interrupted zfs send
Reviewed by: George Wilson [email protected]
Reviewed by: Paul Dagnelie [email protected]
Reviewed by: Richard Elling [email protected]
Reviewed by: Xin Li [email protected]
Reviewed by: Arne Jansen [email protected]
Approved by: Dan McDonald [email protected]
References:
https://www.illumos.org/issues/2605
illumos/illumos-gate@9c3fd12
Porting notes:
The tests from Illumos in usr/src/test/zfs-tests were left out
Further the following files which don't exist in ZoL:
usr/src/cmd/truss/expound.c
usr/src/lib/libzfs/common/mapfile-vers
usr/src/lib/libzfs_core/common/mapfile-vers
usr/src/pkg/manifests/system-test-zfstest.mf
From a readability point of view the changes in [module/zfs/dmu_traverse.c]
from zfsonlinux@ecfb0b5 Fix misuse of input argument in traverse_visitbp
appear to be the optimum - also there's no functional change,
when going with upstream the compiler would likely complain about ISO C90 errors - so only switching
cdnp = buf->b_data;
to
child_dnp = buf->b_data;
diverged code base from Illumos:
[cmd/zstreamdump/zstreamdump.c]
zfsonlinux@37f8a88 Illumos 5746 - more checksumming in zfs send
[include/sys/zfs_ioctl.h]
zfsonlinux@0cee240 Speed up 'zfs list -t snapshot -o name -s name'
[module/zfs/zfs_ioctl.c]
zfsonlinux@13fe019 Illumos 3464 zfs synctask code needs restructuring
[module/zfs/dmu_send.c]
zfsonlinux@b58986e Use large stacks when available
zfsonlinux@044baf0 Use taskq for dump_bytes()
ISO C90 - mixed declarations and code: struct send_block_record *to_data;
ISO C90 - mixed declarations and code: struct receive_ign_obj_node *n;
[module/zfs/dsl_dataset.c]
Illumos 4185 add new cryptographic checksums to ZFS: SHA-512, Skein, Edon-R
hasn't landed yet, thus
' #include sys/dmu_send.h
' #include sys/zio_checksum.h
are still missing
[libzfs_sendrecv.c]
' #include <sha2.h>
isn't listed in the header list since sha2, sha256 is implemented differently in ZoL
zfsonlinux@b8864a2 Fix gcc cast warnings
[module/zcommon/zfs_prop.c]
zfsonlinux@11b9ec2 Add full SELinux support
[module/zfs/dmu_traverse.c]
zfsonlinux@ecfb0b5 Fix misuse of input argument in traverse_visitbp
zfsonlinux@79c76d5 Change KM_PUSHPAGE -> KM_SLEEP
zfsonlinux@a168788 Reduce stack for traverse_visitbp() recursion
zfsonlinux@b8d06fc Switch KM_SLEEP to KM_PUSHPAGE
zfsonlinux@47050a8 Fix stack traverse_impl()
FIXME:
man/man8/zfs.8
man/man8/zfs.8.rej
Ported-by: kernelOfTruth [email protected]