recursive dataset not sendable if missing child datasets

[prometheanfire]

zfs send -R ${POOL_NAME}@backup-${DATE} > foo.dd
cannot send ${POOL_NAME}@backup-${DATE} recursively: snapshot ${POOL_NAME}/GENTOO@backup-${DATE} does not exist
exits 0

so, not sure what's happening, but something between these two dates that cause a fail maybe?

I create a recursive snapshot, remove some of the snapshots, then send it. That, sending a single dataset still seems to work though

Sep 12 10:53 kernel-4.4.20-gentoo
Sep 16 21:20 kernel-4.4.21-gentoo

[prometheanfire]

only commit that mentions send/recv by name in that timeframe is 2aa3438 but I didn't do an exhaustive search

[prometheanfire]

this effects master as of sept 16 21:20 CDT

[behlendorf]

Well compressed send/recv is a biggie, but all the regression tests do still pass after that change was merged. Are you able to reproduce the issue with an unpatched version or ZFS on Linux from Github?

[prometheanfire]

I'll try again from git master soon (this weekend at latest), busy packaging openstack atm :P

[prometheanfire]

@behlendorf the failure to export recursive datasets when one of the child datasets is deleted persists on current master

also, found another dracut type bug https://goo.gl/photos/YrZRY5PAm2EXSLzV9

[prometheanfire]

any other info you need?

[behlendorf]

I don't think so. Just time.

[prometheanfire]

ok, I modified my backup script to send one at a time instead.

[loli10K]

I think this was introduced by 6635624 (OpenZFS 6111 - zfs send should ignore datasets created after the ending snapshot) to address a problem with incremental (-i|-I) replication.

I could probably try to fix this ...

EDIT: other ZFS implementations are probably affected too, but unfortunately i don't have any recent Illumos/FreeBSD to play with right now.

EDIT2: confirmed to affect other ZFS implementations (tested on SmartOS, the first version to pull in OpenZFS 6111 is the one on the right in the screenshot, 20160527T033529Z)

[loli10K]

I don't know if this is really a bug or just a (small?) POLA violation, reading the comment (https://github.com/zfsonlinux/zfs/blob/6635624020506a76cb79c65c7ff5a580c4685a0b/lib/libzfs/libzfs_sendrecv.c#L815) suggests that the intention was to actually raise an error when sending a replication stream with snapshots deleted from any dataset in the hierarchy ... but probably we also don't want to break user scripts with the next release.

[prometheanfire]

Ya, I've changed my scripts to scan and just send the snapshots that still exist, but this may be an example of a bug becoming a feature (which is now missing).

[behlendorf]

Closing for now. We'll stick with the upstream behavior and users depending on the previous behavior will need to update their scripts. Adding an option to explicitly allow for missing snapshots is the stream would be a nice feature if someone is interested in developing it.

[loli10K]

This same issue was reported as a bug for Illumos: https://www.illumos.org/issues/8896 Fix for 6111 breaks recursive send when snapshot intentionally deleted.

[loli10K]

This might also be related: zfs send -p -i now "skips" sending snapshots that were never requested to begin with

zfs-0.6.5.11

root@linux:~# zfs create testpool/fs
root@linux:~# zfs snap testpool/fs@s1
root@linux:~# zfs snap testpool/fs@s2
root@linux:~# zfs snap testpool/fs@s3
root@linux:~# zfs send -p -v -i testpool/fs@s1 testpool/fs@s2 > /dev/null
send from @s1 to testpool/fs@s2 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
root@linux:~#

zfs-0.7.9

root@linux:~# zfs create testpool/fs
root@linux:~# zfs snap testpool/fs@s1
root@linux:~# zfs snap testpool/fs@s2
root@linux:~# zfs snap testpool/fs@s3
root@linux:~# zfs send -p -v -i testpool/fs@s1 testpool/fs@s2 > /dev/null
skipping snapshot testpool/fs@s3 because it was created after the destination snapshot (s2)
send from @s1 to testpool/fs@s2 estimated size is 624B
total estimated size is 624B
TIME        SENT   SNAPSHOT
root@linux:~#

[Parakleta]

I posted the Illumos bug 8896 and I just want to draw attention to the Oracle Administration document I linked in that bug report: https://docs.oracle.com/cd/E26502_01/html/E29007/recover-3.html. Specifically:

Reduce the snapshot archive by removing the swap and dump snapshots, if desired.

sysA# zfs destroy rpool/[email protected]
sysA# zfs destroy rpool/[email protected]

The swap volume does not contain data that is relevant to a system migration or recovery. Do not remove the dump volume snapshot if you wish to preserve any crash dumps.

That is, the documented procedure for backing up a system is to create a recursive snapshot of the entire pool, then remove the snapshots for filesystems that are not required to be backed up, and then send a replication stream. This breaks now because the missing snapshots cause the replication send to fail. I am not aware of any work-around.