hostid deprecation causes zpool.cache mismatch and zpool import failure

[dajhorn]

Commit openzfs/spl@acf0ade deprecates the /etc/hostid file, relaxes its handler, and sets a default of zero. The new default breaks userland imports by causing the /etc/zfs/zpool.cache file to be updated without a ZPOOL_CONFIG_HOSTID nvpair.

When the pool cache is in this state, invoked pool imports always fail. For example:

# zpool create tank ...
# zfs umount -a
# rmmod zfs
# modprobe zfs zfs_autoimport_disable=1
# zpool import tank
cannot import 'tank': pool may be in use from other system
use '-f' to import anyway

This happens because:

1. The zpool program assumes that ZPOOL_CONFIG_HOSTID exists in the configuration at https://github.com/zfsonlinux/zfs/blob/master/cmd/zpool/zpool_main.c#L1919
2. The zpool program still calls gethostid() from the system library, which causes an SPA mismatch if a generated value is returned at https://github.com/zfsonlinux/zfs/blob/master/cmd/zpool/zpool_main.c#L1921
3. Unlike zpool, the libzfs library uses zero to skip its mismatch check at https://github.com/zfsonlinux/zfs/blob/master/lib/libzfs/libzfs_status.c#L222
4. The zfs module uses zero to skip a VERIFY at https://github.com/zfsonlinux/zfs/blob/master/module/zfs/spa_config.c#L407

Automatic pool import is unaffected because it runs in the kernel on a different code path. This corner-case is easier to notice when the proposal in #2779 is enabled. (ie: zfs_autoimport_disable=1 is the system default.)

Fuzzing this behavior on a test bench sometimes causes the additional disappearance of ZPOOL_CONFIG_HOSTNAME, which causes assertion failures later.

Solaris 11 updates its /etc/zfs/zpool.cache file identically when its /etc/hostid file is forced to zero using the "_I________" string, but imports are not broken when the hostid is missing or the hostname is empty.

For code consistency, zpool could be patched to skip its mismatch check on zero too.

Alternatively, given that zero is a valid hostid on Linux and seems to be something special on Solaris, the solution could be one or more of:

• Disable the hostid checks entirely.
• Never write out the affected nvpair unless the /etc/hostid file actually exists.
• Wrap the system gethostid() so that it behaves like zone_get_hostid() in the SPL.
• Use any number except zero as the dummy value, preferably with upper bits that fit into the nvpair but are masked out by glibc.

[behlendorf]

@dajhorn Thanks for getting to the bottom on this. This clearly explains why this caused issues on some systems and not others.

For code consistency, zpool could be patched to skip its mismatch check on zero too.

I think we should definitely do this. For better or worse the upstream code is designed to use a hostid of 0 to disable these checks and it would be desirable to remain compatible with that logic.

Disable the hostid checks entirely.

Sadly we can't disable these checks entirely. For sites which are using ZFS in a legitimate failover configuration it's the only multimount protect they have. At least until a robust system is implemented like that described in #745.

Never write out the affected nvpair unless the /etc/hostid file actually exists.
Wrap the system gethostid() so that it behaves like zone_get_hostid() in the SPL.

I like this idea a lot. Unifying the behavior between user space and kernel space will simplify things. For example, we'd be able to remove the conditional logic here and here if we provided a zone_get_hostid() wrapper. It looks like we could retire the hw_serial global entirely and replace it with a global hostid variable just like was done in the SPL.

Can you propose a patch for this?