Ubuntu Trusty (and likely other) do not build from master

[sempervictus]

Merging pkg-zfs/master/ubuntu/trusty produces merge conflicts:

$ git checkout -b pkg-test origin/master 
Branch pkg-test set up to track remote branch master from origin.
Switched to a new branch 'pkg-test'
$ git log | head
commit 541a09016d652d92a9a665a3e62d83907b8b9941
Author: Brian Behlendorf <[email protected]>
Date:   Tue Mar 22 18:08:59 2016 -0700

    Add zloop.sh test script

    Add Chris Williamson's "new" zloop script so that it may be
    intergated with ZoLs automated testing.  The original script may
    be found in the openzfs-build repository on Github.

$ git merge pkg-zfs/master/ubuntu/trusty 
Auto-merging rpm/generic/zfs.spec.in
Auto-merging rpm/generic/zfs-kmod.spec.in
Auto-merging rpm/generic/zfs-dkms.spec.in
CONFLICT (content): Merge conflict in rpm/generic/zfs-dkms.spec.in
Auto-merging module/zfs/zvol.c
CONFLICT (content): Merge conflict in module/zfs/zvol.c
Auto-merging module/zfs/zfs_ioctl.c
CONFLICT (content): Merge conflict in module/zfs/zfs_ioctl.c
Auto-merging module/zfs/vdev_queue.c
Auto-merging module/zfs/dsl_destroy.c
CONFLICT (content): Merge conflict in module/zfs/dsl_destroy.c
Auto-merging module/zfs/dmu_send.c
CONFLICT (content): Merge conflict in module/zfs/dmu_send.c
Auto-merging module/zcommon/zfs_uio.c
CONFLICT (content): Merge conflict in module/zcommon/zfs_uio.c
Auto-merging include/sys/spa_impl.h
CONFLICT (content): Merge conflict in include/sys/spa_impl.h
Auto-merging cmd/mount_zfs/mount_zfs.c
Auto-merging TEST
CONFLICT (add/add): Merge conflict in TEST
Automatic merge failed; fix conflicts and then commit the result.

The merge is reasonably trivial - master is just newer. One minor note is that there appears to be divergence in zvol.c. After merge and reconfiguration of the debian/source/format to native, the build goes fine, but the packaging fails:

make[7]: Leaving directory `/build/zfs/tests/zfs-tests/tests/functional/zvol'
make[6]: Leaving directory `/build/zfs/tests/zfs-tests/tests/functional'
 (cd stress && /usr/bin/make  top_distdir=../../../../zfs-0.6.5.6 distdir=../../../../zfs-0.6.5.6/tests/zfs-tests/tests/stress \
     am__remove_distdir=: am__skip_length_check=: am__skip_mode_fix=: distdir)
make[6]: Entering directory `/build/zfs/tests/zfs-tests/tests/stress'
make[6]: *** No rule to make target `Makefile.am', needed by `distdir'.  Stop.

Pulling out the commit which adds the zfs test suite allows the build, but then the dkms source tree is empty. Kind of a show stopper for testing on ubuntu.

[sempervictus]

Turns out that the commit creating the zfstest suite can be left in if the following hack is applied:

diff --git i/debian/rules w/debian/rules
index 0d3a99d..f7923b4 100755
--- i/debian/rules
+++ w/debian/rules
@@ -18,7 +18,7 @@ override_dh_auto_configure:
      --sbindir=/sbin \
      --libdir=/lib \
      --with-udevdir=/lib/udev \
-     --with-config=user
+     --with-config=user && touch tests/zfs-tests/tests/stress/Makefile.am

 override_dh_auto_test:
    # The dh_auto_test rule is disabled because

But zfs-dkms deb package still has has no sources and produces:

does not appear to have module source located within it.  Build halted.

[behlendorf]

@sempervictus I'm not sure I understand, but it sounds like you've just somehow lost the tests/zfs-tests/tests/stress/Makefile.am in your merge. It's definitely part of the original patch which was merged to master.

[sempervictus]

Its not being lost in the merge, its being deleted by the Debian packaging scripts every time they run, hence that little hack. Even with that hack though, its not building the actual DKMS source tree into the deb. Basically, the packaging scripts break the build system far as i can tell

[dajhorn]

@sempervictus,

Note that pkg-zfs installs a /usr/src that is intended only for DKMS and should not (and likely cannot) be used to build the test suite.

This quirk is the result of an old production goal, and is a wontfix for PPA releases.

Past that, the pkg-zfs/master/ubuntu/trusty branch should merge cleanly with the upstream release branches, but not necessarily with the upstream head.

[sempervictus]

@dajhorn: thank you very much, i was starting to think i'd lost my marbles in a merge somewhere.

Is there any means by which the packaging could be configured to ignore the test suite, or allow it to build and simply not include it? My apologies for ignorance of the depths of the debian packaging stack, the worst i've had to do is in-house DKMS packages for network drivers and we don't use a build system like this, so it was a lot simpler.

Could you suggest any hackish/short-term solutions to allow us to build from master with patches against this repo? We operate a datacenter wherein we dedicate VM farms and occasionally full physical storage platforms to testing new ZFS functionality. Its helped track down some serious problems in the past, including the realization that free space maps can diverge from the tree of data blocks and actually conflict (FSM thinks its free, block has data) since a scrub doesn't touch free blocks (sort of useful to understanding potential catastrophic situations). At present, we're actually pretty stuck in our own testing as we evaluate a number of kernel builds + zfs builds + SCST and other things across a set of VMs prior to taking the finalists to hardware. We use DKMS for all this stuff, and our kernel builds roll things which have to be baked in (EoIP, etc). We then have Chef deploy it all, kick off tests via ztest, fio, and others, and push results to logstash. Seemed like a rather clever approach in terms of engineering time allotted to the test cycles, until this happened.

[dajhorn]

@sempervictus,

thank you very much,

Welcome.

i was starting to think i'd lost my marbles in a merge somewhere.

I often get merge glitches in the RPM area during DEB builds, and just buff them down before pushing the to pkg-zfs repository. In particular, git-buildpackage et al mishandle the symlink changes that sometimes happen in the ZoL HEAD.

Is there any means by which the packaging could be configured to ignore the test suite, or allow it to build and simply not include it?

Yes, although it would be more natural (and a similar amount of work) to put these things into separate zfs-testsuite and/or zfs-contrib packages. However, given that pkg-zfs is in legacy maintenance since Ubuntu decided to import ZoL, this kind of work would best continue downstream.

Seemed like a rather clever approach in terms of engineering time allotted to the test cycles, until this happened.

I would describe that less as clever and more as a common deployment method at big sites. :) The upgrade path between Trusty and Xenial is cloudy in this regard.

Nevertheless, try to retool around the official kernel packages, which should be easier since the core team at Canonical began using git. (NB: https://wiki.ubuntu.com/Kernel/MainlineBuilds)

[sempervictus]

Given the systemd piece and what looks to be their own spin on "ZoL integration" we're keeping away from 16.04 for the time being. LTS' are usually not worth the headache for the first 3-5 months anyway.
Far as this repo going into legacy maintenance mode, wouldn't that force users who want to keep ZoL versions and their Ubuntu kernel versions diverged to build new kernels with some ungodly merge process every time they update ZoL? Seems like significant overhead and a loss of functionality to drop the DKMS approach.

[dajhorn]

wouldn't that force users who want to keep ZoL versions and their Ubuntu kernel versions diverged to build new kernels with some ungodly merge process every time they update ZoL?

Yes.

Seems like significant overhead and a loss of functionality to drop the DKMS approach.

Perhaps, but this is reasonable business decision since ZoL became a stable supported first-class part of the core system.

[sempervictus]

We've been building kernel-specific packages using the make deb functionality for the last few months, but a bit of a problem arises when the OS is booting from ZFS using the resulting packages - the root pool won't import after the encrypted volume it's on is available. Even a manual import from the initramfs shell results in breakage: mount errors after pool import with rpool/root being at /root.

Master and the tagged releases diverge a good deal, as does master and the 16.04 version from what i've seen in testing. Any chance you would reconsider updating the Ubuntu Trusty branch to work with master again?