System unbootable after upgrading ostree

[guyshapiro]

I tried to update the OSTree recipe to newer version (current master, 2017.12+) and it made the OTA image unbootable.

When booting such image, the init.sh script fails with the error "mkdir: can't create directory 'run/initramfs': Invalid argument", followed by a kernel panic.

When I mounted the image on my desktop and tried writing to /ostree/deploy/poky/deploy/*/, I got a similar error.

Inspecting the image with the 'debugfs' utility (version 1.42.12), it looks like the directory has extended attribute with an empty name:

# debugfs /dev/mmcblk0p2
debugfs:  cd ostree/deploy/poky/deploy
debugfs:  stat b75cf8e31aed705fe7cf5ea40d43f6325416345ea80def0acb84dad67d3301c7.0
Inode: 16   Type: directory    Mode:  0755   Flags: 0x80000
Generation: 0    Version: 0x00000000:00000000
User:     0   Group:     0   Size: 4096
File ACL: 0    Directory ACL: 0
Links: 13   Blockcount: 8
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x59db3191:00000000 -- Mon Oct  9 11:21:37 2017
 atime: 0x59db31b8:00000000 -- Mon Oct  9 11:22:16 2017
 mtime: 0x59db3191:00000000 -- Mon Oct  9 11:21:37 2017
crtime: 0x59db31b9:00000000 -- Mon Oct  9 11:22:17 2017
Size of extra inode fields: 32
Extended attributes stored in inode body: 
   = "02 00 00 00 " (4)
EXTENTS:
(0):2398

The content of the generated ostree repo contains the "system.posix_acl_default" extended attribute:

$ ostree --repo=$src ls -X b75cf8e31aed705fe7cf5ea40d43f6325416345ea80def0acb84dad67d3301c7 / | head -n 1
d00755 0 0      0 { [(b'system.posix_acl_default', [byte 0x02, 0x00, 0x00, 0x00])] } /

The ostree repo generated before the version change didn't had such attribute.

As a workaround, I added --no-xattrs to the ostree commit command on image_types_ostree.bbclass.

(cc: @cgwalters )

[cgwalters]

Hmm. This might be fallout from this: ostreedev/ostree#1170 (comment) You're using the cmdline ostree commit sounds like, but...

It seems to me the old code should have had the behavior of picking up whatever's on disk as well; which for OE-style builds you don't want.

The only xattrs you want (AIUI) are SMACK, which I still think based on previous discussion should likely be done equvialently to the SELinux support in libostree. Which now actually has a convenient command line --selinux-policy /path/to/rootfs, so it'd likely be the same.

Anyways for now I think adding --no-xattrs is probably the correct thing for you.

[OYTIS]

Hi @guyshapiro ,
thank you for reporting. Last time we had a similar issue fixing it involved patching pseudo, but that was quite a while ago. Could you please share the /proc/config.gz from your build machine (the one where you run bitbake) so that we can reproduce it?

[guyshapiro]

The build machine is running Debian 8.9 with kernel version 4.9.30 from backports.
I uploaded the config file from /boot to here.

[OYTIS]

@guyshapiro thank you. One more thing: do I understand it right that with current OSTree revision as it is set in the pyro branch (3b09620c2738bce4ed45e099cf2e4c5df7671d39) everything works fine? If so, could you please share the exact commit hash for the broken one.

[guyshapiro]

I still working on the morty branch.
I pushed the related commits to a new ostree-upgrade branch.
On the original morty branch (9cf14ec) everything works fine (except that the ostree version is quite old).
After updating the ostree version on the recipe (9006adc), the issue occurs.
The last commit on the branch (6ea840a) is the mentioned workaround.

[OYTIS]

Sorry, @guyshapiro, could you reupload the config please? I can't find where I've put it on my machine.

[guyshapiro]

Sure. Uploaded that to here.
Hopefully it will stay there...

[cgwalters]

Is this the same symptom even with --no-xattrs? Do you still see the system.posix_acl_default entry?

Are there any new errors?

[guyshapiro]

Adding --no-xattrs still works-around the issue.
There is nothing new on my side...

[OYTIS]

@guyshapiro
One more thing: what branch of poky are you using? If it's "morty" or earlier, could you try to copy pseudo_1.8.2.bb recipe from poky/meta/recipes-devtools in "pyro" branch to your poky and rebuild everything?

[guyshapiro]

I'm using the morty branch of poky. I'll try to check the pseudo_1.8.2.bb recipe within a few days.

[OYTIS]

@guyshapiro
Sorry, that's most likely the source of problems. I'll add pseudo_1.8.2.bb to morty branch of meta-updater.

[cajun-rat]

Related PR #153

[guyshapiro]

After PR #153, it works for me.
Thanks!

[cajun-rat]

Excellent, I'm glad to hear that!