Move away from nss-altfiles (was: Messed up permissions in /var) #362

shdwchn10 · 2022-09-30T09:44:15Z

Workaround and related tracking issues

Documentation: https://docs.fedoraproject.org/en-US/fedora-silverblue/troubleshooting/#_unable_to_add_user_to_group

Other references:

Original issue

Describe the bug
Sometimes rpm-ostree deploys cause wrong permissions randomly in /var.

To Reproduce
Please describe the steps needed to reproduce the bug:

Install tor and tang. Additional installation of cockpit can increase chances of catching this bug.
Enable tor and tang: systemctl enable --now tor.service and systemctl enable --now tang.socket.
Make new deploy. I was able to reproduce this bug with following commands: rpm-ostree upgrade, rpm-ostree remove and rpm-ostree rebase.

Expected behavior
Correct permissions should persist.

Some examples (reproduced in clean VMs)
System before tests:

$ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora:fedora/36/x86_64/silverblue
                  Version: 36.20220928.0 (2022-09-28T11:32:19Z)
               BaseCommit: 0c8fb996df77c338563b4e066ef2a3fe61d6a04f1a67d6954d74f159fdc95a61
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
          LayeredPackages: cockpit tang tor

Normal permissions state:

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang tang 188 Sep 29 00:49 .
drwxr-xr-x. 1 root root  16 Sep 29 00:48 ..
-r--r-----. 1 tang tang 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1 tang tang 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root        144 Sep 29 00:50 .
drwxr-xr-x. 1 root    root        856 Sep 29 00:48 ..
-rw-------. 1 toranon toranon   20442 Sep 29 00:49 cached-certs
-rw-------. 1 toranon toranon 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1 toranon toranon 8790462 Sep 29 00:50 cached-microdescs.new
drwx------. 1 toranon toranon       0 Sep 29 00:49 keys
-rw-------. 1 toranon toranon       0 Sep 29 00:49 lock
-rw-------. 1 toranon toranon    3466 Sep 29 00:50 state

Some results of reproducing this bug:

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang tang 188 Sep 29 00:49 .
drwxr-xr-x. 1 root root  16 Sep 29 00:48 ..
-r--r-----. 1  961  961 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1  961  961 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 11036
drwxr-x---. 1 toranon root     178 Sep 29 13:42 .
drwxr-xr-x. 1 root    root     856 Sep 29 00:48 ..
-rw-------. 1     962  962   20442 Sep 29 00:49 cached-certs
-rw-------. 1     962  962 2245089 Sep 29 13:38 cached-microdesc-consensus
-rw-------. 1     962  962 8790462 Sep 29 13:38 cached-microdescs
-rw-------. 1     962  962  228806 Sep 29 13:39 cached-microdescs.new
drwx------. 1     962  962       0 Sep 29 00:49 keys
-rw-------. 1     962  962       0 Sep 29 13:38 lock
-rw-------. 1     962  962    4325 Sep 29 13:42 state

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang               tang               188 Sep 29 00:49 .
drwxr-xr-x. 1 root               root                16 Sep 29 00:48 ..
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 11036
drwxr-x---. 1 toranon root     178 Sep 29 13:51 .
drwxr-xr-x. 1 root    root     962 Sep 29 19:24 ..
-rw-------. 1 tang    tang   20442 Sep 29 00:49 cached-certs
-rw-------. 1 tang    tang 2245089 Sep 29 13:38 cached-microdesc-consensus
-rw-------. 1 tang    tang 8790462 Sep 29 13:38 cached-microdescs
-rw-------. 1 tang    tang  228806 Sep 29 13:39 cached-microdescs.new
drwx------. 1 tang    tang       0 Sep 29 00:49 keys
-rw-------. 1 tang    tang       0 Sep 29 13:44 lock
-rw-------. 1 tang    tang    4806 Sep 29 13:51 state

Additional context
Related to coreos/rpm-ostree#3179

The text was updated successfully, but these errors were encountered:

travier · 2022-09-30T10:16:03Z

As a short term workaround, I recommend writing a tmpfiles.d config that sets the right ownership at boot.
Medium term, converting to those packages to sysusers configs helps make them more reproduce-able.
Long term, we'll have to move away from nss-altfiles. See coreos/fedora-coreos-tracker#155 for more details (for Fedora CoreOS but it's likely the same issue).

travier · 2022-09-30T10:17:33Z

If you can compare the content of /etc/passwd, /usr/etc/passwd & same for group between each deployments that would be useful.

shdwchn10 · 2022-10-05T12:36:26Z

I did some retests and saw no difference difference in /etc/{passwd,group} and /usr/etc/{passwd,group} between deployments:

Before:

$ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora:fedora/36/x86_64/silverblue
                  Version: 36.20220928.0 (2022-09-28T11:32:19Z)
               BaseCommit: 0c8fb996df77c338563b4e066ef2a3fe61d6a04f1a67d6954d74f159fdc95a61
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
          LayeredPackages: cockpit cockpit-file-sharing cockpit-machines cockpit-navigator cockpit-networkmanager cockpit-ostree cockpit-podman cockpit-selinux cockpit-session-recording tang tor

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang tang 188 Sep 29 00:49 .
drwxr-xr-x. 1 root root  16 Sep 29 00:48 ..
-r--r-----. 1 tang tang 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1 tang tang 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root        144 Sep 29 00:50 .
drwxr-xr-x. 1 root    root        856 Sep 29 00:48 ..
-rw-------. 1 toranon toranon   20442 Sep 29 00:49 cached-certs
-rw-------. 1 toranon toranon 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1 toranon toranon 8790462 Sep 29 00:50 cached-microdescs.new
drwx------. 1 toranon toranon       0 Sep 29 00:49 keys
-rw-------. 1 toranon toranon       0 Sep 29 00:49 lock
-rw-------. 1 toranon toranon    3466 Sep 29 00:50 state

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
systemd-coredump:x:970:970:systemd Core Dumper:/:/usr/sbin/nologin
shdwchn10:x:1000:1000:shadowchain:/var/home/shdwchn10:/bin/bash

$ cat /usr/etc/passwd
root:x:0:0:root:/root:/bin/bash

$ cat /etc/group
root:x:0:
wheel:x:10:shdwchn10
systemd-coredump:x:970:
shdwchn10:x:1000:

$ cat /usr/etc/group 
root:x:0:
wheel:x:10:

1st test — $ rpm-ostree remove cockpit-file-sharing cockpit-machines cockpit-navigator cockpit-networkmanager cockpit-ostree cockpit-podman cockpit-selinux cockpit-session-recording:

$ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora:fedora/36/x86_64/silverblue
                  Version: 36.20220928.0 (2022-09-28T11:32:19Z)
               BaseCommit: 0c8fb996df77c338563b4e066ef2a3fe61d6a04f1a67d6954d74f159fdc95a61
             GPGSignature: Valid signature by 53DED2CB922D8B8D9E63FD18999F7CBF38AB71F4
          LayeredPackages: cockpit tang tor

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang tang 188 Sep 29 00:49 .
drwxr-xr-x. 1 root root  16 Sep 29 00:48 ..
-r--r-----. 1  961  961 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1  961  961 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root     178 Oct  5 14:33 .
drwxr-xr-x. 1 root    root     856 Sep 29 00:48 ..
-rw-------. 1     962  962   20442 Sep 29 00:49 cached-certs
-rw-------. 1     962  962 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1     962  962 8790462 Oct  5 14:26 cached-microdescs
-rw-------. 1     962  962       0 Oct  5 14:26 cached-microdescs.new
drwx------. 1     962  962       0 Sep 29 00:49 keys
-rw-------. 1     962  962       0 Oct  5 14:26 lock
-rw-------. 1     962  962    3955 Oct  5 14:33 state

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
systemd-coredump:x:970:970:systemd Core Dumper:/:/usr/sbin/nologin
shdwchn10:x:1000:1000:shadowchain:/var/home/shdwchn10:/bin/bash

$ cat /usr/etc/passwd
root:x:0:0:root:/root:/bin/bash

$ cat /etc/group
root:x:0:
wheel:x:10:shdwchn10
systemd-coredump:x:970:
shdwchn10:x:1000:

$ cat /usr/etc/group 
root:x:0:
wheel:x:10:

2nd test — $ rpm-ostree rebase fedora:fedora/$(($(rpm -E %fedora) + 1))/x86_64/silverblue:

$ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora:fedora/37/x86_64/silverblue
                  Version: 37.20221004.n.0 (2022-10-04T08:04:42Z)
               BaseCommit: 0d0cbfffdf49b08c94f66d97c86b41098fa144e724c5547a92c13e6097f99fe3
             GPGSignature: Valid signature by ACB5EE4E831C74BB7C168D27F55AD3FB5323552A
          LayeredPackages: cockpit tang tor

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang               tang               188 Sep 29 00:49 .
drwxr-xr-x. 1 root               root                16 Sep 29 00:48 ..
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root     178 Oct  5 14:33 .
drwxr-xr-x. 1 root    root     926 Oct  5 15:31 ..
-rw-------. 1 tang    tang   20442 Sep 29 00:49 cached-certs
-rw-------. 1 tang    tang 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1 tang    tang 8790462 Oct  5 14:26 cached-microdescs
-rw-------. 1 tang    tang       0 Oct  5 14:26 cached-microdescs.new
drwx------. 1 tang    tang       0 Sep 29 00:49 keys
-rw-------. 1 tang    tang       0 Oct  5 14:26 lock
-rw-------. 1 tang    tang    3955 Oct  5 14:33 state

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
systemd-coredump:x:970:970:systemd Core Dumper:/:/usr/sbin/nologin
shdwchn10:x:1000:1000:shadowchain:/var/home/shdwchn10:/bin/bash

$ cat /usr/etc/passwd
root:x:0:0:root:/root:/bin/bash

$ cat /etc/group
root:x:0:
wheel:x:10:shdwchn10
systemd-coredump:x:970:
shdwchn10:x:1000:

$ cat /usr/etc/group
root:x:0:
wheel:x:10:

travier · 2022-10-05T22:59:36Z

Sorry I meant /usr/lib/passwd & /usr/lib/group.

shdwchn10 · 2022-10-24T14:55:19Z

Before is the same state as in the previous comment plus:
/usr/lib/passwd: https://pastebin.com/eaC3G6jn
/usr/lib/group: https://pastebin.com/EhjEzUKP

1st test — $ rpm-ostree remove cockpit-file-sharing cockpit-machines cockpit-navigator cockpit-networkmanager cockpit-ostree cockpit-podman cockpit-selinux cockpit-session-recording:

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang tang 188 Sep 29 00:49 .
drwxr-xr-x. 1 root root  16 Sep 29 00:48 ..
-r--r-----. 1  961  961 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1  961  961 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root     178 Oct 24 17:35 .
drwxr-xr-x. 1 root    root     856 Sep 29 00:48 ..
-rw-------. 1     962  962   20442 Sep 29 00:49 cached-certs
-rw-------. 1     962  962 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1     962  962 8790462 Oct 24 17:25 cached-microdescs
-rw-------. 1     962  962       0 Oct 24 17:25 cached-microdescs.new
drwx------. 1     962  962       0 Sep 29 00:49 keys
-rw-------. 1     962  962       0 Oct 24 17:25 lock
-rw-------. 1     962  962    3956 Oct 24 17:35 state

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
systemd-coredump:x:970:970:systemd Core Dumper:/:/usr/sbin/nologin
shdwchn10:x:1000:1000:shadowchain:/var/home/shdwchn10:/bin/bash

$ cat /etc/group
root:x:0:
wheel:x:10:shdwchn10
systemd-coredump:x:970:
shdwchn10:x:1000:

$ sudo chown toranon:toranon -R /var/lib/tor
$ sudo chown tang:tang -R /var/db/tang

/usr/lib/passwd: https://pastebin.com/X6kcUkVX (diff from previous deploy: https://pastebin.com/D1edXzWW)
/usr/lib/group: https://pastebin.com/9CFQrc1T (diff from previous deploy: https://pastebin.com/ffZLgj6c)

2nd test — $ rpm-ostree rebase fedora:fedora/$(($(rpm -E %fedora) + 1))/x86_64/silverblue:

$ sudo ls -la /var/{lib/tor,db/tang}
/var/db/tang:
total 8
drwx------. 1 tang               tang               188 Sep 29 00:49 .
drwxr-xr-x. 1 root               root                16 Sep 29 00:48 ..
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 367 Sep 29 00:49 oehQ3bK3dkxaiVBi-ORnyeeFpxDyirGTufLWBIjG-GY.jwk
-r--r-----. 1 cockpit-wsinstance cockpit-wsinstance 361 Sep 29 00:49 tbok-4BDZorNVUUpMFCqkhu1LDOKA99CBnFlj1PHXvM.jwk

/var/lib/tor:
total 10812
drwxr-x---. 1 toranon root     178 Oct 24 17:35 .
drwxr-xr-x. 1 root    root     926 Oct 24 17:48 ..
-rw-------. 1 tang    tang   20442 Sep 29 00:49 cached-certs
-rw-------. 1 tang    tang 2249230 Sep 29 00:49 cached-microdesc-consensus
-rw-------. 1 tang    tang 8790462 Oct 24 17:25 cached-microdescs
-rw-------. 1 tang    tang       0 Oct 24 17:25 cached-microdescs.new
drwx------. 1 tang    tang       0 Sep 29 00:49 keys
-rw-------. 1 tang    tang       0 Oct 24 17:25 lock
-rw-------. 1 tang    tang    3956 Oct 24 17:35 state

$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
systemd-coredump:x:970:970:systemd Core Dumper:/:/usr/sbin/nologin
shdwchn10:x:1000:1000:shadowchain:/var/home/shdwchn10:/bin/bash

$ cat /etc/group
root:x:0:
wheel:x:10:shdwchn10
systemd-coredump:x:970:
shdwchn10:x:1000:

/usr/lib/passwd: https://pastebin.com/nDKBXyTd (diff from previous deploy: https://pastebin.com/0BvAsS69)
/usr/lib/group: https://pastebin.com/fVnDUHF2 (diff from previous deploy: https://pastebin.com/HCqac2nA)

travier · 2022-10-26T08:57:25Z

The toranon user and group have a different IDs in each one of those deployments:

/usr/lib/passwd: https://pastebin.com/eaC3G6jn
/usr/lib/passwd: https://pastebin.com/X6kcUkVX
/usr/lib/passwd: https://pastebin.com/nDKBXyTd

The easiest workaround for this issue is to make a tpmfiles.d config to make sure the directories are chown'ed on boot to the correct UIDs/GIDs:

$ cat /etc/tmpfiles.d/tor.conf
Z /var/lib/tor - toranon toranon - -

$ cat /etc/tmpfiles.d/tang.conf
Z /var/lib/tang - tang tang - -

travier · 2022-12-08T13:52:52Z

Documentation: https://docs.fedoraproject.org/en-US/fedora-silverblue/troubleshooting/#_unable_to_add_user_to_group

Other references:

shdwchn10 added the bug Something isn't working label Sep 30, 2022

travier added enhancement New feature or request f36 Related to Fedora 36 and removed bug Something isn't working labels Sep 30, 2022

travier added the need-info Further information is requested label Oct 5, 2022

travier added f37 Related to Fedora 37 rawhide f38 Related to Fedora 38 and removed need-info Further information is requested labels Oct 26, 2022

travier changed the title ~~Messed up permissions in /var~~ Move away from nss-altfiles (was: Messed up permissions in /var) Oct 26, 2022

travier added the fedora-change Needs a Fedora Change label Oct 26, 2022

travier removed f37 Related to Fedora 37 f36 Related to Fedora 36 f38 Related to Fedora 38 labels Nov 28, 2022

travier added the kinoite Also affect Fedora Kinoite label Dec 8, 2022

travier removed the rawhide label May 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move away from nss-altfiles (was: Messed up permissions in /var) #362

Move away from nss-altfiles (was: Messed up permissions in /var) #362

shdwchn10 commented Sep 30, 2022 •

edited by travier

Loading

travier commented Sep 30, 2022

travier commented Sep 30, 2022

shdwchn10 commented Oct 5, 2022

travier commented Oct 5, 2022

shdwchn10 commented Oct 24, 2022

travier commented Oct 26, 2022 •

edited

Loading

travier commented Dec 8, 2022 •

edited

Loading

Move away from nss-altfiles (was: Messed up permissions in /var) #362

Move away from nss-altfiles (was: Messed up permissions in /var) #362

Comments

shdwchn10 commented Sep 30, 2022 • edited by travier Loading

Workaround and related tracking issues

Original issue

travier commented Sep 30, 2022

travier commented Sep 30, 2022

shdwchn10 commented Oct 5, 2022

travier commented Oct 5, 2022

shdwchn10 commented Oct 24, 2022

travier commented Oct 26, 2022 • edited Loading

travier commented Dec 8, 2022 • edited Loading

shdwchn10 commented Sep 30, 2022 •

edited by travier

Loading

travier commented Oct 26, 2022 •

edited

Loading

travier commented Dec 8, 2022 •

edited

Loading