Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

provide nss modules globally instead of hacking them with nscd #55276

Closed
flokli opened this issue Feb 5, 2019 · 22 comments
Closed

provide nss modules globally instead of hacking them with nscd #55276

flokli opened this issue Feb 5, 2019 · 22 comments
Labels
2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md

Comments

@flokli
Copy link
Contributor

flokli commented Feb 5, 2019

We currently proxy all nss requests through nscd so we can add nss modules (see 1d5f4cb)

This is bad, as glibc quickly falls back to resolving locally if resolution through nscd takes too long, and in that case without custom nss modules loaded.

Not relying solely on nscd, but rather assembling together an nss modules path, and writing that nix store path into /etc/ld.so.conf seems to be a better solution and should work.

Using the actual path instead of a symlink should also avoid most problems with binaries and ABI breakages of glibc as described in #50316 (comment) that might occur when switching to a more recent glibc - at least it should be better than the half-broken state with nscd timing out that we currently might have.

cc @arianvp @Mic92

@Mic92
Copy link
Member

Mic92 commented Feb 5, 2019

I don't think this is a good solution to the problem as it can likely make mixing applications linked against different glibc versions impossible. This is in my opinion worse then breaking the mdns resolver. I don't see how an ld.so.conf would help with ABI incompatibilities (which we currently also disable).

@edolstra
Copy link
Member

edolstra commented Feb 5, 2019

👎 since this adds yet another bit of ad hoc global state, and breaks the case of running 32-bit binaries on 64-bit systems.

@edolstra
Copy link
Member

edolstra commented Feb 5, 2019

I mean, nscd is how we provide NSS modules globally, in a standard, ABI-neutral way.

@flokli
Copy link
Contributor Author

flokli commented Feb 5, 2019

I was only thinking about adding a path to additional nss modules (and only those not provided via glibc itself).

About being able to run 32bit binaries, I need to read on on how that works on other distros - they should have a global /etc/ld.so.conf.

This is not only to fix mdns resolving, but also about other resources provided via additional nss modules, like ldap user lookup through nss.

I'd be fine with proxying things through nscd, but glibc falling back to local resolving which does things differently is really inconsistent too.

If I find the time, I might dig into glibc and whether there is some keepalive mechanism.

@flokli
Copy link
Contributor Author

flokli commented Jun 14, 2019

After some thinking, I guess using nscd is fine for resolving - couldn't find extremely short fallback timeouts either.

@flokli
Copy link
Contributor Author

flokli commented May 7, 2020

In the meantime, I could spot the "resolve-via-nscd" timeouts - see https://gitlab.com/freedesktop-sdk/mirrors/sourceware/glibc/-/blob/master/nscd/nscd_helper.c#L216. (5secs)

If these timeout, glibc gives up waiting for nscd and tries to resolve locally (without having nss modules available, so it'll likely fail as well).

5 seconds is at least more time than Microsoft is willing to wait for slow lookups like LLMNR (https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn305896(v=ws.11) - so it should be enough for us as well.

What do others think?

@arianvp
Copy link
Member

arianvp commented May 7, 2020

The assessment that putting nss modules in a global path leads to issues seems to be debunked.

dlsym/dlopen'ing nss modules from different versions of Glibc should just work; and is ABI-stable according to the glibc developers:

https://sourceware.org/legacy-ml/libc-help/2016-12/msg00006.html

Note that the question was even asked in relation to a NixOS issue :)

The original question:

Nikolay (CC) and I are trying[1] to properly integrate third-party NSS modules
into NixOS[2]. We'd like to share existing modules compiled against different versions of glibc. Is it a safe thing to do? What are the chances of modules breaking that way for updated versions of glibc?

The answer:

TLDR; You can mix any NSS modules you want with any version of glibc. The dlopen and dlsym interface used isolates you from ABI issues. The API issues that result could be problematic though, and some features could return errors if the underlying NSS service module DSOs don't implement the required low-level functions required by the high-level APIs.

@arianvp
Copy link
Member

arianvp commented May 7, 2020

Aha; but further in the thread it reads:

https://sourceware.org/legacy-ml/libc-help/2016-12/msg00008.html

in general it is not possible to update glibc safely because it
consists of multiple shared libs without stable abi between them
and the update cannot be done atomically, with runtime loaded
components even atomic file system update does not help: existing
process with old libc may load a new libc component after the
update (even if the loaded module has stable abi, it may depend
on a not yet loaded libc component that does not).

the safe way to implement extensible libc behaviour is to have it
in a separate component (daemon) with stable interface (name lookup
protocol) that survives independent updates of the two sides
(you only need to make sure that the daemon is not loading libc
components during libc update and that it can reexec itself without
dropping connections for daemon update).

which is a long way to say that i think nss modules should support
nscd and sssd should be fixed to work with it.

(emphasis mine)

Which is exactly what we're doing.

I'm just leaving these comments here for historical context in case somebody stumbles over this issue too

@arianvp
Copy link
Member

arianvp commented May 7, 2020

However; the issue that is being described the the problem of updating glibc in place; (which we never do; as each glibc version lives in a different nix store path) whilst the issue we're trying to solve here is updating nss modules in place . This still sounds safe to do to me.

However I do not particularly mind keeping the nscd system in place

@flokli
Copy link
Contributor Author

flokli commented Nov 29, 2020

So what you're saying is if we'd patch glibc to also discover other nss modules from a "mutable" /run/glibc-nss-modules/ (or similar), like we have for opengl drivers, even if built with another glibc version, the dlsym abstraction should be stable enough?

I'm no fan of just adding it into LD_LIBRARY_PATH globally, as this won't necessarily update for all new processes.

@flokli
Copy link
Contributor Author

flokli commented Dec 8, 2020

Let's give this some more thought and experimentation.

Maybe the ABI is stable enough, at to support this as a separate option when nscd is disabled.

There's valid usecases where people do not want to use nscd, for example when having a separate network namespace with a default gateway via a VPN tunnel. In the current setup, host lookups are done through nscd in the main network namespace, possibly leaking DNS lookups.

@stale
Copy link

stale bot commented Jun 6, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 6, 2021
@flokli flokli removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jun 8, 2021
@flokli
Copy link
Contributor Author

flokli commented Sep 16, 2021

As can be seen in #135888, nscd seems to do some caching (at least recently), and doesn't just proxy requests through, so it might make sense to get rid of nscd, and try a path as outlined in #55276 (comment).

I did a quick hack on this some months ago, but got stuck somewhere. I'll try to resurrect and push it somewhere.

flokli added a commit to flokli/nixpkgs that referenced this issue Sep 21, 2021
NSS modules are now globally provided (by providing a `/run/nss-modules`
symlink), similar to how we handle OpenGL drivers.

This removes the need for nscd as a proxy for all NSS requests, and avoids
DNS requests leaking across network namespaces.

While doing this upgrade, existing applications need to be restarted, so
they know how to pick up NSS modules from `/run/nss-modules`.

If you want to defer application restart to a later time, explicitly enable
`nscd` via `services.nscd.enable` until the application restart.

We can mix NSS modules from any version of glibc according to
https://sourceware.org/legacy-ml/libc-help/2016-12/msg00008.html,
so glibc upgrades shouldn't break old userland loading more recent NSS
modules (and most likely, NSS modules are already loaded)

Fixes: NixOS#55276
Fixes: NixOS#135888
Fixes: NixOS#105353
Cc:    NixOS#52411 (comment)
@flokli
Copy link
Contributor Author

flokli commented Sep 21, 2021

#138178 now uses /run/nss-modules, and lookup for that path is patched into glibc, in addition to LD_LIBRARY_PATH.

I don't think we should use nix to manage NSS modules on non-NixOS systems, but load the ones provided by the distro, as it's distro configuration.

This doesn't really change any behaviour on non-NixOS compared to before right now.

We could add /usr/lib into that patch as well (only for NSS module loading, to make nix-built binaries and NSS behave better on non-NixOS without nscd running), but that should be a followup.

flokli added a commit to flokli/nixpkgs that referenced this issue Sep 22, 2021
NSS modules are now globally provided (by providing a `/run/nss-modules`
symlink).

See the text added to `rl-2111.section.md` for further details.

Fixes: NixOS#55276
Fixes: NixOS#135888
Fixes: NixOS#105353
Cc:    NixOS#52411 (comment)
erikarvstedt added a commit to erikarvstedt/nixpkgs that referenced this issue Oct 18, 2021
NSS modules are now globally provided by a symlink in `/run`.

See the description in `add-extra-module-load-path.patch` for further details.

Fixes: NixOS#55276
Fixes: NixOS#135888
Fixes: NixOS#105353
Cc:    NixOS#52411 (comment)

Co-authored-by: Erik Arvstedt <[email protected]>
erikarvstedt added a commit to erikarvstedt/nixpkgs that referenced this issue Oct 24, 2021
NSS modules are now globally provided by a symlink in `/run`.

See the description in `add-extra-module-load-path.patch` for further details.

Fixes: NixOS#55276
Fixes: NixOS#135888
Fixes: NixOS#105353
Cc:    NixOS#52411 (comment)

Co-authored-by: Erik Arvstedt <[email protected]>
@abbradar
Copy link
Member

While working on #154320 I found out that Fedora deprecated nscd because of slow maintenance and existing serious issues. These issues sound serious enough for us to also consider it a problem.

@Mic92
Copy link
Member

Mic92 commented Jan 12, 2022

There is also https://busybox.net/~vda/unscd/ which seems to be still maintained.

@veprbl
Copy link
Member

veprbl commented Jan 12, 2022

We could add /usr/lib into that patch as well (only for NSS module loading, to make nix-built binaries and NSS behave better on non-NixOS without nscd running), but that should be a followup.

Would be nice to have something like that working, however, how can that work if the NSS modules in /usr/lib are built with a different glibc?

@flokli
Copy link
Contributor Author

flokli commented Jan 19, 2022

Yet another issue caused by this: #154928

erikarvstedt added a commit to erikarvstedt/nixpkgs that referenced this issue Jan 19, 2022
NSS modules are now globally provided by a symlink in `/run`.

See the description in `add-extra-module-load-path.patch` for further details.

Fixes: NixOS#55276
Fixes: NixOS#135888
Fixes: NixOS#105353
Cc:    NixOS#52411 (comment)

Co-authored-by: Erik Arvstedt <[email protected]>
erikarvstedt added a commit to erikarvstedt/nixpkgs that referenced this issue Jan 19, 2022
NSS modules are now globally provided by a symlink in `/run`.

See the description in `add-extra-module-load-path.patch` for further details.

Fixes: NixOS#55276
Fixes: NixOS#135888
Fixes: NixOS#105353
Cc:    NixOS#52411 (comment)

Co-authored-by: Erik Arvstedt <[email protected]>
erikarvstedt pushed a commit to erikarvstedt/nixpkgs that referenced this issue Jan 19, 2022
NSS modules are now globally provided (by providing a `/run/nss-modules`
symlink).

See the text added to `rl-2111.section.md` for further details.

Fixes: NixOS#55276
Fixes: NixOS#135888
Fixes: NixOS#105353
Cc:    NixOS#52411 (comment)
@stale stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 31, 2022
@flokli
Copy link
Contributor Author

flokli commented Oct 20, 2022

I think for now we need to have a stable interface. #196917 is probably the better alternative to glibc-nscd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md
Projects
Status: Done
Development

No branches or pull requests

6 participants