Don't use ' dup2(0, 3)' unconditionally

[alexey-tikhonov]

When a system is configured to use 'libnss_sss.so.2' as NSS provider (i.e. /etc/nsswitch.conf contains "passwd: sss files ...") this code -

tigervnc/unix/xserver/hw/vnc/xvnc.c

Line 369 in effd854

dup2(0, 3);

- ruins internal state of 'libnss_sss.so.2'

The flow is as follows:

• after all the fork/exec/etc, Xvnc calls one of NSS functions (most probably getpwuid(getuid)) but not necessarily);
• this initializes NSS stack, and calls to 'libnss_sss'
• this initializes internal data structures of 'libnss_sss', in particular opens some file descriptors
• one of file descriptors obtained by 'libnss_sss' is 3 (first free)
• Xvnc "hijacks" this fd with dup2(0, 3);
• Xvnc call NSS again, 'libnss_sss' tries to work with this fd, discovers there is "something wrong", closes it (fd=3), reinits (and gets fd=3 again)
• Xvnc keeps working with fd=3, discovers there is "something wrong", terminates

strace looks like:

fstat(3, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
munmap(0x7f3bf9792000, 9253600)         = 0
close(3)                                = 0
openat(AT_FDCWD, "/var/lib/sss/mc/passwd", O_RDONLY|O_CLOEXEC) = 3
...
setsockopt(3, SOL_TCP, TCP_CORK, [1], 4) = -1 ENOTSOCK (Socket operation on non-socket)

And Xvnc log:

Xvnc[76330]: XserverDesktop: client gone, sock 3
Xvnc[76330]: Selection: Remote clipboard lost, removing local ownership
Xvnc[76330]: VNCSConnST: closing [::ffff:...]::56002: read: Socket operation on non-socket (88)

To Reproduce
Configure system:

• to use Xvnc with '-inetd'
• nsswitch.conf to use "sss" before "files"
• very recent SSSD (that has SSSD/sssd@9c4ac1b)

and try to login: after successful authentication session terminates.

Client (please complete the following information):
Doesn't matter.

Server (please complete the following information):
Linux/TigerVNC (Xvnc)

What is the reason to use dup2() and assign vncInetdSock=3 unconditionally instead of dup() and obtain assigned fd using F_GETFD?

Issue is worked around with the help of SSSD/sssd#7085 but I think this is a bug in TigerVNC

[CendioOssman]

That code is absolutely ancient, so unfortunately you would have to ask RealVNC why it was written that way.

But I agree, using dup() seems the sensible approach here.

Have you tried changing it and see if everything seems to work?

[alexey-tikhonov]

Have you tried changing it and see if everything seems to work?

Not yet.
I can try to craft a patch. But wanted to get feedback first, maybe this '3' is "hardcoded"/expected in other places of the code...

[CendioOssman]

It might be, unfortunately. It shouldn't, of course, but it might. So testing will be needed.

It does set vncInetdSock, though, which hopefully everything else looks at.