Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

elusive error in "make install" for Qt5-5.15.2-GCCcore-10.3.0.eb #16614

Closed
boegel opened this issue Nov 10, 2022 · 6 comments
Closed

elusive error in "make install" for Qt5-5.15.2-GCCcore-10.3.0.eb #16614

boegel opened this issue Nov 10, 2022 · 6 comments
Labels
EESSI Related to EESSI project problem report
Milestone

Comments

@boegel
Copy link
Member

boegel commented Nov 10, 2022

When trying to install Qt5-5.15.2-GCCcore-10.3.0.eb, I'm running into a weird problem in the install step:

/tmp/boegel/easybuild/build/Qt5/5.15.2/GCCcore-10.3.0/qt-everywhere-src-5.15.2/qtbase/bin/qmake -install qinstall /tmp/boegel/easybuild/build/Qt5/5.15.2/GCCcore-10.3.0/qt-everywhere-src-5.15.2/qtbase/include/QtGui/QGenericPlugin /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin
Error copying /tmp/boegel/easybuild/build/Qt5/5.15.2/GCCcore-10.3.0/qt-everywhere-src-5.15.2/qtbase/include/QtGui/QGenericPlugin to /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin: Cannot create /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin for output
make[3]: *** [Makefile:87752: install_gen_headers] Error 3
make[3]: Leaving directory '/tmp/boegel/easybuild/build/Qt5/5.15.2/GCCcore-10.3.0/qt-everywhere-src-5.15.2/qtbase/src/gui'
make[2]: *** [Makefile:489: sub-gui-install_subtargets] Error 2
make[2]: Leaving directory '/tmp/boegel/easybuild/build/Qt5/5.15.2/GCCcore-10.3.0/qt-everywhere-src-5.15.2/qtbase/src'
make[1]: *** [Makefile:63: sub-src-install_subtargets] Error 2
make[1]: Leaving directory '/tmp/boegel/easybuild/build/Qt5/5.15.2/GCCcore-10.3.0/qt-everywhere-src-5.15.2/qtbase'
make: *** [Makefile:98: module-qtbase-install_subtargets] Error 2

I'm only seeing this on some systems, in particular VMs in AWS where /tmp is on an NVMe block device (see also here):

$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p2  200G  6.0G  194G   3% /

This same problem is also reported in various Qt bugs, see QTBUG-86592, QTBUG-63891, QTBUG-63193, but the underlying cause was never identified it seems (since the problem "just disappeared" for them)...

@boegel
Copy link
Member Author

boegel commented Nov 10, 2022

It seems like this problem only manifests when the build is done through the EESSI build container (ghcr.io/eessi/build-node:debian10), and not when Qt5 is installed directly in the VM, so perhaps there's some incompatiblity between the OS in the container and the NVMe /tmp filesystem...

@boegel
Copy link
Member Author

boegel commented Nov 11, 2022

The patch from qt/qtbase@d34282d (which applies fine to Qt5 5.15.2) may reveal more information when the error occurs...

@boegel
Copy link
Member Author

boegel commented Nov 12, 2022

Trying to build Qt5-5.15.2-GCCcore-10.3.0.eb with the patch below included reveals the underlying problem, but it's still a bit of a mystery what's exactly going on:

Error copying /tmp/boegel/easybuild/build/Qt5/5.15.2/GCCcore-10.3.0/qt-everywhere-src-5.15.2/qtbase/include/QtGui/QGenericPlugin to /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin:
Cannot create /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin for output:
File exists

Especially since:

$ ls -ld /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin
ls: cannot access '/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin':
No such file or directory

$ ls -l /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui | grep QGenericPlugin
-rw-r--r--. 1 boegel users 35 Oct 27  2020 QGenericPluginFactory

The length of the path makes me wonder if we're hitting a magic limit of 128 somewhere (could be a coincidence):

$ echo '/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/' | wc -c
127
$ echo '/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin' | wc -c
141

I've only seen this when using the EESSI build container, where we use fuse-overlayfs to create a writable overlay for /cvmfs/pilot.eessi-hpc.org/, which may well have something to do with this weird issue...
It could be a bug in fuse-overlayfs for example...

$ fuse-overlayfs --version
fuse-overlayfs: version 0.3
FUSE library version 3.4.1
using FUSE kernel interface version 7.27
fusermount3 version: 3.4.1

Patch to reveal underlying problem:

add underlying error message to errors produced by Qfile::copy;
see https://github.com/qt/qtbase/commit/d34282dba0ebe67c16c1ee6e25d85f019b48b615
and https://github.com/qt/qtbase/commit/f17648bf55c7bf5b43143bc529754ce125c60088
--- qt-everywhere-src-5.15.2/qtbase/src/corelib/io/qfile.cpp
+++ qt-everywhere-src-5.15.2/qtbase/src/corelib/io/qfile.cpp
@@ -817,7 +817,8 @@ QFile::copy(const QString &newName)
                             totalRead += in;
                             if (in != out.write(block, in)) {
                                 close();
-                                d->setError(QFile::CopyError, tr("Failure to write block"));
+                                d->setError(QFile::CopyError, tr("Failure to write block: %1")
+                                            .arg(out.errorString()));
                                 error = true;
                                 break;
                             }
@@ -837,7 +838,8 @@ QFile::copy(const QString &newName)
                         if (!out.rename(newName)) {
                             error = true;
                             close();
-                            d->setError(QFile::CopyError, tr("Cannot create %1 for output").arg(newName));
+                            d->setError(QFile::CopyError, tr("Cannot create %1 for output: %2")
+                                        .arg(newName, out.errorString()));
                         }
                     }
 #ifdef QT_NO_TEMPORARYFILE

@boegel
Copy link
Member Author

boegel commented Nov 12, 2022

If I look into the overlay-upper directory that fuse-overlayfs uses to make the writable overlay, there is a include/QtGui/QGenericPlugin "file" present it seems:

$ ls -l /tmp/$USER/EESSI/overlay-upper/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin*
c---------. 1 boegel users 0, 0 Nov 11 19:34 /tmp/boegel/EESSI/overlay-upper/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin
-rw-r--r--. 1 boegel users   35 Oct 27  2020 /tmp/boegel/EESSI/overlay-upper/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPluginFactory

$ file /tmp/boegel/EESSI/overlay-upper/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin
/tmp/boegel/EESSI/overlay-upper/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin: character special (0/0)

From within the build container with the writable overlay, it's not there though:

Singularity> ls -l /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin
ls: cannot access '/cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin': No such file or directory

And I can create the file just fine:

Singularity> echo test > /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin

Singularity> ls -l /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin
-rw-r--r--. 1 boegel users 5 Nov 12 07:01 /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin

And then the file also looks like expected in the overlay-upper directory:

$ file /tmp/boegel/EESSI/overlay-upper/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin
/tmp/boegel/EESSI/overlay-upper/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin: ASCII text

$ ls -l /tmp/$USER/EESSI/overlay-upper/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin*
-rw-r--r--. 1 boegel users  5 Nov 12 07:01 /tmp/boegel/EESSI/overlay-upper/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPlugin
-rw-r--r--. 1 boegel users 35 Oct 27  2020 /tmp/boegel/EESSI/overlay-upper/versions/2021.12/software/linux/aarch64/graviton3/software/Qt5/5.15.2-GCCcore-10.3.0/include/QtGui/QGenericPluginFactory

@boegel
Copy link
Member Author

boegel commented Nov 12, 2022

One more clue: the file being copied is a regular C source file, nothing special it seems:

Singularity> file /tmp/boegel/easybuild/build/Qt5/5.15.2/GCCcore-10.3.0/qt-everywhere-src-5.15.2/qtbase/include/QtGui/QGenericPlugin
/tmp/boegel/easybuild/build/Qt5/5.15.2/GCCcore-10.3.0/qt-everywhere-src-5.15.2/qtbase/include/QtGui/QGenericPlugin: C source, ASCII text

Singularity> ls -l /tmp/boegel/easybuild/build/Qt5/5.15.2/GCCcore-10.3.0/qt-everywhere-src-5.15.2/qtbase/include/QtGui/QGenericPlugin
-rw-r--r--. 1 boegel users 28 Oct 27  2020 /tmp/boegel/easybuild/build/Qt5/5.15.2/GCCcore-10.3.0/qt-everywhere-src-5.15.2/qtbase/include/QtGui/QGenericPlugin

@boegel
Copy link
Member Author

boegel commented Nov 12, 2022

This problem doesn't present itself anymore when using an EESSI build container with more recent version of fuse-overlayfs (see also EESSI/filesystem-layer#133):

Singularity> fuse-overlayfs --version
fuse-overlayfs: version 1.9
FUSE library version 3.10.5
using FUSE kernel interface version 7.31
fusermount3 version: 3.10.3

For completeness sake, some details on the local disk in the AWS VM:

$ mount | grep nvme0n1p2
/dev/nvme0n1p2 on / type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

@boegel boegel closed this as completed Nov 12, 2022
@boegel boegel added the EESSI Related to EESSI project label Nov 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
EESSI Related to EESSI project problem report
Projects
None yet
Development

No branches or pull requests

1 participant