Prevent .pyc files from being added to tarballs

[bedroge]

Not sure when this started happening, but for the tarballs of PR #195 (see e.g. https://github.com/EESSI/staging/pull/129) I noticed that they were changing an existing Python installation. It turns out that it's because of a new .pyc file that is generated in the Python installation directory:

$ ls -l 2021.12/software/linux/aarch64/generic/software/Python/3.9.5-GCCcore-10.3.0/easybuild/python/__pycache__/
total 12
-rw-r--r-- 1 bob bob 1493 Feb  1 02:38 sitecustomize.cpython-310.pyc
-rw-r--r-- 1 bob bob 1469 Jan 19 13:08 sitecustomize.cpython-39.opt-1.pyc
-rw-r--r-- 1 bob bob 1469 Jan  8  2022 sitecustomize.cpython-39.pyc

We have run into something similar before, and added a fix for this to the installation script that should write the .pyc files to a temporary directory:
https://github.com/EESSI/software-layer/blob/main/EESSI-pilot-install-software.sh#L81
But for some reason that doesn't seem to work here.

[trz42]

We have seen this happening when building FlexiBLAS-3.0.4-GCC-10.3.0.eb for NESSI/2022.11, too. The existing sitecustomize.cpython-310.pyc is overwritten. Possibly this happens when tests are run. FlexiBLAS-3.0.4-GCC-10.3.0.eb has a build dependency on Python/3.9.5

    ('Python', '3.9.5'),  # required for running the tests

This happens despite PYTHONPYCACHEPREFIX being set in EESSI-pilot-install-software.sh. It might be that the test run for LAPACK (see https://gitlab.mpi-magdeburg.mpg.de/software/flexiblas-release/-/blob/master/test/lapack-3.10.0/CMakeLists.txt#L27) does not take the setting into account or it is simply not present when the test is called.

In the present case (building FlexiBLAS-3.0.4-GCC-10.3.0.eb) the whole Python/3.9.5 directory is added to the tarball despite only one file for Python/3.9.5 being in the overlay-upper directory. This happens because of how the list of files to be added to the tarball is generated (see line 55 below) and from where the files are sourced (-C ${topdir} in the tar command, line 61). See

software-layer/create_tarball.sh

Lines 53 to 61 in 7ea7058

	if [ -d ${pilot_version}/software/${os}/${cpu_arch_subdir}/software ]; then
	# installation directories
	ls -d ${pilot_version}/software/${os}/${cpu_arch_subdir}/software/*/* | grep -v '/\.wh\.' >> ${files_list}
	fi
	topdir=${cvmfs_repo}/versions/
	echo ">> Creating tarball ${target_tgz} from ${topdir}..."
	tar cfvz ${target_tgz} -C ${topdir} --files-from=${files_list}

There may be different options to not let leak in not-built packages into the tarball:

• filtering what gets tar'ed by the modules being generated (rationale: module files are generated by EasyBuild, so we understand it very well, it is independent of what side effects an installation has ... would also allow us to keep .pyc files if we chose to do so)
• filtering what gets tar'ed by certain file types/endings, e.g., all files ending with .pyc are filtered out (rationale: this is what let not-built packages leak into the tarball, we need to verify that filtering files is enough)
• somehow ensuring that .pyc files are not added to /cvmfs/... (rationale: if they are not created under /cvmfs/... they won't be tar'ed, so we can continue using the existing create_tarball.sh)
• changing from what directory create_tarball.sh creates the tarball (rationale: if we only use files that are in the overlay-upper only those get into the tarball, we may still need to filter out some files/directories)