Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: use -j1 for makeinstall_host #1392

Merged
merged 1 commit into from
Feb 26, 2017
Merged

Python: use -j1 for makeinstall_host #1392

merged 1 commit into from
Feb 26, 2017

Conversation

MilhouseVH
Copy link
Contributor

@MilhouseVH MilhouseVH commented Feb 26, 2017

While building Python:host on a heavily loaded system (ie. a system concurrently building several builds - FX-8350, 8 cores) I will occasionally experience one of two errors during Python makeinstall_host:

  1. file too short
changing mode of build/scripts-2.7/2to3 from 664 to 775
changing mode of build/scripts-2.7/smtpd.py from 664 to 775
/usr/bin/install -c python /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/toolchain/bin/python2.7
if test -f libpython2.7.so; then \
        if test -n "" ; then \
                /usr/bin/install -c -m 555  /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/toolchain/bin; \
        else \
                /usr/bin/install -c -m 555 libpython2.7.so /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/toolchain/lib/libpython2.7.so.1.0; \
                if test libpython2.7.so != libpython2.7.so.1.0; then \
                        (cd /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/toolchain/lib; ln -sf libpython2.7.so.1.0 libpython2.7.so) \
                fi \
        fi; \
else    true; \
fi
./python: error while loading shared libraries: /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/toolchain/lib/libpython2.7.so.1.0: file too short
Creating directory /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/toolchain/lib/python2.7
make: *** [Makefile:559: sharedmods] Error 127
  1. Bus error (core dumped)
/usr/bin/install -c -m 644 /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/Python-2.7.13/Include/ucnhash.h /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/toolchain/include/python2.7
/usr/bin/install -c -m 644 /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/Python-2.7.13/Include/unicodeobject.h /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/toolchain/include/python2.7
/usr/bin/install -c -m 644 /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/Python-2.7.13/Include/warnings.h /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/toolchain/include/python2.7
/usr/bin/install -c -m 644 /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/Python-2.7.13/Include/weakrefobject.h /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/toolchain/include/python2.7
/usr/bin/install -c -m 644 pyconfig.h /home/neil/projects/scratch/alternates/LibreELEC.tv/build.LibreELEC-RPi.arm-9.0-devel-next/toolchain/include/python2.7/pyconfig.h
Bus error (core dumped)
make: *** [Makefile:559: sharedmods] Error 135

Based on repeated testing[1], there is about a 1-in-13 chance of either error occurring - sometimes Python:host will fail after 6 or 7 attempts, and never more than 13 attempts. When building 6 concurrent builds this means there is at most a 1-in-2 chance of all 6 builds succeeding.

Error #2 is more common than error #1, but both errors always occur in exactly the same place, ie. they are repeatable, and I see these errors only with Python:host (I don't see these errors with any other package), which suggests it is not a CPU/RAM/hardware issue.

By adding -j1 to makeinstall_host I have now repeatedly built Python:host on a heavily loaded system over 50 times without failure. The addition of -j1 does not significantly increase the build time.


[1] Test procedure
In one LE repository, repeatedly build Python:host (my system has 8 cores/threads, so basically -j8 here):

COUNT=0
while [ : ]; do
  COUNT=$((COUNT + 1))
  rm -fr build.LibreELEC-RPi.arm-9.0-devel/toolchain/lib/python2.7
  rm -f build.LibreELEC-RPi.arm-9.0-devel/toolchain/lib/libpython2.7.so*
  PROJECT=RPi ARCH=arm scripts/clean Python
  PROJECT=RPi ARCH=arm scripts/build Python:host || break
done
echo "FAILED AFTER ${COUNT} BUILDS"

and in another completely separate LE repository, concurrently build multiple LE projects (just to stress the system - ie. max out CPU, IO, RAM etc.). In my case all projects were clean, so only the "INSTALL" steps and mksquashfs were performed but this is enough to keep the system "busy" while building 6 concurrent projects (RPi/RPi2/Generic, without and without DEBUG).

Eventually, the Python:host build will fail (unless -j1 is added).

@chewitt chewitt merged commit a9277bf into LibreELEC:master Feb 26, 2017
@chewitt
Copy link
Member

chewitt commented Feb 26, 2017

@MilhouseVH can you backport to 8.0 please, I see the same on the main build server

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants