ADA IO errors

[driesmp]

When I start synth with the following command: synth just-compile /usr/local/etc/synth/LiveSystem-build.conf, synth compiles fine for a few ports but at one point I get a continuous flood of ADA IO errors. At the moment these errors start occurring, synth just falls dead. The synth process keeps running but no more ports are building. Any ideas? Config files are shown in picture.

Log file:
00_last_results.log

My system specs are quite "strong" for a home server, 64GB ECC ram, 4 core 8 thread skylake xeon etc.
My swap space is however only 5GB (swap file). How could I go about and monitor if the swapspace gets full? Does synth gradually start using swap, or at one point it will hit swap for a large amount?

[jrmarino]

Not enough information.
Things like: Did you run out of swap space?
Do you have a "weak" machine, but have an aggressive configuration (e.g. many slaves + jobs per slave).
It would help to describe the machine and the configuration.

chances are you either ran out of memory or disk space.

[driesmp]

Updated my first post with some information. Dont think its disk space related as I have about 400GB left on my nvme SSD.

[jrmarino]

My system specs are quite "strong" for a home server, 64GB ECC ram, 4 core 8 thread skylake xeon etc.
My swap space is however only 5GB (swap file). How could I go about and monitor if the swapspace gets full? Does synth gradually start using swap, or at one point it will hit swap for a large amount?

If you uses curses mode, the swap usage is displayed in realtime at the top of the screen.
If you use the text mode, you'll have to use swapinfo manually to check it.

5G is pretty much nothing. The rule of thumb is use 4 x ram, so you'd be looking at a swap page of 64x4 = 256G. I'd say you'd need at least 100G of swap (and may have to back off the builders/jobs). Ideally you'd use a separate ssd for swap, but using a swapfile would be okay.

My guess is you're running out of swap really quick on something like llvm or gcc or chromium

[jrmarino]

any news on this?
You really don't have an aggressive configuration. 4 builders and 2 jobs per builder should be a piece of cake for your machine. And you shouldn't be dipping into swap unless you are building something like llvm or chromium. If it happens again, note which ports are being built.

[driesmp]

I tried rebuilding my build-list yesterday, no luck in trying to reproduce the error.
Like you just said, it didn't even touch swap when rebuilding my list even when building gcc or llvm.

It might be port related as you suggested, I saw that devel/ninja was building when the error occured (shown in log file). When I let it rebuild my list yesterday, devel/ninja wasnt sucked in to be build by synth. Maybe it is a specific error with a port (devel/ninja). Strange that it didn't get sucked in again tho (only thing I did was update the ports tree and add a few more default_version options in LiveSystem-make.conf, which shouldn't impact dependencies in a direct way (I think)).

=> I'm gonna rebuild my list again tonight and if anything comes up I'll keep you up-to date.

[driesmp]

Yea ... no more ADA IO errors ... Strange...

I have a few ports not building tho, could you help me troubleshoot some of them?

net___samba46.log
sysutils___uefi-edk2-bhyve.log
sysutils___uefi-edk2-bhyve-csm.log
databases___py-MySQLdb.log
devel___libvirt.log

I used the following entries in LiveSystem-make.conf:
DEFAULT_VERSIONS+=mysql=10.2m
DEFAULT_VERSIONS+=php=7.1
DEFAULT_VERSIONS+=python=3.6
DEFAULT_VERSIONS+=samba=4.6
DISABLE_LICENSES=yes
OPTIONS_UNSET+=X11

As I read in the enormous thread at forums.freebsd.net, synth doesnt suck in anything from the hosts make.conf (this is still accurate right?). I feel like there is an issue with python. My goal is to build every port against the most recent distribution of software. Like php71 instead of php56, python3.6 instead of python2.7 etc ... This should be possible using ports no? I also have a few ports not building because of USE_PACKAGE_DEPENDS_ONLY. Does synth set this automatically? I have no such entry in LiveSystem-make.conf.conf, altought when checking logs, it is clear that it is set to yes.

[jrmarino]

Does synth set this automatically?

yes. It means "stop if this package doesn't exist".

I looked at the first log and see couldn't find a python27 package. So the hard-change to python3.6 is probably what caused it.

the whole py27/py36 thing is hacked in ports and poudriere brought in another hack to deal with it. I'm not going to bring in that hack because it goes away when flavors are incorporated (which removes the original hackage)

I guess you have to either:

1. go back to python27 (remove the DEFAULT VERSION of python 3.6)
2. build these particular ports either manually or via poudriere

[jrmarino]

libvirt is a different problem:
mv util/virkeycode-linux.7-t2 util/virkeycode-linux.7
Can't open util/virkeycode-linux.pod: No such file or directory at /usr/local/bin/pod2man line 70.
not sure what's causing that.

[driesmp]

Yea most ports had an issue trying to compile against python 3.6, I removed the "DEFAULT_VERSIONS+=python=3.6" line and everything builds fine except one.
py-MySQLdb doesnt want to build in synth. Same error as before as seen in attached logfile:

databases___py-MySQLdb.log

I have found this open issue: farcepest/MySQLdb1#132
Do you think I'm dealing with the same problem?
I find it strange that the py-MySQLdb port compiles fine on my host system, while trying to compile it in synth it fails. I find this strange as I have the same version of MariaDB installed (10.2.9; as package on host system, and 10.2.9 build from ports in synth). I even have the exact same make.conf entries as in LiveSystem-make.conf

[jrmarino]

try adding "MAKE_JOBS_UNSAFE=yes" to the databases/py-MySQLdb/Makefile file and try again

[driesmp]

Alright, that port doesn't matter to me anyway, I have disabled python-poller in librenms port. I really love synth tho. <3 Awesome program. Just one more question which I couldn't find anywhere.
What is the proper way to launch synth in background? Using cron?

When I issue: synth just-build /usr/local/etc/LiveSystem-build.conf & to launch in background I get quite a lot of echos on my screen of my ssh session.

[jrmarino]

hmmm, I thought that was a FAQ entry but it's not.
generally you have 2 profiles: one with ncurses and one without. (use this one for cron)
You set the profile for cron, something like "/usr/bin/env SYNTHPROFILE=cron /usr/local/bin/synth ... > /dev/null" or something like that.

[driesmp]

At least one useful thing that came out of this thread for you; an undescribed FAQ... :)
Thanks for the help!