Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Perl CPAN Support, Frozen cpanm Command, Broken I/O Handling In execmd() #1522

Closed
wbraswell opened this issue Jul 4, 2018 · 9 comments
Closed

Perl CPAN Support, Frozen cpanm Command, Broken I/O Handling In execmd() #1522

wbraswell opened this issue Jul 4, 2018 · 9 comments

Comments

@wbraswell
Copy link
Contributor

wbraswell commented Jul 4, 2018 •
edited
Loading

@liger1978
Discussion of this issue started here: #1519

Specifically, these two comments:
#1519 (comment)
#1519 (comment)

Essentially, this issue causes 2 different negative situations.

First, we can not see any of the output of the cpanm commands, including the critically important configure & build processes such as Makefile.PL and Build.PL and make and gcc and as etc etc. If I don't know what is happening with cpanm then I really have no idea what is happening at all.

Second, we experience an actual freeze/hang during one of the gcc and/or as commands, as shown in the comments linked above.

I have tracked this problem to one line:
https://github.com/wbraswell/fpm/blob/master/lib/fpm/util.rb#L165

if block_given?

If I simply replace this line with if false then both parts of this issue appear to be solved! I can now see all the output generated in real time, and the mysterious hanging gcc/as processes no longer hang! Unfortunately, by doing so I also re-introduce the previous bug 1519, because we need an empty STDIN to be present for any possible interactive processes:
#1519

So, in order to achieve a hybrid solution I actually have to disable all the STDOUT and STDERR parts, while leaving the STDIN parts in tact. I have done this in my latest commit:
wbraswell@3171ee9

Since I am not familiar with the execmd() subroutine and how Ruby handles all the I/O magic, I will not presume that my temporary solution is correct enough to create a pull request. In fact, I assume this would be a not-entirely-correct solution, and that we need to somehow figure out why the hanging is actually happening in order to create a permanent solution.
@wbraswell wbraswell mentioned this issue Jul 4, 2018
Closed
@liger1978
Copy link
Contributor

I'm not seeing this other issue (CentOS 7). I've pushed again to https://github.com/liger1978/fpm/tree/1519 which adds the extra required safesystemin() calls in cpan.rb.

@wbraswell
Copy link
Contributor Author

@liger1978
This is the command which freezes w/out my hybrid solution:

$ fpm --no-cpan-test --cpan-verbose --verbose --debug-workspace --workdir /home/wbraswell/rperl_packager_tmp/ -t rpm -s cpan RPerl
Setting workdir {:workdir=>"/home/wbraswell/rperl_packager_tmp/", :level=>:info}
[[[ DEBUG CHECKPOINT 010 ]]] {:level=>:info}
Asking metacpan about a module {:module=>"RPerl", :level=>:info}
[[[ DEBUG CHECKPOINT 000 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 001 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 002 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 006 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 007 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 008 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 011 ]]] {:level=>:info}
Downloading perl module {:distribution=>"RPerl", :version=>nil, :level=>:info}
[[[ DEBUG CHECKPOINT 012 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT 013 ]]] {:level=>:info}
Setting package name from 'distribution' {:distribution=>"RPerl", :level=>:info}
Installing any build or configure dependencies {:level=>:info}
[[[ DEBUG CHECKPOINT 060 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 100 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 101 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 120 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 121 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 122 ]]] {:level=>:info}
[[[ DEBUG CHECKPOINT, safesystemin(), 123 ]]] {:level=>:info}
***FROZEN HERE***

#1519 (comment)

@liger1978
Copy link
Contributor

What OS are you using? I'll spin up a Docker container running the same as you.

@wbraswell
Copy link
Contributor Author 

[USER@MACHINE ~]$ cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

[USER@MACHINE ~]$ perl -V
Summary of my perl5 (revision 5 version 16 subversion 3) configuration:

  Platform:
    osname=linux, osvers=3.10.0-514.16.1.el7.x86_64, archname=x86_64-linux-thread-multi
    uname='linux c1bm.rdu2.centos.org 3.10.0-514.16.1.el7.x86_64 #1 smp wed apr 12 15:04:24 utc 2017 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Dccdlflags=-Wl,--enable-new-dtags -Dlddlflags=-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic -Wl,-z,relro  -DDEBUGGING=-g -Dversion=5.16.3 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.8.5 20150623 (Red Hat 4.8.5-16)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -fstack-protector'
    libpth=/usr/local/lib64 /lib64 /usr/lib64
    libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbm_compat
    perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.17'
  Dynamic Linking:   
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,--enable-new-dtags -Wl,-rpath,/usr/lib64/perl5/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 
-mtune=generic -Wl,-z,relro '


Characteristics of this binary (from libperl):
  Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
                        PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT
                        PERL_MALLOC_WRAP PERL_PRESERVE_IVUV USE_64_BIT_ALL
                        USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
                        USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
                        USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
                        USE_REENTRANT_API USE_SITECUSTOMIZE
  Locally applied patches:
        Fedora Patch1: Removes date check, Fedora/RHEL specific
        Fedora Patch3: support for libdir64
        Fedora Patch4: use libresolv instead of libbind
        Fedora Patch5: USE_MM_LD_RUN_PATH
        Fedora Patch6: Skip hostname tests, due to builders not being network capable
        Fedora Patch7: Dont run one io test due to random builder failures
        Fedora Patch9: Fix find2perl to translate ? glob properly (RT#113054)
        Fedora Patch10: Fix broken atof (RT#109318)
        Fedora Patch13: Clear $@ before "do" I/O error (RT#113730)
        Fedora Patch14: Do not truncate syscall() return value to 32 bits (RT#113980)
        Fedora Patch15: Override the Pod::Simple::parse_file (CPANRT#77530)
        Fedora Patch16: Do not leak with attribute on my variable (RT#114764)
        Fedora Patch17: Allow operator after numeric keyword argument (RT#105924)
        Fedora Patch18: Extend stack in File::Glob::glob, (RT#114984)
        Fedora Patch19: Do not crash when vivifying $|
        Fedora Patch20: Fix misparsing of maketext strings (CVE-2012-6329)
        Fedora Patch21: Add NAME headings to CPAN modules (CPANRT#73396)
        Fedora Patch22: Fix leaking tied hashes (RT#107000) [1]
        Fedora Patch23: Fix leaking tied hashes (RT#107000) [2]
        Fedora Patch24: Fix leaking tied hashes (RT#107000) [3]
        Fedora Patch25: Fix dead lock in PerlIO after fork from thread (RT#106212)
        Fedora Patch26: Make regexp safe in a signal handler (RT#114878)
        Fedora Patch27: Update h2ph(1) documentation (RT#117647)
        Fedora Patch28: Update pod2html(1) documentation (RT#117623)
        Fedora Patch29: Document Math::BigInt::CalcEmu requires Math::BigInt (CPAN RT#85015)
        RHEL Patch30: Use stronger algorithm needed for FIPS in t/op/crypt.t (RT#121591)
        RHEL Patch31: Make *DBM_File desctructors thread-safe (RT#61912)
        RHEL Patch32: Use stronger algorithm needed for FIPS in t/op/taint.t (RT#123338)
        RHEL Patch33: Remove CPU-speed-sensitive test in Benchmark test
        RHEL Patch34: Make File::Glob work with threads again
        RHEL Patch35: Fix CRLF conversion in ASCII FTP upload (CPAN RT#41642)
        RHEL Patch36: Do not leak the temp utf8 copy of namepv (CPAN RT#123786)
        RHEL Patch37: Fix duplicating PerlIO::encoding when spawning threads (RT#31923)
  Built under linux  
  Compiled at Aug  2 2017 17:45:03
  @INC:
    /usr/local/lib64/perl5
    /usr/local/share/perl5
    /usr/lib64/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib64/perl5 
    /usr/share/perl5 
    .

[USER@MACHINE ~]$ uname -a
Linux MACHINE 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

@liger1978
Copy link
Contributor

@wbraswell Thanks William. I see you're on CentOS 7 as well. The only difference is that my container is CentOS 7.5. I'm just trying to build your package at the moment. It's a big one, so it is taking some time! :)

@wbraswell
Copy link
Contributor Author

The place where it freezes due to this issue 1522 is confusing and mysterious:
#1519 (comment)
I was not able to figure out even what particular CPAN module was being built when it freezes on as -I . -I /usr/local/include -I /usr/lib64/perl5/CORE --64 -o XS.o.
Thankfully, my fix for this issue at least allows me to move forward, even without a full explanation!

@liger1978
Copy link
Contributor

liger1978 commented Jul 5, 2018 •
edited
Loading

@jordansissel I give up with this. What @wbraswell says is correct. execmd is behaving mysteriously.

@wbraswell
Copy link
Contributor Author

@liger1978
Thanks for trying! :-)

This was referenced Jul 10, 2018
Closed
Closed
jordansissel added a commit that referenced this issue Feb 23, 2019
@jordansissel
Always close stdin when running without a block.
ae9594e 
This prevents subprocesses from mistakenly getting stuck trying to read
from a stdin that won't say anything ;)

Fixes #1519 and #1522.
@jordansissel jordansissel mentioned this issue Feb 23, 2019
Closed
@NicholasBHubbard NicholasBHubbard mentioned this issue Oct 25, 2022
Closed
@NicholasBHubbard NicholasBHubbard mentioned this issue Nov 6, 2022
Merged
@wbraswell
Copy link
Contributor Author

As with #1519, this issue is solved by PR #1955 by @NicholasBHubbard, great job everyone! :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wbraswell @liger1978