Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ppc64le support #43

Closed
5 tasks done
simaishi opened this issue May 8, 2020 · 41 comments
Closed
5 tasks done

Add ppc64le support #43

simaishi opened this issue May 8, 2020 · 41 comments
Assignees
Labels
enhancement New feature or request

Comments

@simaishi
Copy link
Contributor

simaishi commented May 8, 2020

Need to add ppc64le specific changes that's currently done in manageiq-pods build:

@Fryguy
Copy link
Member

Fryguy commented Jul 9, 2020

cc @seth-priya

@seth-priya
Copy link
Contributor

seth-priya commented Jul 10, 2020

@Fryguy I am looking at this.

I was able to build this image for Power using
docker build -t manageiq/rpm_build:latest --build-arg ARCH=ppc64le .
after pulling the qpid-proton-c dependency from fedora

Post that, I did run into the utf_ext gem issue while building manageiq-pods using
./bin/build -d images -r manageiq -b true
and included that and the related sassc change in the manageiq-rpm_build Dockerfile itself for now (I am not yet sure if that would be the right place).

I did not run into the "wmi" issue, however I am hitting the python2 issue in the manageiq-ui build during installation of node-sass. I have tried a couple of ways to install and specify a dependency on python2, however seems that python3 is in the PATH and taking precedence.

Just for reference, at a high level, I have the following changes so far in the Dockerfile

Installing qpid-proton-c as

Pre-installing the sassc and unf_ext gems with the required options for ppc64le
+RUN if [ ${ARCH} = "ppc64le" ] ; then gem install sassc -- --disable-march-tune-native && \

  • gem install unf_ext -v '0.0.7.2' -- --with-cxxflags="-fsigned-char"; fi

and I had to comment this out in the manageiq-pods build file to force it to pick my locally built rpm_build image

  • #docker pull $RPM_BUILD_IMAGE

@seth-priya
Copy link
Contributor

This is the error I am looking at currently

gyp verb check python checking for Python executable "/usr/bin/python3" in the PATH
gyp verb which succeeded /usr/bin/python3 /usr/bin/python3
gyp ERR! configure error
gyp ERR! stack Error: Command failed: /usr/bin/python3 -c import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack File "", line 1
gyp ERR! stack import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack ^
gyp ERR! stack SyntaxError: invalid syntax
gyp ERR! stack
gyp ERR! stack at ChildProcess.exithandler (child_process.js:303:12)
gyp ERR! stack at ChildProcess.emit (events.js:315:20)
gyp ERR! stack at maybeClose (internal/child_process.js:1021:16)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
gyp ERR! System Linux 3.10.0-1127.el7.ppc64le
gyp ERR! command "/usr/bin/node" "/root/BUILD/manageiq-ui-service/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
gyp ERR! cwd /root/BUILD/manageiq-ui-service/node_modules/node-sass

@simaishi
Copy link
Contributor Author

@seth-priya you will see the wmi error at install time (rpms won't install due to missing dependency) - you can either build wmi rpm for power, or comment out Requires: wmi in rpm_spec/subpackages/manageiq-gemset

As for node-sass, I'm surprised adding python2 doesn't work as that's how it was done before. Will check if version is different and if that's causing this new issue... If you don't add python2, do you see Can't find Python executable "python" error?

@simaishi
Copy link
Contributor Author

simaishi commented Jul 10, 2020

@bdunne Maybe we can take the Fedora qpid-proton ppc64le build and put into our yum repo? The qpid-proton package includes multiple RPMs, but what we require are: qpid-proton-c and qpid-proton-c-devel

https://koji.fedoraproject.org/koji/buildinfo?buildID=1479891

@seth-priya
Copy link
Contributor

thanks @simaishi I will comment out wmi for Power for now at least to see if everything else works, in any case, I think that is what we had done last time as well.

No, it is not complaining for missing python2, but trying to use python3 and failing with a syntax error
gyp ERR! stack Error: Command failed: /usr/bin/python3 -c import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack File "", line 1
gyp ERR! stack import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack ^
gyp ERR! stack SyntaxError: invalid syntax

maybe the default is python3 and so python2 install is not taking effect and maybe some kind of an override is needed?

@seth-priya
Copy link
Contributor

@simaishi I have also posted the build instructions for qpid-proton here in case they are needed https://github.com/ppc64le/build-scripts/tree/master/qpid-proton, but yes, the RPM should suffice.

@simaishi
Copy link
Contributor Author

@himdel Is there any plan to switch over to dart sass in Service UI?

@himdel
Copy link

himdel commented Jul 10, 2020

@simaishi Not easily. Currently used version of style-loader doesn't support dart-sass, while the current version of style-loader won't support includePaths, which SUI uses, and won't work with older css-loader. And newer css loader breaks for unclear reasons in SUI (likely too old webpack). (So, not as easy as ui-classic.)

So, this would essentially involve redoing how SUI does builds and css.

Cc @gtanzillo any info about the future of SUI? It has been (mostly) unmaintained for years, pending a decision on whether to rewrite SUI in react, integrate into ops UI, or deprecate. Sounds like it would be relevant.

@simaishi
Copy link
Contributor Author

maybe the default is python3 and so python2 install is not taking effect and maybe some kind of an override is needed?

How about setting NODE_GYP_FORCE_PYTHON to point to python2? (option 4 in https://github.com/nodejs/node-gyp#configuring-python-dependency)

@himdel
Copy link

himdel commented Jul 10, 2020

Yup, that should work for now. (As long as the machine has python2 installed.)

@seth-priya
Copy link
Contributor

seth-priya commented Jul 13, 2020

thanks @simaishi @himdel I am able to build all the images except the manageiq operator (need to install operator-sdk), though it is option 2 in https://github.com/nodejs/node-gyp#configuring-python-dependency that finally worked for me, I have the manageiq-ui-service build failure scenario replicated in a separate container with python3 and will dig more into why option 4 is not working

docker images | grep manageiq
manageiq/manageiq-ui-worker latest f62012180d36 About a minute ago 2.07GB
manageiq/manageiq-webserver-worker latest ce89c152aa11 About a minute ago 2.06GB
manageiq/manageiq-orchestrator latest 953b3743e9ab About a minute ago 1.86GB
manageiq/manageiq-base-worker latest 612cd93ebe3e About a minute ago 1.86GB
manageiq/manageiq-base latest 5b5fe544c5fd About a minute ago 1.85GB
manageiq/rpm_build latest 14061d926b55 About an hour ago 1.63GB

Just for reference, this is the change I made
Dir.chdir(ui_service_dir) do

  •      shell_cmd("npm config --global set python /usr/bin/python2.7")
         shell_cmd("yarn install")
    

I did not run into any other build issues

@simaishi
Copy link
Contributor Author

Updated the issue description based on @seth-priya's testing. I didn't add commenting out docker pull $RPM_BUILD_IMAGE as you can override that value to be any image you'd like to be.

@seth-priya
Copy link
Contributor

seth-priya commented Jul 14, 2020

@simaishi I am running into an issue in building the operator image, do you know if there a dependency on a specific operator sdk version?

go: github.com/operator-framework/[email protected] requires
github.com/ghodss/[email protected]: invalid pseudo-version: git fetch --unshallow -f https://github.com/ghodss/yaml in /home/Priya/go/pkg/mod/cache/vcs/5c75ad62eb9c289b6ed86c76998b4ab8c8545a841036e879d703a2bbc5fcfcea: exit status 128:
fatal: git fetch-pack: expected shallow list
Error: failed to build operator binary

@simaishi
Copy link
Contributor Author

simaishi commented Jul 14, 2020

We're using 0.15.2 on the build machine, but not sure what version works (or doesn't work) for other versions. I've seen that error before, but I don't recall what was causing it...

operator-sdk version: "v0.15.2", commit: "ffaf278993c8fcb00c6f527c9f20091eb8dd3352", go version: "go1.13.3 linux/amd64"

@bdunne
Copy link
Member

bdunne commented Jul 14, 2020

@bdunne Maybe we can take the Fedora qpid-proton ppc64le build and put into our yum repo? The qpid-proton package includes multiple RPMs, but what we require are: qpid-proton-c and qpid-proton-c-devel

https://koji.fedoraproject.org/koji/buildinfo?buildID=1479891

Done.

@seth-priya
Copy link
Contributor

ok, so version of git (being old) was the issue, the operator image builds successfully and confirmed that no changes needed for qpid-proton-c and qpid-proton-c-devel in the Dockerfile now, so these are the three things I have now

1- wmi for ppc64le missing in default repos (can be skipped for now)
2- node-sass install failure due to missing python2 (need to install python2 and set npm/gyp to use it)
3- special handling of sassc and unf_ext gems

to get a successful build, validating the changes now

@sethpeterson
Copy link

@seth-priya can you provide a link to your modifications to the Dockerfile to implement this?

@Fryguy
Copy link
Member

Fryguy commented Jul 15, 2020

or create a WIP PR with the changes? (so we can iterate on it and merge into upstream?)

@seth-priya
Copy link
Contributor

@sethpeterson @Fryguy I hit a run time issue trying to get manageiq up, am debugging that to see if I can get it resolved quickly and include in PR

/var/www/miq/vmdb /
bundler: failed to load command: rake (/opt/manageiq/manageiq-gemset/bin/rake)
Bundler::GemNotFound: Could not find ffi-1.13.1 in any of the sources
/opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/spec_set.rb:86:in block in materialize' /opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/spec_set.rb:80:in map!'
/opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/spec_set.rb:80:in materialize' /opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/definition.rb:170:in specs'
/opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/definition.rb:237:in specs_for' /opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/definition.rb:226:in requested_specs'
/opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/runtime.rb:101:in block in definition_method' /opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/runtime.rb:20:in setup'
/opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler.rb:149:in setup' /opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/setup.rb:20:in block in <top (required)>'
/opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/ui/shell.rb:136:in with_level' /opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/ui/shell.rb:88:in silence'
/opt/manageiq/manageiq-gemset/gems/bundler-2.1.4/lib/bundler/setup.rb:20:in `<top (required)>'
/

@seth-priya
Copy link
Contributor

but this is what I have at this point
seth-priya@4500fe0

@seth-priya
Copy link
Contributor

seth-priya commented Jul 15, 2020

Though all the images were created successfully. found a few errors in the build logs for manageiq-base python modules (notice that the build proceeds despite the error) , these errors seem to be all related to the cryptography module and an attempt to build it is being made on ppc64le, which in turn requires the following additional packages
libffi-devel
libxml2-devel
libxslt-devel
openssl-devel

c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
#include <ffi.h>
^~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /var/lib/manageiq/venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7pnmjwy5/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7pnmjwy5/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-4i5bjida/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-od0rm3ur/overlay --compile --install-headers /var/lib/manageiq/venv/include/site/python3.6/cffi Check the logs for full command output.
  ----------------------------------------
^[[0m^[[91mERROR: Command errored out with exit status 1: /var/lib/manageiq/venv/bin/python /var/lib/manageiq/venv/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-od0rm3ur/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=18.5' wheel 'cffi>=1.8,!=1.11.3; python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.
^[[0m/
Removing intermediate container 17f010141259
 ---> 56b7debdaf11
Step 24/28 : RUN chgrp -R 0 $APP_ROOT &&     chmod -R g=u $APP_ROOT

Post this build for PyNaCl was failing as it was unable to find "make". Another build is in progress …

I am still looking at the runtime ffi / gem issue above (not sure if it is related to the build ERRORS) but is this line in rpm_spec/manageiq.spec.in intentionally removing it @simaishi

%{__rm} -rf %{buildroot}/%{gemset_root}/gems/ffi-*/ext

@simaishi
Copy link
Contributor Author

simaishi commented Jul 15, 2020

I am still looking at the runtime ffi / gem issue above (not sure if it is related to the build ERRORS) but is this line in rpm_spec/manageiq.spec.in intentionally removing it @simaishi

%{__rm} -rf %{buildroot}/%{gemset_root}/gems/ffi-*/ext

Yes, that's intentionally being removed as we don't need "ext" directory once gem is installed and that particular one was causing rpm build failure.

Are you able to share more logs than what you've pasted for ffi/gem issue? Since the log is trimmed, I can't see where the error is happening.

@seth-priya
Copy link
Contributor

@simaishi I was able to resolve the errors, the ppc64le specific installation / fix for unf_ext was not placed correctly in my earlier changes. The python issues are resolved as well, but those needed additional dependencies to be installed in the manageiq-pods/images/manageiq-base/Dockerfile, I am trying to see if there is a better way to fix that one, but I am going ahead and raising a PR for this repo. The manageiq image and service is coming up correctly at my end now

56870e1f1d24 manageiq/manageiq:latest "/usr/bin/dumb-init …" About an hour ago Up About an hour 3000/tcp, 0.0.0.0:8443->443/tcp

@seth-priya
Copy link
Contributor

ManageIQ/manageiq-pods#585 WIP PR for manageiq-pods

@simaishi
Copy link
Contributor Author

simaishi commented Jul 16, 2020

Since the extra packages are needed only to install python modules, I'm thinking it's more appropriate to install them conditionally (only for ppc64le) in https://github.com/ManageIQ/manageiq-appliance-build/blob/master/kickstarts/partials/post/python_modules.ks.erb.

That's the file manageiq-base build uses and already has some packages needed for x86_64.

I think you'll run into the same issue if you try to build pcc64le appliance, so adding the packages there will help appliance build as well.

@seth-priya
Copy link
Contributor

thanks @simaishi yes, will do that and also I think that all these may not be required, for most part the problem can be resolved by installing the python3-cryptography package from the distro, but I will confirm

@seth-priya
Copy link
Contributor

Copying over the discussion comments from the above PR here for further discussion


@simaishi needed a bit of help to understand how can I test out this change in manageiq-appliance-build, I have not had to look at the appliance repositories before so just trying to figure out how this fits in , thanks in advance!

Since you really just need to test this particular change, I think the easiest way would be:
Clone manageiq-appliance-build
In that repo, run bin/kickstart_build.rb - this will create kickstart files, kickstarts/generated/base-*.ks
Use the generated kickstart file to create a VM from it
When OS is installed, you can check /root/anaconda-post.log on the VM and you should see some errors because of missing packages (should be same/similar to what you see when you run podified build). Then, you can just run modified python_modules.ks.erb on the machine to test out your changes.
There are things that can be removed from kickstart files for testing purpose, but it's probably easier if you just take what's generated. What's important is to have repos set up the same way and packages installed the same way.


@seth-priya
Copy link
Contributor

So @simaishi I tried the above, the kickstart files are generated but they have references to x86_64 repos, am I missing something?

cat base-libvirt.ks

build time repos - these repos are used to install the initial packages
repo --name=BaseOS --baseurl=http://mirror.centos.org/centos/8/BaseOS/x86_64/os/
repo --name=AppStream --baseurl=http://mirror.centos.org/centos/8/AppStream/x86_64/os/
repo --name=PowerTools --baseurl=http://mirror.centos.org/centos/8/PowerTools/x86_64/os/
repo --name=extras --baseurl=http://mirror.centos.org/centos/8/extras/x86_64/os/
repo --name=epel --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=epel-8&arch=x86_64 --excludepkgs=qpid-proton
repo --name=ovirt --mirrorlist=https://resources.ovirt.org/pub/yum-repo/mirrorlist-ovirt-4.4-el$releasever

@seth-priya
Copy link
Contributor

the ^^ is for the stalk https://github.com/ManageIQ/manageiq-appliance-build repo

@simaishi
Copy link
Contributor Author

@seth-priya Oh yeah... no one ever needed to build appliance for ppc64le, so x86_64 is hardcoded. You can probably try by just replacing them.

If you don't really care about building appliances for other reasons than testing for python modules and If there are more changes needed to get this working, we can probably skip testing in appliance for now.

@seth-priya
Copy link
Contributor

yes, thanks @simaishi for your inputs, so as far as I understand the podified version is more important for ppc64le for now, @sethpeterson and @Fryguy may want to add there thoughts here... and if so would you suggest that I keep the installation of the additional Python dependencies in manageiq-pods (but making that change Power specific)?

@simaishi
Copy link
Contributor Author

No, either way, I think it's better to put the dependencies into manageiq-appliance-build. It's just we can't test it for appliance build, but we can't build ppc64 appliances as is anyway.

@seth-priya
Copy link
Contributor

seth-priya commented Jul 20, 2020

thanks @simaishi , so I guess, its possible to still test the change in the manageiq-appliance-build file even by just re-building the podified version? I am trying to track that part of the code to see how .. BTW, I do observe that the build goes thru even without these additional dependencies (mainly cryptography) installed and all the images are built and I am able to get the manageiq image up as well, but I am guessing, there will be some impact somewhere of this
manageiq-base-buildlogs.txt

Detailed logs attached ^^

@seth-priya
Copy link
Contributor

and I do see that python3-cryptography is being installed successfully in the manageiq-base build logs (attached above), before the cryptography install failure, so unless there is some hard dependency on cryptography==2.6.1 (installed later), we might just be good?

@seth-priya
Copy link
Contributor

but yes the installation of the python modules is being aborted, so that will need to be addressed ...

@simaishi
Copy link
Contributor Author

simaishi commented Jul 20, 2020

@seth-priya The installation of python modules are not required to run manageiq app, but it's required for some ansible playbooks. I don't know the exact requirement of the python packages and their versions as what we specify in the python_modules.ks.erb is a known working set taken from somewhere else.

The podified build will take python_modules.ks.erb from manageiq-appliance-build repo, so running podified buid is how it can be tested for now. The yum repos used are slightly different between podified and appliance builds, but I doubt that matters for this particular part.

In manageiq-base/Dockerfile, you can replace

COPY --from=0 build/kickstarts/partials/post/python_modules.ks.erb /tmp/python_modules

with

COPY <your modified copy of python_modules.ks.erb> /tmp/python_modules

to test out your changes. Once you have a working copy, that can be put into manageiq-appliance-build repo.

The podified build not aborting on error is a separate bug and I will take a look.

@seth-priya
Copy link
Contributor

thanks @simaishi for the pointers, I have validated and raised the PR

@simaishi
Copy link
Contributor Author

All PRs are merged.

@seth-priya Please confirm ppc64le builds ok now.

@seth-priya
Copy link
Contributor

@simaishi confirmed that the ppc64le builds are all good now, thanks!

@simaishi
Copy link
Contributor Author

Great, thank you for the PRs!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

6 participants