Missing /etc/ && /bin/ && /src/ toplevel areas in the pypi packages

[todor-ivanov]

Impact of the bug
Python based deployment

Describe the bug
While working on the deployment of wmagent through pypi packages only, I've stumbled on the missing /etc, /bin and /src top level areas in the finally deployed python package. Even though /etc/ is listed for packaging here:

WMCore/setup_build.py

Line 122 in 31d6516

for toplevel in ['bin+', 'etc+']:

it is missing from the packaged source in the tar file.

Here is the actual content of the package: [1]
[1]

(WMCore.venv3) [user@unit02 wmagent-2.2.0.5]$ tree -L 2
.
├── bin
│   ├── acdcserver-tools
│   ├── adhoc-scripts
│   ├── attempt-to-patch.sh
...
│   ├── wmcore-new-config
│   └── wmcore-new-flow
├── LICENSE
├── MANIFEST.in
├── NOTICE
├── PKG-INFO
├── README.md
├── requirements.txt
├── setup_build.py
├── setup.cfg
├── setup_dependencies.py
├── setup.py
└── src
    ├── couchapps
    ├── css
    ├── javascript
    ├── python
    └── templates

The other missing bit is /bin/, but this time the directory is actually present in the pypi package: [1], but is not installed with pip. The bellow command returns no result:

(WMAgent.dock) [cmst1@unit01:/data/srv/wmagent/current]$ find / -name adhoc-scripts

Same with the /src area - missing in the finally deployed package, while present at the package tar file:

(WMAgent.dock) [cmst1@unit01:/data/srv/wmagent/current]$ find / -name couchapps

The impact of missing the two top level areas from the finally deployed python package is different, and for one of them is more severe than for the other:

• The lack of /etc in the package concerns only the missing agent config templates, which are a matter of a single download from source at build time for the docker image. While,
• The lack of /bin in the package is a blocker, because it is preventing any $manage command to be executed at the agent and is a matter of several files rather than a single template.
• The lack of /src in the the finally installed package is a blocker for the CouchDB initialization step at the agent. Here is the error that we observe:

wmcoreroot: /usr/local
develPath; /usr/local/lib/python3.8/site-packages/src/couchapps
Traceback (most recent call last):
  File "/data/srv/wmagent/current/install/wmagent/bin/wmagent-couchapp-init", line 294, in <module>
    installCouchApp(wmagentConfig.JobStateMachine.couchurl, fwjrDumpName, "FWJRDump")
  File "/data/srv/wmagent/current/install/wmagent/bin/wmagent-couchapp-init", line 54, in installCouchApp
    basePath = couchAppRoot(couchAppName)
  File "/data/srv/wmagent/current/install/wmagent/bin/wmagent-couchapp-init", line 34, in couchAppRoot
    raise OSError('Cannot find couchapp: %s' % couchapp)
OSError: Cannot find couchapp: FWJRDump
Done: Performing init_agent

A temporary workaround has been implemented in the CMSKubernetes repository for some parts of the problem (namely /etc && /bin), with the following commit: dmwm/CMSKubernetes@367370f , but this must go away ASAP.

How to reproduce it
Setup the wmagent pypi package with: pip install wmagent==2.2.0.2 and try to find the WMCore /bin/ and /etc/ directories or anything from their content.

to check the actual content of the package, just download the it directly from pypi (e.g. Pypi/wmagent-2.2.0.5.tar.gz), untar/zip it and look at the contents.

Expected behavior
The package to contain the /etc and /bin top level areas of WMCore.

Additional context and error message
These two issues are blockers for completing the python based deployment for the wmagent package:
#11564
#8797

Part of the following meta issue: #11314

[todor-ivanov]

Hi @amaltaro , @vkuznet ,
Since this issue became a complete blocker for #11564 I had to rise its QPrio today and finally fix this. The workarounds were piling up and became unbearable, actually even impossible once I've hit the missing couchapps from the /src area in the packge, because without the FWJRDump and few others, the agent cannot initialize its couchDB.

Just a heads up - I did find a solution.

Here follow the whole story:

In order to debug this, I had to do some manual packaging and disassembled deployment only of the python package from the tar.gz file locally. This procedure actually, would be quite helpful for resolving this issue as well: #11331.
Following this exact line in the instructions, I have built the package only locally, but did not upload it to pypi. Then I used directly the setup.py file provided from the package:

$ python3 -m venv pypi-deploy.venv
$ cd pypi-deploy.venv/
$ . ./bin/activate
(pypi-deploy.venv) $ git clone https://github.com/dmwm/WMCore.git
(pypi-deploy.venv) $ cd WMCore
(pypi-deploy.venv) $ sh tools/build_pypi_packages.sh wmagent  |tee wmagent_build.log 2>&1
(pypi-deploy.venv) $ cd dist/
(pypi-deploy.venv) $ tar -xzvf wmagent-2.2.1rc1
(pypi-deploy.venv) $ cd wmagent-2.2.1rc1
(pypi-deploy.venv) $ python setup.py install |tee wmagent_install.log 2>&1

• The missing /etc top level area inside the packaged tar.gz was due to a misconfiguration of the setup-dependencies.py file for the wmagent package:

  WMCore/setup_dependencies.py

  Lines 182 to 194 in 31d6516

  	'wmagent': {
  	'packages': ['WMCore.Agent+', 'WMCore.Algorithms+',
  	'WMCore.JobStateMachine', 'WMComponent+',
  	'WMCore.ThreadPool',
  	'WMCore.BossAir+', 'WMCore.Credential',
  	'WMCore.JobSplitting+', 'WMCore.ProcessPool',
  	'WMCore.Services+', 'WMCore.WMSpec+',
  	'WMCore.WMBS+', 'WMCore.ResourceControl+'],
  	'systems': ['wmc-web', 'wmc-database', 'global-workqueue', 'wmc-runtime'],
  	'statics': ['src/javascript/WMCore/WebTools/Agent',
  	'src/javascript/WMCore/WebTools/WMBS',
  	'src/javascript/external/graphael',
  	'src/templates/WMCore/WebTools/WMBS'],

  - they were not listed in the static area for the module. It was also affecting the missing FWJrDump couchapps and few others as well. Once I added those in the list, they immediately showed up in the final tar.gz file. Here follows the content of the fixed package:

(pypi-deploy.venv) $ cd wmagent-2.2.1rc1
(pypi-deploy.venv) $ tree -L 2
.
├── bin
│   ├── acdcserver-tools
│   ├── adhoc-scripts
...
├── deploy
...
│   ├── checkProxy.py
│   ├── env.sh
│   ├── renew_proxy.sh
│   ├── restartComponent.sh
│   ├── WMAgent.production
│   └── WMAgent.testbed
├── etc
...
│   └── WMAgentConfig.py
├── LICENSE
├── MANIFEST.in
├── NOTICE
├── PKG-INFO
├── README.md
├── requirements.txt
├── setup_build.py
├── setup.cfg
├── setup_dependencies.py
├── setup.py
└── src
    ├── couchapps
..
    │   ├── FWJRDump
    │   ├── GroupUser
...
    │   ├── WMStats
    │   ├── WMStatsAgent
    │   └── WorkQueue
    ├── css
    │   ├── external
    │   └── WMCore
    ├── javascript
    │   ├── external
    │   └── WMCore
    ├── python
    │   ├── PSetTweaks
    │   ├── Utils
    │   ├── wmagent.egg-info
    │   ├── WMComponent
    │   └── WMCore
    └── templates
        └── WMCore

• The missing /bin area from the finally installed package (even though present inside the tar.gz), was due to broken OS level dependencies of some of the packages the wmagent was trying to install. The default setup.py script was interrupting the installation and was reporting exit code 1, but maybe, due to the way how we build the Docker containers, the exit code from the pip install command has been ignored and also the output has been masked or redirected somewhere..., we have missed to notice the error and we are not interrupting the whole container build (I need to double check this later). Here follows the set errors from system level dependencies that were breaking the python setup.py install command inside a virtual env:

  • pip install pycrul: No such file or directory: 'curl-config'
  • src/pycurl.h:5:10: fatal error: Python.h: No such file or directory
  • pip install mysqlclient /bin/sh: line 1: mysql_config: command not found

  And here are the 3 OS packages that solved the issue:

  • yum install openssl-devel python-devel mysql-devel

• The missing /src/couchapps/* area was affected by both errors from above.

Here follows the final destination of the properly installed /bin and /src

(pypi-deploy.venv) $ cd ../

(pypi-deploy.venv) $ find . -name adhoc-scripts
./lib/python3.9/site-packages/wmagent-2.2.1rc1-py3.9.egg/bin/adhoc-scripts

(pypi-deploy.venv) $ find . -name FWJRDump
./lib/python3.9/site-packages/wmagent-2.2.1rc1-py3.9.egg/data/couchapps/FWJRDump

(pypi-deploy.venv) $ find . -name checkProxy.py
./lib/python3.9/site-packages/wmagent-2.2.1rc1-py3.9.egg/deploy/checkProxy.py

p.s. The PR for adding those static dirs to the final package is coming in a moment. The missing OS dependencies and error handling from the pip install wmagent command we should tackle in the pypi/wmagent Dockerfile at the CMSkubernetes repository.