Build a minimal PyPI based Docker image for T0Reqmon

[amaltaro]

Impact of the new feature
Overhaul the Dockerfile for T0Reqmon to be as small as possible. Container will use PyPI packages instead of the current RPM based deployment method.

Is your feature request related to a problem? Please describe.
Reqmon container is currently very large, which is not ideal for k8s.

Describe the solution you'd like
Build a minimal Docker image using Reqmon from PyPI, which is a base package for ReqMon and T0Reqmon.
Configuration files (services_config) also need to be provided.

Describe alternatives you've considered
NA

Additional context
Very similar to: #10405

[amaltaro]

Initial services_config configuration for the test branch have been provided in: https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/179

I can deploy secrets and the image with these changes (even though the relevant PyPi docker images are still missing).

[vkuznet]

Alan,
I do not understand your statement that relevant PyPi images are still missing, could you please clarify since I do have the following images in my docker:

registry.cern.ch/cmsweb/wmagent                 pypi-20221129-base   22f824169c66   13 days ago    1.51GB
registry.cern.ch/cmsweb/reqmon                  pypi-20221129-base   3172225cea89   13 days ago    1.19GB
registry.cern.ch/cmsweb/reqmgr2ms-unmerged      pypi-20221129-base   6ba66dc745dd   13 days ago    2.83GB
registry.cern.ch/cmsweb/reqmgr2ms-transferor    pypi-20221129-base   ad5b37d1266c   13 days ago    1.22GB
registry.cern.ch/cmsweb/reqmgr2ms-rulecleaner   pypi-20221129-base   321eeb397fdb   13 days ago    1.22GB
registry.cern.ch/cmsweb/reqmgr2ms-output        pypi-20221129-base   70f4f6d3bc6c   13 days ago    1.23GB
registry.cern.ch/cmsweb/reqmgr2ms-monitor       pypi-20221129-base   4acffda202b9   13 days ago    1.22GB
registry.cern.ch/cmsweb/reqmgr2ms               pypi-20221129-base   c421f90b8f6b   13 days ago    1.23GB
registry.cern.ch/cmsweb/reqmgr2                 pypi-20221129-base   5a9579bd5f7e   13 days ago    1.19GB
registry.cern.ch/cmsweb/global-workqueue        pypi-20221129-base   003b149b0e0a   13 days ago    1.3GB
registry.cern.ch/cmsweb/dmwm-base               pypi-20221129        532966e1983b   13 days ago    1.02GB

And, here is one specific link to reqmgr2ms-output

The t0_reqmon image cannot be build due to missing tag in pypi:

docker build -t veknet/t0_reqmon pypi/t0_reqmon
Sending build context to Docker daemon  3.072kB
Step 1/12 : FROM registry.cern.ch/cmsweb/dmwm-base:pypi-20221129
 ---> 532966e1983b
Step 2/12 : MAINTAINER Valentin Kuznetsov [email protected]
 ---> Using cache
 ---> dd8f3b01b409
Step 3/12 : ENV TAG=2.1.4
 ---> Using cache
 ---> e9142f63ff64
Step 4/12 : RUN pip install t0_reqmon==$TAG
 ---> Running in 7597dda80097
ERROR: Could not find a version that satisfies the requirement t0_reqmon==2.1.4 (from versions: none)
ERROR: No matching distribution found for t0_reqmon==2.1.4
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
The command '/bin/sh -c pip install t0_reqmon==$TAG' returned a non-zero code: 1

In fact, t0_reqmon does not exist at all in pypi repo, but there is a reqmon. Please advise which pypi package should be used for t0_reqmon. If it is reqmon, then pypi based reqmon image exists in registry, see here.

[amaltaro]

The t0_reqmon image cannot be build due to missing tag in pypi:

This is exactly what I meant. I know that all the other WMCore docker images are available, not t0reqmon (or t0_reqmon).

If you would like to build the t0_reqmon docker image, then please use the reqmon PyPi package for that. It's the same baseline code, only with configuration changes. Thanks

[vkuznet]

So you want to build identical image as reqmon? Why not to you reqmon image then in k8s manifest file? Sent from Proton Mail mobile

…

-------- Original Message --------

On Dec 12, 2022, 9:06 AM, Alan Malta Rodrigues wrote: > The t0_reqmon image cannot be build due to missing tag in pypi: This is exactly what I meant. I know that all the other WMCore docker images are available, not t0reqmon (or t0_reqmon). If you would like to build the t0_reqmon docker image, then please use the reqmon PyPi package for that. It's the same baseline code, only with configuration changes. Thanks — Reply to this email directly, [view it on GitHub](#11390 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAA6RUXM6KYM6N7PINXGZ73WM4WN5ANCNFSM6AAAAAAS3NSZKI). You are receiving this because you commented.Message ID: ***@***.***>

[amaltaro]

So you want to build identical image as reqmon? Why not to you reqmon image then in k8s manifest file?

Yes, that's a good idea. This way we only need to update the yaml (and I do not think we will need to create any aliases/symlinks):
https://github.com/dmwm/CMSKubernetes/blob/master/kubernetes/cmsweb/services/t0_reqmon.yaml#L90
https://github.com/dmwm/CMSKubernetes/blob/master/kubernetes/cmsweb/services/t0_reqmon-tasks.yaml#L75

Can you please add these changes to your already opened PR: dmwm/CMSKubernetes#1261
? @vkuznet

[amaltaro]

Keeping it open for a little longer until we confirm basic functionality is in place.

[amaltaro]

Basic functionality of these PyPi-based services - both t0_reqmon and t0_reqmon-tasks - have been tested with docker image label: pypi-20221215-base

In addition to the changes already linked above, the following was required as well:
(mount rucio-secrets and rename the docker image name for microservices, global workqueue and t0_reqmon)
dmwm/CMSKubernetes#1261
dmwm/CMSKubernetes#1283
dmwm/CMSKubernetes#1287

(fix to the run.sh script in the dmwm-base image)
dmwm/CMSKubernetes#1288

@vkuznet from my side we can close this out and follow up with a more detailed validation in #11354. What do you think?

[vkuznet]

fine with me.