CMS development nodes (cmsdev) migration to el8

[smuzaffar]

@cms-sw/all-l2 , as discussed in Core software meeting few weeks ago , we are in process of migrating existing CentOS7 (End of Life on June 30, 2024) based cmsdev nodes to AlmaLinux8 (EOL 01 Mar 2029) nodes. Migration to AlmaLinux8 will allow developers to directly use these nodes for CMSSW development for newer release cycles (e.g. 12.4.X and above where production arch is el8) instead of using cmssw-el8 singularity wrapper script. For any non-el8 arch you need to use cmssw-elX wrapper scripts.

We do not have enough resources to keep both CentOS7 and AlmaLinux8 development nodes alive. So we are planning to decommission existing cmsdev nodes[a] and replace them with new nodes [b](better/newer hardware and large/better Ceph Volume).

We will also copy existing /build directory to new nodes so that you can continue your development there. Note that if your existing cmssw area, on [a] nodes, is based on slc7 then you need to use cmssw-el7 on new cmsdev machines[b].

We already have two cmsdev41-42 el8 based nodes to replace existing cmsdev21-22. Please do not use these new nodes yet, we need to copy the existing data from cmsdev21-22 to these nodes first. In order to proceed with the migration we would like to

• disable login access to cmsdev21-22
• copy /build from cmsdev21-22 to cmsdev41-42
• enable login access to cmsdev41-42

Once we are happy with the new dev nodes then we can proceed with the rest. If no objections then we would like to disabled login access to cmsdev21-22 on Monday 18 March at 08h30 and hopefully allow access to new cmsdev41-42 nodes around 12h00 same day.

Please let us know if you have any questions/suggestions/concerns.

[a] cmsdev20-25 (also has alias cmsdev30-35): CentOS 7 based development nodes

• 16 cores
• 32GB memory
• 1TB Ceph Volume (io1 type: Max IOPS 500, Max Throughput 120MB)

[b] cmsdev40-45: AlmaLinux8 based development nodes

• 16 cores
• 32GB memory
• 2TB Ceph Volume (io3 type: Max IOPS 1/GB, Max Throughput 300MB)

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @smuzaffar.

@makortel, @antoniovilela, @Dr15Jones, @rappoccio, @smuzaffar, @sextonkennedy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[smuzaffar]

In core software meeting we decided to

• go with this migration plan
• ask developers to check if they can delete unused/unneeded stuff from existing cmsdev:/build directory. I already have sent email message to those who have some data in cmsdev20-25
• setup one cmsdev node with AlmaLinux9 for el9 based tests.

[smuzaffar]

@cms-sw/all-l2 , please note that cmsdev21-22 (alias cmsdev31-32) will be inaccessible at 08h30 today. These will be replaced by new AlmaLinux nodes cmsdev41-42.

[smuzaffar]

el8 migration for cmsdev21 (alias cmsdev31) and cmsdev22 (alias cmsdev32) have finished. These nodes are now replaced by new cmsdev41 and cmsdev42. Your data from /build directory is available on new nodes. You should be able to use new dev nodes. Please let us now if anything is missing on these new nodes.

[civanch]

@smuzaffar , I have tried cmsdev21 and it is working fine for me. It also seems to be faster for compilation. Thank you from preserving my user area.

Is it possible to add package "tkdiff" to the new node? Or it is incompatible with el8?

[smuzaffar]

Thanks @civanch for testing these new nodes.

About tkdiff, it is provided by an old obsolete/unmaintained package tkcvs which is not available for rhel8 and above ( https://src.fedoraproject.org/rpms/tkcvs). So I am afraid we can not install it

[civanch]

@smuzaffar , at AlmaLinux9 I am using "meld" - is it available for el8?

[smuzaffar]

@civanch , yes meld is available. I will install it on new cmsdev nodes

[iarspider]

We will now decommission of cmsdev31 and cmsdev32, and prepare for migrating cmsdev23 and cmsdev34.

[smuzaffar]

@iarspider , are new cmsdev43 (to repalce cmsdev23) and cmsdev44 (to replace cmsdev24/34) ready to use? If yes then how about bringing these to production on Monday 16th April at 9h00?

[iarspider]

The nodes are ready. We can run final sync over the weekend or on Monday morning,

[smuzaffar]

I would suggest that

• post a message here about the migration of cmsdev23/24.
• Disabled login access to cmsdev23/24 on Monday monring at 9h00. Make sure no one is connected.
• Do a final sync of /build
• Enable login access to cmsdev43/44

[iarspider]

el8 migration for cmsdev23 and cmsdev24 (alias cmsdev34) have finished. These nodes are now replaced by new cmsdev43 and cmsdev44. Your data from /build directory is available on new nodes. You should be able to use new dev nodes. Please let us now if anything is missing on these new nodes.

[smuzaffar]

We will now decommission cmsdev23 and cmsdev24 (alias cmsdev34) today. These has been replaced by cmsdev43 and cmsdev44 last week.

[iarspider]

We will perform migration of cmsdev30 and cmsdev25 nodes tomorrow, May 15th, after 11:00. These nodes will be replaced with cmsdev40 and cmsdev45. Your data from /build directory will be available on the new nodes

[smuzaffar]

cmsdev30 and cmsdev25 have been migrated to new AlmaLinux8 based cmsdev40 and cmsdev45 nodes. Migration of CentOS7 to AlmaLinux8 for cmsdev nodes has been completed. We will decommission cmsdev30 and cmsdev25 on Tuesday 21st of May.

[smuzaffar]

cmsdev30 and cmsdev25 has been decommissioned now.

[smuzaffar]

type doc

[smuzaffar]

closing this issue, cmsdevXX nodes are now running almalinux 8