Make cmsRun-compatible argparse parser

[kpedro88]

Currently, the "official" recommendation to enable command-line arguments in cmsRun configs is VarParsing. This is a very old and limited parser, with strange behavior (e.g. assigning to a VarParsing.multiplicity.list object actually appends to it). However, it does handle the difference in sys.argv content when a config is executed with python vs. with cmsRun.

Having a drop-in extension of argparse's parser would be a useful improvement to this situation. argparse offers significantly more functionality and is more familiar to incoming developers who have experience with python outside of CMSSW.

See also: #29338

attn: @davidlange6

[cmsbuild]

A new Issue was created by @kpedro88 Kevin Pedro.

@Dr15Jones, @perrotta, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[kpedro88]

Apparently there is already some support for this: https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideAboutPythonConfigFile#argparse

Encapsulating this behavior in a derived class like cmsArgumentParser (with tests) and providing more usage examples would be helpful for users.

[kpedro88]

Following up from Core Software meeting:

• Central production does not use the -p,--parameter-set argument:
  https://github.com/dmwm/WMCore/blob/e920d7224779ec06383969b1eca3de3eb677edd6/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py#L249-L277
  https://github.com/dmwm/WMCore/blob/e920d7224779ec06383969b1eca3de3eb677edd6/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py#L439
• CRAB does use the -p,--parameter-set argument, but it should be easy to remove:
  https://github.com/dmwm/CRABServer/blob/1ebaa01e363c4ed6c0e31564f1590548a7dfb962/scripts/CMSRunAnalysis.py#L649
• The CRAB twiki also has a note about passing command-line arguments that "double dashes break things", but not clear if this breaks things on the CRAB side (don't see why it should) or if it's just to prevent users from trying something that won't work with the current version of cmsRun. Needs followup.

With that in mind, as @Dr15Jones requested, I've created a draft PR that modifies cmsRun directly to support argparse: #42650

Some items to consider:

• The two interface changes are detailed in the PR description.
• These interface changes imply that tests in 15+ other packages need to be updated. Mostly easy to do, but not done yet. Maybe better to have a separate branch/PR with all of those test updates, and then test them together? (for review purposes)
• The combination of the above points suggests difficulty for backporting this change to other release cycles, as nice as the consistency would be...

Feedback welcome!

[kpedro88]

This is done now (#42650, #42798, #42823, #43042, #43102, #43116). @cms-sw/core-l2 can you sign and close?

(edit: and also dmwm/CRABServer#7888, dmwm/CRABClient#5210 (ref. dmwm/CRABClient#5208))

[makortel]

+core

[makortel]

@cmsbuild, please close

[cmsbuild]

This issue is fully signed and ready to be closed.

[kpedro88]

For future reference when considering similar interface changes: the site tests from computing should also be considered as a separate user/stakeholder. (They did use -p, even though production did not; they have now been updated for the new interface.)