MSUnmerged: Create a REST API for all RSEs' state

[todor-ivanov]

Impact of the new feature
MSUnmerged

Is your feature request related to a problem? Please describe.
While monitoring the overall work performed by the service tracked in the following issue [1], we've found that the single database query used for extracting the error counters for all the RSEs is quite useful. Provided this is a single call, It is quite logical to have it implemented as a REST API to the service, so that the information is easily exposed.

Describe the solution you'd like
Implement the following REST API:
https://cmsweb.cern.ch/ms-unmerged/data/info?rse=ALL_DOCS

Which should provide the output from the following query to MongoDB:

In [47]: filter = {}

In [48]: projection = {
    ...:     "_id": False,
    ...:     "name": True,
    ...:     "isClean": True,
    ...:     "counters": {
    ...:         "gfalErrors": True,
    ...:         "dirsToDelete": True,
    ...:         "dirsDeletedSuccess": True,
    ...:         "dirsDeletedFail": True}}

In [49]: list(msUnmergedCollProd.find(filter, projection))
Out[49]: 
[{'name': 'T1_US_FNAL_Disk',
  'isClean': True,
  'counters': {'dirsToDelete': 37,
   'dirsDeletedSuccess': 37,
   'dirsDeletedFail': 0,
   'gfalErrors': {}}},
   ...
   {'name': 'T2_ES_CIEMAT',
  'isClean': False,
  'counters': {'dirsToDelete': 514,
   'dirsDeletedSuccess': 512,
   'dirsDeletedFail': 2,
   'gfalErrors': {'No such file or directory': 2586}}}]

Describe alternatives you've considered
Using the query only from python for debugging.

Additional context
None

[todor-ivanov]

FYI @amaltaro

[todor-ivanov]

@amaltaro already made a really good suggestion regarding micro services REST APIs consistency. In order to stay in line with the similar interface already implemented for MStransferor we should use the following API + Auxiliary function:

https://github.com/dmwm/WMCore/wiki/ReqMgr2-apis#retrieve-a-workflow-transfer-document

WMCore/src/python/WMCore/ReqMgr/Service/Auxiliary.py

Line 173 in 90a3cb7

if subName.lower() == "all_docs":

[todor-ivanov]

While working on the issue I realized that the above method and class suggested by @amaltaro are not part of any of the micro services' APIs but they are part of the Reqmgr ones. And also this whole piece of code actually relates to the CouchDB interface. And we have no MongoDB interface developed and plugged into Reqmgr. So I would not create a new API to Reqmgr2 of the sort:
https://cmsweb.cern.ch/reqmgr2/data/unmerged/ALL_DOCS ,

only to be able to use this Auxiliaryclass (which has been developed around CouchDB interface and will need quite of a change to get linked to MongoDB as well), but I'd rather mimic this uri and use the already developed micro service's info REST API to get the information, which is already properly linked with MongoDB and very well tested. So the uri should look more like:
https://cmsweb.cern.ch/ms-unmerged/data/info?rse=ALL_DOCS

As stated into the issue description. And in case we decide we'd want an API from Reqmgr linked to the information stored in MongoDB, then we should create a separate issue, which should not be related to MSUnmerged only.

[amaltaro]

That's the way to go, Todor. I didn't mean to use the same implementation of ReqMgr Aux module, but only the same convention to retrieve all the documents (from the user point of view).

[todor-ivanov]

Got it. Thanks Alan! A PR is on its way already.