Invalid json values in StatisticsSenderService UDP packages to CMSSW Popularity service

[mrceyhun]

Affected service
As CMS Monitoring, our CMSSSW Popularity service(cms udp collector) collects UDP packages which are sent from CMSSW services. The url and the port of the service can be found in site configurations: i.e. /cvmfs/cms.cern.ch/SITECONF/T0_CH_CERN/JobConfig/site-local-config_EOS_PILOT_EC.xml, <statistics-destination name= part. And AFAIK, this CMSSW service sends the UDP packages: cms-sw/cmssw/Utilities/StorageFactory/src/StatisticsSenderService.cc

Bug description
In incoming UDP packages, some JSONs include invalid character for read_single_sigma field, see: "read_single_sigma":-nan. -nan has - prefix. For security reasons, full JSON is not shared.

Request and solutions
We can parse these values in our end but the best approach is to fix it in producer side.

[cmsbuild]

A new Issue was created by @mrceyhun Ceyhun.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

@mrceyhun Could you share me privately (e.g. via mattermost or email) the full JSON?

[makortel]

We had this problem earlier in #29412 and #35142, and a related problem in #36349. The NaNs were fixed with #35362, and the latter problem with #35505. Both fixes should have been backported to all relevant old release cycles.

I see we also added CMSSW version to the StatisticsSenderService in #37220, that was also backported to the same set of old release cycles.

@mrceyhun shared the message content to me privately, and it did not have the CMSSW version, so it sounds very likely to me that these jobs are using an old CMSSW release that does not have the fixes.

[makortel]

With further correlation attempts with @mrceyhun it seems plausible that the jobs used CMSSW versions in 10_6_X (part of the UL campaigns) earlier than 10_6_29_patch1 (which was the first to contain the fix #35505). So it seems there we can't really do anything.

(I also noticed that no 10_6_X release has been cut after the backport of #37220 was merged)

[makortel]

+1

[cmsbuild]

This issue is fully signed and ready to be closed.

[makortel]

@cmsbuild, please close

[makortel]

Written that, if any further investigation indicates that a CMSSW version with the fix still results in invalid JSON, let's reopen this issue (or open a new one).