UL: Corrections #72

mcremone · 2023-09-22T04:45:43Z

First of all, pull the current most up to date version from the master branch:

https://github.com/mcremone/decaf/blob/master/analysis/utils/corrections.py

then replace the corrections one by one with the ones recommended for UL. Please be mindful of a couple of things:

Use 'correctionlib': https://github.com/cms-nanoAOD/correctionlib
Use the coffea lookup tools to interface with correctionlib. If need help, ask Nick Smith.
Comment on each single correction, including links from where correction files have been taken etc.
Clean up the 'analysis/data' folder from non-UL files, keep only the ones that are used. With correctionlib, they should be reduced to just a bunch of json files

mcremone · 2023-09-23T20:34:58Z

An additional comment, we need to double check one by one against the corrections implemented by KIT. Still, we want to implement ours using correctionlib.

ParticleChef · 2023-10-01T06:18:29Z

Electron id, photon id, MET phi corrections, and pu weight are included at correction.py file with json files (https://gitlab.cern.ch/cms-nanoAOD/jsonpog-integration)
The electron trigger weight and reco sf should be included.

ParticleChef · 2023-10-01T06:26:14Z

For nlo ewk scale factor, the root files are made by monojet analysis. At the last meeting, we discussed that this scale factor can be used. https://github.com/ParticleChef/decaf/blob/master/analysis/utils/corrections.py#L220

mcremone · 2023-10-06T19:29:57Z

Electron id, photon id, MET phi corrections, and pu weight are included at correction.py file with json files (https://gitlab.cern.ch/cms-nanoAOD/jsonpog-integration)
The electron trigger weight and reco sf should be included.

Can you point me to the part of your code where these are used? Also, how about muon isolation weights? For what concerns trigger weight, most likely you're also missing single muon and MET trigger, am I right?

ParticleChef · 2023-10-09T06:21:34Z

Yes I'm not uploaded muon trigger, isolation weight yet.
And I pointed the part of current codes.
Electron ID sf : https://github.com/ParticleChef/decaf/blob/UL/analysis/utils/corrections.py#L32
Photon ID sf: https://github.com/ParticleChef/decaf/blob/UL/analysis/utils/corrections.py#L32
pu weight: https://github.com/ParticleChef/decaf/blob/UL/analysis/utils/corrections.py#L32
MET correction: https://github.com/ParticleChef/decaf/blob/UL/analysis/utils/corrections.py#L32

And I included the MET trigger and nlo sf from previous corrections.py file.
nlo sf: https://github.com/ParticleChef/decaf/blob/UL/analysis/utils/corrections.py#L154
met trigger: https://github.com/ParticleChef/decaf/blob/UL/analysis/utils/corrections.py#L15

The btag weight part is modifying now.

ParticleChef · 2023-10-26T01:51:46Z

I make quick test with json file and btageff.merged file existed.

# generate 20 dummy jet features
jet_pt    = np.random.exponential(50., 15) 
jet_eta   = np.random.uniform(0.0, 2.4, 15) 
jet_flav  = np.random.choice([0, 4, 5], 15) 
jet_discr = np.random.uniform(0.0, 1.0, 15) 

# separate light and b/c jets
light_jets = np.where(jet_flav == 0)
bc_jets    = np.where(jet_flav != 0)

btag = load('hists/btageff2017.merged')
bpass = btag[tagger].integrate('dataset').integrate('wp',workingpoint).integrate('btag', 'pass').values()[()]
ball = btag[tagger].integrate('dataset').integrate('wp',workingpoint).integrate('btag').values()[()]
nom = bpass / np.maximum(ball, 1.) 
eff = lookup_tools.dense_lookup.dense_lookup(nom, [ax.edges() for ax in btag[tagger].axes()[3:]])

btvjson = correctionlib.CorrectionSet.from_file('data/BtagSF/'+year+'_UL/btagging.json.gz')
sf_nom = btvjson["deepJet_comb"].evaluate('central','M', jet_flav[bc_jets], jet_eta[bc_jets], jet_pt[bc_jets])
print('sf_nom: ', sf_nom, len(sf_nom))

def P(eff):
    weight = eff.ones_like()
    weight[istag] = eff[istag]
    weight[~istag] = (1 - eff[~istag])
    return weight.prod()

eff = eff(jet_pt, jet_eta, jet_flav)                                                                                                                       
print('extract eff:', eff, len(eff))

eff_data_nom  = np.minimum(1., sf_nom*eff)
nnom = P(eff_data_nom)/P(eff)
print('P(eff_data_nom)/P(eff)', nnom)

I printed the values and I got error like this

sf_nom:  [0.94694163 0.95233112 0.9551299  0.95522698 0.95875001 0.94341749
 0.95456105 0.94572292 0.94499175 0.95435803 0.94332464] 11
extract eff: [0.9375     0.9375     0.9375     0.9375     0.9375     0.61748634
 0.9375     0.9375     0.91052632 0.91052632 0.91052632 0.91052632
 0.9375     0.9375     0.61748634] 15
Traceback (most recent call last):
  File "utils/cortest.py", line 89, in <module>
    eff_data_nom  = np.minimum(1., sf_nom*eff)
ValueError: operands could not be broadcast together with shapes (11,) (15,)

Which part should I fix to solve this error?

mcremone · 2023-10-28T16:49:14Z

To avoid this shape mismatch you can use real data/MC in the test. My suggestion is that we finish first implementing all corrections with correctionlib (when possible). I'll then do a quick review of the code and then we structure a test.

ParticleChef · 2023-11-02T06:08:45Z

I finished the modifying btag. How you implement the jec? Other than jec, I modified all corrections I need.

mcremone · 2023-11-02T14:18:19Z

For jet you can follow this:

https://github.com/nsmith-/boostedhiggs/blob/master/boostedhiggs/build_jec.py

ParticleChef · 2023-11-08T07:21:26Z

I update the corrections.py and jet energy correction files.

ParticleChef · 2023-11-13T09:05:37Z

When run the correction.py, the error is accured at import uproot_methods

Traceback (most recent call last):
  File "utils/corrections.py", line 6, in <module>
    import uproot, uproot_methods
  File "/uscms/home/jhong/.local/lib/python3.6/site-packages/uproot_methods/__init__.py", line 5, in <module>
    from uproot_methods.classes.TVector2 import TVector2, TVector2Array
  File "/uscms/home/jhong/.local/lib/python3.6/site-packages/uproot_methods/classes/TVector2.py", line 8, in <module>
    import awkward.array.jagged
ModuleNotFoundError: No module named 'awkward.array'

Instead of this, update error lines to uproot3.

And separate the '2016' to '2016preVFP' and '2016postVFP' at btag part.
(https://github.com/ParticleChef/decaf/blob/UL/analysis/utils/corrections.py#L465)
(https://github.com/ParticleChef/decaf/blob/UL/analysis/utils/common.py#L19)

mcremone · 2023-11-13T16:21:42Z

Hi think you want to do the other way around, which means using the latest awkward version, changing the code lines to use what the latest awkward version wants you to use.

ParticleChef · 2023-11-14T02:09:36Z

Then, is there no need to change the version of Awkward?

mcremone · 2023-11-14T02:17:47Z

In general you want to use the latest version of everything, both awkward and uproot. If we need to change the code a bit to adjust to the format the new versions may want, that's what I would do.

ParticleChef · 2023-11-16T03:08:21Z

The current correction.py works in awkward version 1.9.0. I checked my current setup and latest version of some module.

        (current / latest)
awkward ( 1.9.0 / 2.4.10 )
uproot  ( 4.3.7 / 5.1.2 )
uproot3 ( 3.14.4 / 3.14.4 )
numpy  ( 1.17.0 / 1.26.0 )

Is it okay to change the version in current coffea (0.7.12)?

mcremone · 2023-11-16T04:00:19Z

where the current version automatically installed when you installed coffea 0.7.12?

ParticleChef · 2023-11-16T05:00:54Z

I forgot which version is installed when I installed coffea 0.7.12. All module are installed at /uscms/home/jhong/.local/lib/python3.6/site-packages/

mcremone · 2023-11-21T17:02:39Z

I think that if you didn't upgrade packages by hand, those are the versions that coffea installed by itself. I wouldn't touch them then, but in the correction.py code, wherever you are using using uproot3, use uproot instead. If this makes the code crash, then we need to understand why.

mcremone · 2023-12-08T19:53:08Z

@ParticleChef were you able to use uproot instead of uproot3? Besides that I don't think that this needs more work.

mcremone · 2023-12-08T19:57:22Z

Actually, we need also to implement the UL ttbar corrections:

https://github.com/mcremone/decaf/blob/master/analysis/utils/corrections.py#L352-L353

To be found here:

https://twiki.cern.ch/twiki/bin/view/CMS/TopPtReweighting

mcremone · 2023-12-13T05:06:27Z

@ParticleChef were you able to use uproot instead of uproot3? Besides that I don't think that this needs more work.

@ParticleChef any news on this?

ParticleChef · 2023-12-14T03:05:45Z

Hi. I I checked the uproot, uproot3 and uproot_methods.
The firstly error is accurred in uproot_methods. The error is like this:

[jhong@cmslpc175 analysis]$ python utils/corrections.py 
Traceback (most recent call last):
  File "utils/corrections.py", line 4, in <module>
    import uproot, uproot_methods
  File "/uscms/home/jhong/.local/lib/python3.6/site-packages/uproot_methods/__init__.py", line 5, in <module>
    from uproot_methods.classes.TVector2 import TVector2, TVector2Array
  File "/uscms/home/jhong/.local/lib/python3.6/site-packages/uproot_methods/classes/TVector2.py", line 8, in <module>
    import awkward.array.jagged
ModuleNotFoundError: No module named 'awkward.array'

And using uproot without uproot_methods, the lookup_tools has error:

Traceback (most recent call last):
  File "utils/corrections.py", line 28, in <module>
    get_met_trig_weight[year] = lookup_tools.dense_lookup.dense_lookup(met_trig_hist.values, met_trig_hist.edges)
AttributeError: 'Model_TH1F_v1' object has no attribute 'edges'

Everything works well when I change all uproot to uproot3 (I don't include uproot_methods)
In all lines using uproot.open, uproot changed to uproot3. One of the line is https://github.com/ParticleChef/decaf/blob/UL/analysis/utils/corrections.py#L20

mcremone · 2023-12-14T22:12:36Z

Actually, we need also to implement the UL ttbar corrections:

https://github.com/mcremone/decaf/blob/master/analysis/utils/corrections.py#L352-L353

To be found here:

https://twiki.cern.ch/twiki/bin/view/CMS/TopPtReweighting

@alejands can you look into this?

mcremone · 2023-12-14T22:14:59Z

I forgot to mention that we also need to updated these corrections:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L311-L320

A good place to start would be asking the boosted Higgs team, or digging into their code:

https://github.com/nsmith-/boostedhiggs/tree/master/boostedhiggs

@alejands @ParticleChef

ParticleChef · 2023-12-18T08:17:11Z

I modified the btageff.py file for making btageff merged file used in corrections.py file.
https://github.com/ParticleChef/decaf/blob/forBtagw/analysis/processors/btageff.py

Does it need any other process other than reduce.py and merge.py to make btageff.merged files?

mcremone · 2023-12-18T15:14:56Z

@ParticleChef I really have a strong preference for adding boolean as attributes of objects. For example, here:

https://github.com/ParticleChef/decaf/blob/forBtagw/analysis/processors/btageff.py#L52

I really prefer this:

https://github.com/mcremone/decaf/blob/master/analysis/processors/btageff.py#L49

mcremone · 2023-12-20T11:33:21Z

@ParticleChef I really have a strong preference for adding boolean as attributes of objects. For example, here:

https://github.com/ParticleChef/decaf/blob/forBtagw/analysis/processors/btageff.py#L52

I really prefer this:

https://github.com/mcremone/decaf/blob/master/analysis/processors/btageff.py#L49

@ParticleChef can you open a separate issue for this? Also in this case we should move from coffea.hist to hist following these instructions:

scikit-hep/coffea#705

alejands · 2024-01-23T20:55:36Z

Actually, we need also to implement the UL ttbar corrections:

https://github.com/mcremone/decaf/blob/master/analysis/utils/corrections.py#L352-L353

To be found here:

https://twiki.cern.ch/twiki/bin/view/CMS/TopPtReweighting

After going through the twiki above and going around some TOP PAG twikis to double check, it appears that no updates have been made to the top pt reweighting function for data-NLO (data/POWHEG+Pythia8). The recommendation still matches our code.

decaf/analysis/utils/corrections.py

Lines 308 to 309 in ed33cc1

    
           def get_ttbar_weight(pt): 
        
               return np.exp(0.0615 - 0.0005 * np.clip(pt, 0, 800))

I did notice this line in the twiki...

New plots with full Run 2 data and different predictions are expected to replace these soon (08/2020).

alejands · 2024-01-23T20:59:07Z

I was able to update corrections.py script to use the uproot package rather than uproot3. I'll be adding my updates in this PR:

Version compatibility updates to corrections.py #87

alejands · 2024-01-23T21:04:52Z

I noticed these output filenames were changed by @ParticleChef, presumably while testing:

decaf/analysis/utils/ids.py

Line 347 in ed33cc1

save(ids, "data/test_ids.coffea")

decaf/analysis/utils/corrections.py

Line 631 in ed33cc1

save(corrections, 'data/testcorrections.coffea')

Should these be changed back or left as is?

alejands · 2024-01-23T23:23:40Z

Output filenames above updated in commit 87ddf88.

mcremone · 2024-01-24T00:00:00Z

I forgot to mention that we also need to updated these corrections:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L311-L320

A good place to start would be asking the boosted Higgs team, or digging into their code:

https://github.com/nsmith-/boostedhiggs/tree/master/boostedhiggs

@alejands @ParticleChef

Here is the way to implement the new corrections:

https://github.com/jennetd/hbb-coffea/blob/master/boostedhiggs/corrections.py#L25-L47

@alejands you can take msdcorr.json from here:

https://github.com/jennetd/hbb-coffea/blob/master/boostedhiggs/data/msdcorr.json

alejands · 2024-01-30T20:31:15Z

The PR has been updated with the new msd corrections (commit 7471585).

The new get_msd_corr() function takes in fatjet coffea objects rather than pt and eta awkward arrays. The scripts in analysis/processors that call this function are updated accordingly, but have not been tested since the compatibility for these scripts has not been updated.

mcremone · 2024-02-11T21:04:04Z

I had a look and this still needs work.

NLO corrections.

I noticed that only EWK corrections are implemented:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L290-L305

@ParticleChef can you confirm that this is because samples are already NLO in QCD?

Also, systematic variations need to be implemented. They can be taken from here:

https://github.com/mcremone/decaf/blob/master/analysis/utils/corrections.py#L248-L350

JERC

This won't work unfortunately:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L602-L621

We need to implement this:

https://github.com/nsmith-/boostedhiggs/blob/master/boostedhiggs/build_jec.py

I'll open a new issue for this.

mcremone · 2024-02-19T05:58:18Z

I had a look and this still needs work.

NLO corrections.

I noticed that only EWK corrections are implemented:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L290-L305

@ParticleChef can you confirm that this is because samples are already NLO in QCD?

Also, systematic variations need to be implemented. They can be taken from here:

https://github.com/mcremone/decaf/blob/master/analysis/utils/corrections.py#L248-L350

JERC

This won't work unfortunately:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L602-L621

We need to implement this:

https://github.com/nsmith-/boostedhiggs/blob/master/boostedhiggs/build_jec.py

I'll open a new issue for this.

I took care of that. JERCs need to be updated to UL though:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L658-L799

@alejands can you check which are the recommendations?

ParticleChef · 2024-02-19T08:57:33Z

I had a look and this still needs work.

NLO corrections.

I noticed that only EWK corrections are implemented:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L290-L305

@ParticleChef can you confirm that this is because samples are already NLO in QCD?

Also, systematic variations need to be implemented. They can be taken from here:

https://github.com/mcremone/decaf/blob/master/analysis/utils/corrections.py#L248-L350

Yes, I know that KIT people generated those samples in NLO QCD so NLO QCD corrections are not applied additionally.

mcremone · 2024-02-22T22:24:15Z

I had a look and this still needs work.

NLO corrections.

I noticed that only EWK corrections are implemented:
https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L290-L305
@ParticleChef can you confirm that this is because samples are already NLO in QCD?
Also, systematic variations need to be implemented. They can be taken from here:
https://github.com/mcremone/decaf/blob/master/analysis/utils/corrections.py#L248-L350

JERC

This won't work unfortunately:
https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L602-L621
We need to implement this:
https://github.com/nsmith-/boostedhiggs/blob/master/boostedhiggs/build_jec.py
I'll open a new issue for this.

I took care of that. JERCs need to be updated to UL though:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L658-L799

@alejands can you check which are the recommendations?

@alejands ping on this.

mcremone · 2024-02-22T22:24:48Z

I had a look and this still needs work.

NLO corrections.

I noticed that only EWK corrections are implemented:
https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L290-L305
@ParticleChef can you confirm that this is because samples are already NLO in QCD?
Also, systematic variations need to be implemented. They can be taken from here:
https://github.com/mcremone/decaf/blob/master/analysis/utils/corrections.py#L248-L350

Yes, I know that KIT people generated those samples in NLO QCD so NLO QCD corrections are not applied additionally.

Good, but I believe we still want systematic variations. I re-implemented those.

mcremone · 2024-02-22T22:26:29Z

Need to fix this:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L475-L487

@mcremone

ParticleChef · 2024-02-23T07:18:26Z

I'm modifying btagging weight on corrections file with btagging json file. https://github.com/ParticleChef/decaf/blob/forBtagw/analysis/utils/correctionsBTseperate.py#L35
I checked it works with 2018 efficiency file we produced.
But it should be checked if it works on setup of new version and also up and down case should be checked.
https://github.com/ParticleChef/decaf/blob/forBtagw/analysis/utils/correctionsBTseperate.py#L53

mcremone · 2024-02-23T18:13:57Z

I'm modifying btagging weight on corrections file with btagging json file. https://github.com/ParticleChef/decaf/blob/forBtagw/analysis/utils/correctionsBTseperate.py#L35 I checked it works with 2018 efficiency file we produced. But it should be checked if it works on setup of new version and also up and down case should be checked. https://github.com/ParticleChef/decaf/blob/forBtagw/analysis/utils/correctionsBTseperate.py#L53

@ParticleChef I strongly suggest you use the btagging weight calculation I implemented in the latest version of corrections.py, there were a lot of things I fixed. Also, the version you have, as well as the current version of corrections.py, won't work with the new hist format, as I was commenting before. In order to fix that, this part should be changed:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L475-L487

mcremone · 2024-02-23T18:21:00Z

I'm modifying btagging weight on corrections file with btagging json file. https://github.com/ParticleChef/decaf/blob/forBtagw/analysis/utils/correctionsBTseperate.py#L35
I checked it works with 2018 efficiency file we produced.
But it should be checked if it works on setup of new version and also up and down case should be checked.
https://github.com/ParticleChef/decaf/blob/forBtagw/analysis/utils/correctionsBTseperate.py#L53

Also, on a separate note, I don't know which btageff2018.merged file you are using here:

https://github.com/ParticleChef/decaf/blob/forBtagw/analysis/utils/correctionsBTseperate.py#L42

If you are using the one was already in decaf that was obtained with pre-UL samples. If you generated you own using the KIT UL QCD samples, that wouldn't work either because, as of yesterday, the btageff processor was kind of incorrect. Also, a lot of KIT UL QCD root files are corrupted, and they make you coffea jobs crash. That means that even if you managed to run the incorrect processor over them, you're missing a lot of data that wasn't processed.

ParticleChef · 2024-02-28T04:35:42Z

I checked the new version of btag weight at correction file on your area today. I will use new version. And the btageff2018.merged file I used was generated by previous version of btageff.py file. So I tried again and got btageff file with latest version.

I have one question when draw the 2D plot of efficiency.
The hist stored in btageff2018.merged is stored with dictionary type that the keys are name of reduced file:

deepflav = hists['deepflav']
print(deepflav)

>>
{'TTTo2L2Nu_TuneCP5_13TeV-powheg-pythia8.reduced': Hist(
  StrCategory(['loose', 'medium', 'tight'], growth=True),
  StrCategory(['pass', 'fail'], growth=True),
  IntCategory([0, 4, 5, 6]),
  Variable([20, 30, 50, 70, 100, 140, 200, 300, 600, 1000]),
  Variable([0, 1.4, 2, 2.5]),
  storage=Double()) # Sum: 1111061577.0 (1111114875.0 with flow), 'TTToHadronic_TuneCP5_13TeV-powheg-pythia8.reduced': Hist(
  StrCategory(['loose', 'medium', 'tight'], growth=True),
  StrCategory(['pass', 'fail'], growth=True),
  IntCategory([0, 4, 5, 6]),
  Variable([20, 30, 50, 70, 100, 140, 200, 300, 600, 1000]),
  Variable([0, 1.4, 2, 2.5]),
  storage=Double()) # Sum: 4569717606.0 (4569910719.0 with flow), 'TTToSemiLeptonic_TuneCP5_13TeV-powheg-pythia8.reduced': Hist(
  StrCategory(['loose', 'medium', 'tight'], growth=True),
  StrCategory(['pass', 'fail'], growth=True),
  IntCategory([0, 4, 5, 6]),
  Variable([20, 30, 50, 70, 100, 140, 200, 300, 600, 1000]),
  Variable([0, 1.4, 2, 2.5]),
  storage=Double()) # Sum: 4890351531.0 (4890566991.0 with flow)}

So it should be used like this:

deepflav = hists['deepflav']['TTTo2L2Nu_TuneCP5_13TeV-powheg-pythia8.reduced']
loose_pass = deepflav[{'wp': 'tight', 'btag':'pass'}]

Do you have any idea to merge all dataset?

mcremone · 2024-02-28T07:54:14Z

I checked the new version of btag weight at correction file on your area today. I will use new version. And the btageff2018.merged file I used was generated by previous version of btageff.py file. So I tried again and got btageff file with latest version.

I have one question when draw the 2D plot of efficiency. The hist stored in btageff2018.merged is stored with dictionary type that the keys are name of reduced file:

deepflav = hists['deepflav']
print(deepflav)

>>
{'TTTo2L2Nu_TuneCP5_13TeV-powheg-pythia8.reduced': Hist(
  StrCategory(['loose', 'medium', 'tight'], growth=True),
  StrCategory(['pass', 'fail'], growth=True),
  IntCategory([0, 4, 5, 6]),
  Variable([20, 30, 50, 70, 100, 140, 200, 300, 600, 1000]),
  Variable([0, 1.4, 2, 2.5]),
  storage=Double()) # Sum: 1111061577.0 (1111114875.0 with flow), 'TTToHadronic_TuneCP5_13TeV-powheg-pythia8.reduced': Hist(
  StrCategory(['loose', 'medium', 'tight'], growth=True),
  StrCategory(['pass', 'fail'], growth=True),
  IntCategory([0, 4, 5, 6]),
  Variable([20, 30, 50, 70, 100, 140, 200, 300, 600, 1000]),
  Variable([0, 1.4, 2, 2.5]),
  storage=Double()) # Sum: 4569717606.0 (4569910719.0 with flow), 'TTToSemiLeptonic_TuneCP5_13TeV-powheg-pythia8.reduced': Hist(
  StrCategory(['loose', 'medium', 'tight'], growth=True),
  StrCategory(['pass', 'fail'], growth=True),
  IntCategory([0, 4, 5, 6]),
  Variable([20, 30, 50, 70, 100, 140, 200, 300, 600, 1000]),
  Variable([0, 1.4, 2, 2.5]),
  storage=Double()) # Sum: 4890351531.0 (4890566991.0 with flow)}

So it should be used like this:

deepflav = hists['deepflav']['TTTo2L2Nu_TuneCP5_13TeV-powheg-pythia8.reduced']
loose_pass = deepflav[{'wp': 'tight', 'btag':'pass'}]

Do you have any idea to merge all dataset?

The new version of corrections.py already ingests the new format and merges everything:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L487-L498

ParticleChef · 2024-03-05T04:48:44Z

I checked quickly that "deepJet_comb" has only 4 and 5 for hadron flavor in json file. So I should use "deepJet_incl" for light sf (hadron flavor 0).
I think this also cause the error. It is solved already?
https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L498

mcremone · 2024-03-05T15:13:27Z

It depends on what you are loading here:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L475-L476

Also, which error are you referring to?

mcremone · 2024-03-05T15:24:43Z

This should fix it:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L474-L485

ParticleChef · 2024-03-07T08:44:19Z

I updated the codes and I got another error when compile the corrections.py file.
from correctionlib import convert has some issue:

[jhong@cmslpc115 analysis]$ python3 utils/corrections.py 
Traceback (most recent call last):
  File "utils/corrections.py", line 3, in <module>
    from correctionlib import convert
  File "/uscms/home/jhong/.local/lib/python3.8/site-packages/correctionlib/convert.py", line 19, in <module>
    from .schemav2 import (
  File "/uscms/home/jhong/.local/lib/python3.8/site-packages/correctionlib/schemav2.py", line 37, in <module>
    class Variable(Model):
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py3-pydantic/1.8/lib/python3.8/site-packages/pydantic/main.py", line 287, in __new__
    fields[ann_name] = ModelField.infer(
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py3-pydantic/1.8/lib/python3.8/site-packages/pydantic/fields.py", line 392, in infer
    return cls(
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py3-pydantic/1.8/lib/python3.8/site-packages/pydantic/fields.py", line 327, in __init__
    self.prepare()
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py3-pydantic/1.8/lib/python3.8/site-packages/pydantic/fields.py", line 432, in prepare
    self._type_analysis()
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/py3-pydantic/1.8/lib/python3.8/site-packages/pydantic/fields.py", line 532, in _type_analysis
    if issubclass(origin, Tuple):  # type: ignore
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/python3/3.8.2-bcolbf/lib/python3.8/typing.py", line 771, in __subclasscheck__
    return issubclass(cls, self.__origin__)
TypeError: issubclass() arg 1 must be a class

mcremone · 2024-03-07T17:05:37Z

To address this I have already changed the setup file:

https://github.com/mcremone/decaf/blob/UL/setup.sh#L8

mcremone · 2024-03-10T18:21:16Z

It needed a lot of work, but now the b-tagging class works.

ParticleChef · 2024-03-22T11:46:27Z

For nlo scale factor in correction.py, the method using extractor has no error.
Previous method occurs error with ValueError: object of too small depth for desired array from the line `nlo_ewk = get_nlo_ewk_weight['w'](ak.max(genWs.pt, axis=1))

nlo_ewk_hists = {
        'dy': ["* * data/vjets_SFs/merged_kfactors_zjets.root"],
        'w': ["* * data/vjets_SFs/merged_kfactors_wjets.root"],
        'z': ["* * data/vjets_SFs/merged_kfactors_zjets.root"],
        'a': ["* * data/vjets_SFs/merged_kfactors_gjets.root"],
}    
get_nlo_ewk_weight = {} 
for p in ['dy','w','z','a']:
    print(nlo_ewk_hists[p])
    ext = extractor()
    ext.add_weight_sets(nlo_ewk_hists[p])
    ext.finalize()
    get_nlo_ewk_weight[p] = ext.make_evaluator()["kfactor_monojet_ewk"]

mcremone · 2024-03-22T16:32:48Z

@ParticleChef There should not be any nlo_ewk = get_nlo_ewk_weight['w'](ak.max(genWs.pt, axis=1)) line in your processor anymore. I made modifications a couple of days ago and now it looks like this:

https://github.com/mcremone/decaf/blob/UL/analysis/processors/hadmonotopv2.py#L671-L676

Also, is extractor a functionality from coffea.lookup_tools? If yes, since you're modifying this part, it would be good to use correctionlib tools. You can follow this as an example:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L23-L39

mcremone · 2024-04-01T00:40:03Z

@ParticleChef There should not be any nlo_ewk = get_nlo_ewk_weight['w'](ak.max(genWs.pt, axis=1)) line in your processor anymore. I made modifications a couple of days ago and now it looks like this:

https://github.com/mcremone/decaf/blob/UL/analysis/processors/hadmonotopv2.py#L671-L676

Also, is extractor a functionality from coffea.lookup_tools? If yes, since you're modifying this part, it would be good to use correctionlib tools. You can follow this as an example:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L23-L39

I took care of it:

https://github.com/mcremone/decaf/blob/UL/analysis/utils/corrections.py#L369-L507

alejands self-assigned this Dec 13, 2023

mcremone assigned ParticleChef Dec 16, 2023

ParticleChef mentioned this issue Jan 9, 2024

UL: btageff Processor #85

Closed

alejands linked a pull request Jan 23, 2024 that will close this issue

Version compatibility updates to corrections.py #87

Merged

mcremone mentioned this issue Feb 11, 2024

UL: build_jec.py #88

Closed

UL: Corrections #72

UL: Corrections #72

Comments

mcremone commented Sep 22, 2023 • edited Loading

mcremone commented Sep 23, 2023

ParticleChef commented Oct 1, 2023

ParticleChef commented Oct 1, 2023

mcremone commented Oct 6, 2023

ParticleChef commented Oct 9, 2023

ParticleChef commented Oct 26, 2023

mcremone commented Oct 28, 2023

ParticleChef commented Nov 2, 2023

mcremone commented Nov 2, 2023

ParticleChef commented Nov 8, 2023

ParticleChef commented Nov 13, 2023

mcremone commented Nov 13, 2023

ParticleChef commented Nov 14, 2023

mcremone commented Nov 14, 2023

ParticleChef commented Nov 16, 2023 • edited Loading

mcremone commented Nov 16, 2023

ParticleChef commented Nov 16, 2023

mcremone commented Nov 21, 2023

mcremone commented Dec 8, 2023

mcremone commented Dec 8, 2023 • edited Loading

mcremone commented Dec 13, 2023

ParticleChef commented Dec 14, 2023 • edited Loading

mcremone commented Dec 14, 2023

mcremone commented Dec 14, 2023

ParticleChef commented Dec 18, 2023

mcremone commented Dec 18, 2023

mcremone commented Dec 20, 2023

alejands commented Jan 23, 2024 • edited Loading

alejands commented Jan 23, 2024 • edited Loading

alejands commented Jan 23, 2024

alejands commented Jan 23, 2024

mcremone commented Jan 24, 2024

alejands commented Jan 30, 2024

mcremone commented Feb 11, 2024

mcremone commented Feb 19, 2024

ParticleChef commented Feb 19, 2024

mcremone commented Feb 22, 2024

mcremone commented Feb 22, 2024

mcremone commented Feb 22, 2024

ParticleChef commented Feb 23, 2024

mcremone commented Feb 23, 2024

mcremone commented Feb 23, 2024

ParticleChef commented Feb 28, 2024

mcremone commented Feb 28, 2024 • edited Loading

ParticleChef commented Mar 5, 2024

mcremone commented Mar 5, 2024

mcremone commented Mar 5, 2024

ParticleChef commented Mar 7, 2024

mcremone commented Mar 7, 2024

mcremone commented Mar 10, 2024

ParticleChef commented Mar 22, 2024

mcremone commented Mar 22, 2024

mcremone commented Apr 1, 2024

mcremone commented Sep 22, 2023 •

edited

Loading

ParticleChef commented Nov 16, 2023 •

edited

Loading

mcremone commented Dec 8, 2023 •

edited

Loading

ParticleChef commented Dec 14, 2023 •

edited

Loading

alejands commented Jan 23, 2024 •

edited

Loading

alejands commented Jan 23, 2024 •

edited

Loading

mcremone commented Feb 28, 2024 •

edited

Loading