PE layout of 384x3 on cheyenne fails with updated ESMF library

[fischer-ncar]

Issue Type

Infrastructure Update

Issue Description

PE layout for compset="_(CAM50|CAM60)%(CC|CV|CF|WC)" with change from 384x3 to 360x3 on cheyenne.

Will this change answers?

No

Will you be implementing this yourself?

Yes

[fvitt]

Do we understand why 384x3 PE layout is a problem?

[fischer-ncar]

I'll let @jedwards4b answer that. But my understanding it's a change in how ESMF handles tasks and threads.

[cacraigucar]

I was seeing this with our MCT WACCM tests. At a meeting today, the scientists agreed that we can drop these tests which we will be doing with our next CAM tag. Does this issue expand to tests other than this?

[jedwards4b]

Francis had a good point that I did not follow up on. That is that 384x3 fits properly on 32 cheyenne nodes and so my reasoning for this change doesn't make sense. I'll look into it further.

[fischer-ncar]

These are the prealpha tests that failed with 384x3, but pass with 360x3.

SMS_D_Ln9_Vnuopc.f09_f09_mg17.FCHIST.cheyenne_intel.cam-outfrq9s_ocnemis
SMS_D_Ln9_Vnuopc.f09_f09_mg17.FCts2SD.cheyenne_intel.cam-outfrq9s R
SMS_D_Ln9_Vnuopc.f09_f09_mg17.FWHIST.cheyenne_intel.cam-reduced_hist3s
SMS_D_Ln9_Vnuopc.f09_f09_mg17.FWma2000climo.cheyenne_intel.cam-outfrq9s
SMS_D_Ln9_Vnuopc.f09_f09_mg17.FWsc1850.cheyenne_intel.cam-outfrq9s
SMS_D_Ln9_Vnuopc.f09_f09_mg17.FWscHIST.cheyenne_intel.cam-outfrq9s

[cacraigucar]

@fischer-ncar - What are the error messages that you are getting from these runs and have you verified they are repeatable failures? I ran CAM regression test last night for an upcoming CAM tag and I had several weird failures which I'd not seen before in previous testing. I think cheyenne was giving me machine hiccups and I'm running them again. Also, the test ERP_Ld3_Vnuopc.f09_f09_mg17.FWHIST.cheyenne_intel.cam-reduced_hist1d1 ran fine for me last night and it used the 384x3 PE layout. I personally have not had any failures using the 384x3 layouts in anything other than the MCT tests.

[cacraigucar]

Also, I should mention that cam6_3_079 which is for PR #666 will have the problematic MCT WACCM tests removed. I'm hopeful that tag will be coming in the next day or two (dependent on how cheyenne treats my tests)

[fischer-ncar]

@cacraigucar The failures are repeatable, and the error message I'm getting is.

MPT ERROR: Rank 297(g:297) received signal SIGFPE(8).

[fischer-ncar]

@fvitt @peverwhee @nusbaume Do we want to get this issue fixed in the tag that's going into alpha10c? Or do you want to wait for @jedwards4b to look into this issue?

[fvitt]

It would be good to understand why the 384x3 PE layout is a problem. I vote for waiting for Jim to look into it.

[nusbaume]

I agree with Francis, it would be good to get to the bottom of this, which probably means waiting for Jim.

[jedwards4b]

Running test SMS_D_Ln9_Vnuopc.f09_f09_mg17.FCHIST.cheyenne_intel.cam-outfrq9s_ocnemis I set a breakpoint at
dshr_mod.F90 line 1417 which is a call to ESMF_FieldRegridStore. When I try to step into that function many of the tasks appear to hang.

[jedwards4b]

Fixed by ESCOMP/CDEPS#194

[jedwards4b]

@fischer-ncar do you agree that this ticket can be closed now?

[cacraigucar]

We have that this PR will be closed when we update the externals in CAM (PR #700) as we believe that the CDEPS update will fix the problem, correct? PR #700 is in final testing now.

[fischer-ncar]

Bit late, but yes this ticket can be closed.