fix issues in two phase build

[jedwards4b]

This fixes the issue with the NCK test. There still appears to be an intermittent problem - it looks like the mkdir command in preview_namelist at line 137 occasionally fails. Is it because multiple threads arrive here at the same time?

[jgfouca]

Hi @jedwards4b , I've been banging my head against this for several hours. The solution that had me very hopeful was to make an extra Case instance at the beginning of nck.build and then flushing it at the end if sharedlib_only. Unfortunately, this failed along with everything else I tried.

Many thanks for this fix. Would you mind briefly explaining what the core problem was and why this fixes it?

[jgfouca]

Oh, and build.py already creates the sharedlibroot, so preview_namelists does not need to do it. I'll make that change. You are correct, it is not safe to have preview_namelists do anything to the shared area because they run in parallel.

[jedwards4b]

The problem was that we were creating two build directories for the cpl/drv
component, the build was going into cesm/obj and clean was targeting
cpl/obj so that
component was not being properly cleaned before the second build. I'm not
sure why it worked before we separated the build into two steps.
I changed it so that the build is now going into cpl/obj - that cesm/obj
(actually $model/obj) directory is still being created somewhere and we can
get rid of it when we figure out where it is.

On Sun, Feb 28, 2016 at 3:44 PM, James Foucar [email protected]
wrote:

Hi @jedwards4b https://github.com/jedwards4b , I've been banging my
head against this for several hours. The solution that had me very hopeful
was to make an extra Case instance at the beginning of nck.build and then
flushing it at the end if sharedlib_only. Unfortunately, this failed along
with everything else I tried.

Many thanks for this fix. Would you mind briefly explaining what the core
problem was and why this fixes it?

—
Reply to this email directly or view it on GitHub
#72 (comment).

Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO

[jgfouca]

Hi @jedwards4b , I tried your fix and still getting the same segf that I was getting before. I'm gathering some additional info now.

[jgfouca]

Hi @jedwards4b ,
I'm still stuck on this but I've created some tools that can help us debug this issue. One technique that I've used in the past with some success is to make a "baseline" test case, make some changes to the scripts, make another test case, then do a recursive diff on the two cases, hopefully identifying any changes in behavior of the scripts. This technique requires extra work in the sense that lots of things like timestamp and test-id differences can cause huge numbers of uninteresting diffs. I had a process of getting rid of these differences that I've formalized and implemented with a new "normalize_cases" tool that I've added to scripts/Tools. I've also added a "case_diff" tool for doing the recursive diff.

The process of using these new tools for this problem in particular is as follows:
`
% ./create_test NCK_Ld3.f45_g37_rx1.A -t jgf_works --no-build

% ./create_test NCK_Ld3.f45_g37_rx1.A -t jgf_broken --no-run

% * cd to test_root *

% cd NCK_Ld3.f45_g37_rx1.A.melvin_gnu.jgf_works

% ./case.test_build

% cd ..

% normalize_cases NCK_Ld3.f45_g37_rx1.A.melvin_gnu.jgf_broken NCK_Ld3.f45_g37_rx1.A.melvin_gnu.jgf_works

% case_diff NCK_Ld3.f45_g37_rx1.A.melvin_gnu.jgf_broken NCK_Ld3.f45_g37_rx1.A.melvin_gnu.jgf_works
`
The idea here is that if the cases have significant differences in XML, namelists, or build logs, then we've failed at getting the two-phase build to work like the one-phase build. I am seeing such diffs unfortunately.

Let me know what you think! I've pushed these new tools to the same branch on which I'm doing the two-phase work.