st_archive: Failure to archive intermediate dart - and maybe other - restart files

[billsacks]

There is a subtle issue is the current short-term archiver (case_st_archive.py) that prevents archiving intermediate restart files for dart, and possible other components.

The problem is in _archive_process: The relevant parts of the code look like this (some irrelevant pieces deleted for brevity):

    for archive_entry in archive.get_entries():
        # determine compname and compclass
        compname, compclass = archive.get_entry_info(archive_entry)

        datenames = _get_datenames(case, last_date)
        for datename in datenames:

            # archive restarts
            histfiles_savein_rundir = _archive_restarts(case, archive, archive_entry,
                                                        compclass, compname,
                                                        datename, datename_is_last,
                                                        archive_file_fn)

The problem with this is that _get_datenames determines the list of dates based on what cpl restart files are present. If cpl is processed before other components, then those other components will only see the final restart files when they call _get_datenames, because all other cpl restart sets will have been moved.

I have only noticed this problem affecting dart restart sets, but in principle it seems that it could affect any components, depending on the ordering of the results from archive.get_entries()

I have a fix for this in an upcoming PR, where I have refactored _archive_process to have the date loop outside the component loop.

[billsacks]

cc @mvertens @jedwards4b @bertinia

[gold2718]

@billsacks, Would this also prevent archiving of intermediate initial files? I have a report from a user that intermediate CAM initial files are not being archived (i.e., only the last initial file written is archived). Since CAM uses initial files for data assimilation, this might be relevant to this issue.

[billsacks]

@billsacks, Would this also prevent archiving of intermediate initial files? I have a report from a user that intermediate CAM initial files are not being archived (i.e., only the last initial file written is archived). Since CAM uses initial files for data assimilation, this might be relevant to this issue.

Yes, I think the same mechanism is used for CAM initial files, so this issue would affect them (depending on the order in which entries were returned from archive.get_entries()). This should now be fixed on master, or if you want to test with a minimal set of changes, you can apply the minor diffs given by billsacks@02c1616

[jedwards4b]

I am conducting this test now.

[jedwards4b]

I tested IRT_Ld5.f19_f19_mg17.F2000_DEV.cheyenne_intel and added inithist='DAILY' to the user_nl_cam. cam.i files were copied to the archive directory at all the correct interim times.