out-of-memory error on hpc during capture output

[m-bossart]

No description provided.

[m-bossart]

Training occasionally runs out of memory on hpc while writing the output Arrow data files. The function that captures the output is here:

PowerSimulationNODE/src/train/train.jl

Line 633 in 4f0e37a

function _capture_output(output_dict, output_directory, id)

For the cases that fail, the first dataframe is written to file and the process is killed on the second iteration of the loop.
I tried to use df=nothing and GC.gc() to free the memory required from the prior iteration but still got failures. @jd-lara any suggestions? Does it make sense to write data to the output files directly from the training callback?

[jd-lara]

what's the size of the DF you are trying to write? I wonder if the Arrow write command itself is what's causing the problem.

[m-bossart]

Not that big. ~380 parameters x 300 rows. It has worked successfully for much larger cases. Most of the time it works. 10-20% of the runs have this failure in the most recent version.