Write and drop

[mikeknep]

Currently, the preprocessing in synthetics training creates a training DF for each table, keeps each of those in a dictionary, and passes the dictionary to the next method which writes each to a CSV and submits a model.

With this change, each training DF is written to CSV as soon as it is created, so we don't have to hold on to more than one at a time.

There's also a small change to the independent strategy specifically: instead of asking for the whole source dataframe and then dropping columns, we only ask for the columns we intend to keep. This doesn't make too big a difference now (since the whole source DF is already in memory in the graph), but we anticipate eventually storing source data on disk and loading in the data when needed, and presumably at that point this will provide some greater benefit.

[mikeknep]

I did some testing using tracemalloc and a set of tables ranging from about 40–670 MB. Before this change / on main, the peak memory usage during synthetics training was 1.75 GB (1,756,860,969) (and that may have even been a bit artificially low because I cut out some other things to save time). With this change, the peak only reached 0.5 GB (501,836,663).

[pimlock]

Nice!