Closes #4076: bug in reading and writing to/from parquet when the locales change

[ajpotts]

Fixes a bug that was reported by a user that when an array is saved to parquet under 40 nodes, and then saved later to 20 nodes, not all the files from the original file are deleted. Therefore when the array is read in again it contains incorrect data.

[e-kayrakli]

I think the changes here are good. However, I am worried about removing files based on name-matching. IOW, the user could just have a file with a given name that matches the pattern purely by coincidence. I think there are two things that can mitigate that:

• Save per-locale files in a directory. This doesn't eliminate the problem as the user may just create a problematic file in the given directory, which would, again, be removed.
• Like above, but also add a metadata. The directory could contain a metadata file that lists all the files that represent chunks of an array. Instead of matching files by name, we can read that metadata, and delete files based on the names stored there.

I don't think these are necessary for the first step, but could be further action items to increase the robustness. (Could be added as a comment on the client where we call _delete_arkouda_files.

[ajpotts]

The files are deleted according to the regex in this function:

    proc getMatchingFilenames(prefix : string, extension : string) throws {
        return glob("%s_LOCALE*%s".format(prefix, extension));    
    }

So the user would would only delete files following the pattern _LOCALE. I think the probability of random files following this pattern and being deleted is low. I added extra file reads to the unit tests to show that a similarly named prefix would not be deleted.

[ajpotts]

I think you make a valid point though that using prefix's to identify data has some weaknesses that would be work addressing. For example, I noticed this issue with reading files with similar prefixes: #4083

[drculhane]

This looks correct to me, although I'm not at all sure I have the resources to test it the way I'd like to. But walking through the code, I believe this is a correct fix to the issue.

[e-kayrakli]

I think you make a valid point though that using prefix's to identify data has some weaknesses that would be work addressing. For example, I noticed this issue with reading files with similar prefixes: #4083

Thanks! I commented under that issue about what I outlined above.