[COST-1192] Use chunks in pandas to process csv->parquet

[adberglund]

Summary

Use pandas chunk to reduce memory use while converting CSV to parquet.

Testing

1. Set the following environment variables

ENABLE_PARQUET_PROCESSING=True
S3_BUCKET_NAME=koku-bucket
S3_ENDPOINT=http://kokuminio:9000
S3_ACCESS_KEY=kokuminioaccess
S3_SECRET=kokuminiosecret
INITIAL_INGEST_OVERRIDE='True'
INITIAL_INGEST_NUM_MONTHS=1

2. Loaded a large AWS source
3. Let finish
4. Checked AWS report APIs

[codecov]

Codecov Report

Merging #2740 (9b5670d) into master (c50bfd8) will decrease coverage by 0.0%.
The diff coverage is 92.1%.

@@           Coverage Diff            @@
##           master   #2740     +/-   ##
========================================
- Coverage    94.7%   94.7%   -0.0%     
========================================
  Files         289     289             
  Lines       22403   22433     +30     
  Branches     2554    2559      +5     
========================================
+ Hits        21212   21239     +27     
- Misses        722     726      +4     
+ Partials      469     468      -1     

[Red-HAP]

This column name scrubber appears in multiple places. Would it make more sense to make a common or generic scrubber for this to prevent potential issues with replicated code?

Maybe something similar to:

AWS_SCRUB_MAP = {"_": r"[-/:]+"}
AZURE_SCRUB_MAP = AWS_SCRUB_MAP
GCP_SCRUB_MAP = {"_": r"[-/:.]+"}

import re

def standardize_column_name(raw_col_name, translate_map): 
    scrubbed_name = raw_col_name.strip().lower() 
    for repl, srch in translate_map.items(): 
        scrubbed_name = re.sub(srch, repl, scrubbed_name) 
    return scrubbed_name 

[adberglund]

@Red-HAP Totally, yes. I think there's probably a few places where this could use some refactor/consolidation. My ask would be that we move that idea into a new issue we can tackle in this epic. Since we have are facing a crunch and getting this out could let us enable AWS Trino.

[adberglund]

Created https://issues.redhat.com/browse/COST-1208