Allow Parquet records format to be specified on mvrec command line

[vinceatbluelabs]

This allows folks to specify the Parquet records format on the command line. Right now, mvrec implicitly assumes that only delimited format files (like CSVs) are being specified using its command line parameters.

The forcing function for this PR is that we use Parquet when transferring data between Redshift and BigQuery. I want to allow folks to break up that transfer to exporting first to an explicitly provided S3 bucket. That is now expressible as mvrec table2recordsdir --target.format parquet dbname schema table s3://bucket/directory/ in mvrec.

Example:

(records-mover-3.8.5) �]0;bigbookpro�broz@bigbookpro:~/src/records-mover$ mvrec file2table --help
mvrec file2table --help
usage: mvrec file2table [-h] [--source.variant SOURCE.VARIANT]
                        [--source.format {parquet,delimited}]
                        [--source.datetimeformattz {YYYY-MM-DD HH:MI:SSOF,YYYY-MM-DD HH:MI:SS,YYYY-MM-DD HH24:MI:SSOF,MM/DD/YY HH24:MI}]
                        [--source.datetimeformat {YYYY-MM-DD HH24:MI:SS,YYYY-MM-DD HH:MI:SS,YYYY-MM-DD HH12:MI AM,MM/DD/YY HH24:MI}]
                        [--source.no_compression]
                        [--source.compression {GZIP,BZIP,LZO}]
                        [--source.no_quoting]
                        [--source.quoting {all,minimal,nonnumeric}]
                        [--source.no_escape] [--source.escape {\}]
                        [--source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}]
                        [--source.dateformat {YYYY-MM-DD,MM-DD-YYYY,DD-MM-YYYY,MM/DD/YY}]
                        [--source.timeonlyformat {HH12:MI AM,HH24:MI:SS}]
                        [--source.no_doublequote] [--source.doublequote]
                        [--source.no_header_row] [--source.header_row]
                        [--source.quotechar SOURCE.QUOTECHAR]
                        [--source.record_terminator SOURCE.RECORD-TERMINATOR]
                        [--source.field_delimiter SOURCE.FIELD-DELIMITER]
                        [--target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}]
                        [--target.drop_and_recreate_on_load_error]
                        source.filename target.db_name target.schema_name
                        target.table_name

positional arguments:
  source.filename       File path (relative or absolute) of the data file
                        to load.
  target.db_name        SQLAlchemy database engine to write data to.
  target.schema_name    Schema name of a table to write data to.
  target.table_name     Table name of a table to write data to.

optional arguments:
  -h, --help            show this help message and exit
  --source.variant SOURCE.VARIANT
                        Records format variant - valid for 'delimited'
                        records format type
  --source.format {parquet,delimited}
                        Records format type. Note that 'delimited' includes
                        CSV/TSV/etc.
  --source.datetimeformattz {YYYY-MM-DD HH:MI:SSOF,YYYY-MM-DD HH:MI:SS,YYYY-MM-DD HH24:MI:SSOF,MM/DD/YY HH24:MI}
                        Format used to write 'datetimetz' values
  --source.datetimeformat {YYYY-MM-DD HH24:MI:SS,YYYY-MM-DD HH:MI:SS,YYYY-MM-DD HH12:MI AM,MM/DD/YY HH24:MI}
                        Format used to write 'datetime' values
  --source.no_compression
  --source.compression {GZIP,BZIP,LZO}
                        Compression type of the file.
  --source.no_quoting
  --source.quoting {all,minimal,nonnumeric}
                        How quotes are applied to individual fields. all:
                        quote all fields. minimal: quote only fields that
                        contain ambiguous characters (the delimiter, the
                        escape character, or a line terminator). default:
                        never quote fields.
  --source.no_escape
  --source.escape {\}   Character used to escape strings
  --source.encoding {UTF8,UTF16,UTF16LE,UTF16BE,UTF16BOM,UTF8BOM,LATIN1,CP1252}
                        Text encoding of file
  --source.dateformat {YYYY-MM-DD,MM-DD-YYYY,DD-MM-YYYY,MM/DD/YY}
                        Format used to write 'date' values
  --source.timeonlyformat {HH12:MI AM,HH24:MI:SS}
                        Format used to write 'time' values
  --source.no_doublequote
  --source.doublequote  Controls how instances of quotechar appearing
                        inside a field should themselves be quoted. When
                        True, the character is doubled. When False, the
                        escapechar is used as a prefix to the quotechar.
  --source.no_header_row
  --source.header_row   True if a header row is provided in the delimited
                        files.
  --source.quotechar SOURCE.QUOTECHAR
                        A one-character string used to quote fields
                        containing special characters, such as the
                        delimiter or quotechar, or which contain new-line
                        characters.
  --source.record_terminator SOURCE.RECORD-TERMINATOR
                        String used to close out individual rows of data.
  --source.field_delimiter SOURCE.FIELD-DELIMITER
                        Character used between fields.
  --target.existing_table {delete_and_overwrite,truncate_and_overwrite,drop_and_recreate,append}
                        When loading into a database table, controls how
                        any existing table found will be handled. This must
                        be a :class:`records_mover.records.ExistingTableHan
                        dling` object.
  --target.drop_and_recreate_on_load_error
                        If True, table load errors will attempt to be
                        addressed by dropping the target table and
                        reloading the incoming data.
(records-mover-3.8.5) �]0;bigbookpro�broz@bigbookpro:~/src/records-mover$ 

Note: This is set to merge into #128

[vinceatbluelabs]

This isn't related to the theme of this PR, but I found it along the way - our config parser doesn't use double quotes for parsing in its .ini format.

[crvena-sonja]

Huh. Good find.

[vinceatbluelabs]

While using this switch, I wanted to make sure it works, so logging what format I'm writing out was important!

[vinceatbluelabs]

The below logic is all kinds of weird. I get that.

Here's what's going on:

Records Mover is both a Python library and a command line tool.

In terms of the library, the richest surface area of the API are source and target factory methods - where you specify what source and target you want, and configure them exactly as you want:

• https://github.com/bluelabsio/records-mover/blob/master/records_mover/records/sources/factory.py
• https://github.com/bluelabsio/records-mover/blob/master/records_mover/records/targets/factory.py

The naive way to write a command line parser would be to take each of these functions and write an argparse parser. You'd certainly want to do something at least mildly clever to combine those, so you don't have to rewrite the "file2" part of "file2url" and "file2table", etc, or you're dealing with exponential growth as sources and targets are added.

On top of that, it turns out that a lot of those parameters to methods are common - things like the records_format, the db_engine, the initial_hints, etc. Copying and pasting those over and over is kind of silly.

It's also true that a lot of those are pretty simple - strings, booleans, etc, possibly with default values. And they already have documentation written.

So I do something more clever, and I use Python's reflection abilities to pull parameter lists, types and documentation and generate argparse parsers. There's even an intermediate form as JSON schema (we used to have a layer which would accept records mover instructions as a JSON-style request). You can draw your own conclusion as to whether 'clever' is a positive description or not.

Aaaanyway....

[vinceatbluelabs]

First complication - not every argument in a source/target factory method maps 1:1 to a command line parameter. The example we're hitting here is a RecordsFormat object - which has two subtypes (parquet and delimited). The DelimitedRecordsFormat() constructor takes both a variant argument and a hints dict, which can contain various keys.

We have a technique (which will show up further down in this PR) for exploding things like RecordsFormat out into multiple command line arguments.

This function that we're looking at now is on the call path when we're accepting the parsed command line argument results back in (as the kwargs dictionary) and are formulating the call into the factory method.

Its job is to be called for each hint parameter we receive and update the final set of arguments we'll pass to the factory method accordingly.

[vinceatbluelabs]

Since we now might get contradictory combination of arguments (like --source.format parquet --source.delimiter '|', we need to check for that situation.

Note that one day ParquetRecordsFormat will support some very small number of hints (compression is a key one that it's missing today), but as of today, there are no hints it supports.

[vinceatbluelabs]

We now have both a format and variant command line parameter, instead of just variant. These functions handle ingesting two two, respectively.

The 'initial_hints' handling is not new or changed with this PR. It deals with the situation where we want RecordsMover to generally sniff the records format, but specify certain hints to begin with as constraints.

[vinceatbluelabs]

The line above was not necessary as far as I can tell. When the variant command line argument isn't supplied, it doesn't appear in the map, and if it doesn't appear in the map, this function wasn't being called.

[vinceatbluelabs]

This is where we do the original translation of parameters in the factory methods to the intermediate JSON Schema format.

[vinceatbluelabs]

I also added descriptions to these key two so we have specific descriptions (instead of reusing the docs from the factory method).