Fix on column mappings

Use character varying for hstore and jsonb columns instead of the unsupported string data type. Thanks @sjwmoveon for the fix!

Update to use new version of aws-lambda-redshift-loader

Use version 2.8.2 of aws-lambda-redshift-loader

Update Lambdas runtime to Node 16.x

Node 12.x runtime is deprecated and will stop getting security updates on Nov. 4th 2022. Full update from AWS: https://aws.amazon.com/blogs/developer/announcing-the-end-of-support-for-node-js-12-x-in-the-aws-sdk-for-javascript-v3/

Dynamically set Glue job script column mappings on terraform apply

The signatures_job.py script is now generated with the column mappings between CSV files and Redshift based on the signatures table schema as returned by the /api/bulk_data/schema.json API endpoint.
This should make it easier to update the script whenever there are schema changes in the signatures table, just by running terraform apply.

Fixes on column mappings

Fixed data types for multiple columns on the signatures_job.py script that prevented the values from being correctly imported to Redshift.

Update terraform AWS provider to v4

The expected changes on existing resources are the following:

• New aws_s3_bucket_acl.glue_resources resource for setting ACL on the glue scripts bucket. ACL hasn't changed, bucket is still private.
• New aws_s3_bucket_acl.manifest resource for setting ACL on the manifests bucket. ACL hasn't changed, bucket is still private.
• New aws_s3_bucket_lifecycle_configuration.manifest resource for setting the life cycle policy for files in the manifests bucket. Life cycle policy hasn't change, files will still be removed after 5 days.
• New aws_s3_bucket_server_side_encryption_configuration.glue_resources resource for setting server-side encryption on the glue scripts bucket.
• New aws_s3_bucket_server_side_encryption_configuration.manifest resource for setting server-side encryption on the manifests bucket.
• Resource aws_s3_bucket_object.signatures_script is replaced by aws_s3_object.signatures_script. This effectively creates a new file in S3 with the script run by Glue, the script itself hasn't changed though.

Handle CSVs compression

Adds support for setting the compress value for controlshift-redshift-loader Lambda function based on the value returned from the ControlShift instance API.

Upgrade Job to Glue 3.0

Glue 3.0 introduced several improvements, most notably, performance has been improved noticeably (you can see the release notes here).

The upgrade should be transparent if you haven't modified the Job's script, otherwise you can check out the documentation on migrating from AWS Glue 1.0 to AWS Glue 3.0.

Fix VPC endpoint for S3

Fixes a bug where the VPC Endpoint for S3 was created without an associated route table, making it unusable.

Fixes for Glue job setup

• Without setting the physical_connection_requirements property on aws_glue_connection, the AWS Glue job won't be able to connect to Redshift; so we made the glue_physical_connection_requirements variable mandatory.
• AWS Glue jobs need to have access to S3, either via a VPC endpoint or a NAT gateway. We added a new resource for creating the former, for which module consumers will need to provide the VPC's ID via the vpc_id variable.