Fix exceptions, mark required options as required

[HebaruSan]

Problems

Uncaught exception:
Traceback (most recent call last):
  File ".local/bin/netkan", line 8, in <module>
    sys.exit(netkan())
  File "/home/netkan/.local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/netkan/.local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/netkan/.local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/netkan/.local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/netkan/.local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/netkan/.local/lib/python3.7/site-packages/netkan/cli.py", line 131, in scheduler
    init_repo(ckanmeta_remote, '/tmp/CKAN-meta')
  File "/home/netkan/.local/lib/python3.7/site-packages/netkan/utils.py", line 12, in init_repo
    repo = Repo.clone_from(metadata, clone_path)
  File "/home/netkan/.local/lib/python3.7/site-packages/git/repo/base.py", line 1023, in clone_from
    return cls._clone(git, url, to_path, GitCmdObjectDB, progress, multi_options, **kwargs)
  File "/home/netkan/.local/lib/python3.7/site-packages/git/repo/base.py", line 957, in _clone
    proc = git.clone(multi, Git.polish_url(url), clone_path, with_extended_output=True, as_process=True,
  File "/home/netkan/.local/lib/python3.7/site-packages/git/cmd.py", line 333, in polish_url
    url = os.path.expandvars(url)
  File "/usr/local/lib/python3.7/posixpath.py", line 288, in expandvars
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

Uncaught exception:
Traceback (most recent call last):
  File "/home/netkan/.local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/netkan/.local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/netkan/.local/lib/python3.7/site-packages/netkan/webhooks/github_utils.py", line 14, in decorated_function
    return func(*args, **kwargs)
  File "/home/netkan/.local/lib/python3.7/site-packages/netkan/webhooks/github_inflate.py", line 26, in inflate_hook
    inflate(ids_from_commits(commits))
  File "/home/netkan/.local/lib/python3.7/site-packages/netkan/webhooks/github_inflate.py", line 64, in inflate
    Entries=batch
  File "/home/netkan/.local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/netkan/.local/lib/python3.7/site-packages/botocore/client.py", line 634, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/home/netkan/.local/lib/python3.7/site-packages/botocore/client.py", line 682, in _convert_to_request_dict
    api_params, operation_model)
  File "/home/netkan/.local/lib/python3.7/site-packages/botocore/validate.py", line 297, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter Entries[0].MessageAttributes.HighestVersion, value: 2:release-1.7.3-2, type: <class 'str'>, valid types: <class 'dict'>

Cause

The --ckanmeta-remote parameter isn't being passed to the scheduler (because the stack needs to be updated the environment variable was missing for the web hooks pass), but it's running anyway. This is a problem because that parameter is needed for the scheduler to do its work. There are some other instances of this problem in cli.py.

The HighestVersion message attribute from #102 was passed as a string. Apparently it's supposed to be a dict:

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/sqs.html

    {
        'Id': '2',
        'MessageBody': 'boto3',
        'MessageAttributes': {
            'Author': {
                'StringValue': 'Daniel',
                'DataType': 'String'
            }
        }
    }

Changes

Now all required Click options in cli.py are marked as required. As well, some minor formatting tweaks (trailing commas) are made for consistency and ease of maintenance.

Now CKANMETA_REMOTE is passed to the scheduler web hooks pass, which will fix the root cause of the above error.

Now we generate a dict for HighestVersion.

[HebaruSan]

The bot and the webhooks are both down due to this, merging to address emergency circumstances...

[techman83]

We should open an issue to refactor this so that it does the RightThings™ - There is definitely a nicer way to do this.

[HebaruSan]

Yeah, this was just a safe short term fix to get it running again, please feel free to change it to whatever would be best.

[techman83]

I better deploy this!

[techman83]

I have started looking at writing some decorators to make things like this far less likely to be missed. But I'll get the stack updated shortly.