add pod_spec_override to config and job

[godber]

We have needed the ability to overlay arbitrary k8s config onto the podSpec for a while. The idea is the admin or the user should be able to adjust the pod specs for the execution controller or the workers at the config or job levels. This will allow support for things like adding HostAliases

So perhaps the job would go from:

{
    "name": "example-data-generator-job",
    "lifecycle": "once",
    "workers": 2,
    "assets": [
        "example",
        "elasticsearch",
        "standard"
    ],
    "ephemeral_storage": true,
    "operations": [
        {
            "_op": "data_generator",
            "size": 5000000
        },
        {
            "_op": "example-op"
        },
        {
            "_op": "delay",
            "ms": 30000
        },
        {
            "_op": "elasticsearch_bulk",
            "index": "terak8s-example-data",
            "type": "events",
            "size": 5000
        }
    ]
}

To

{
    "name": "example-data-generator-job",
    "lifecycle": "once",
    "workers": 2,
    "assets": [
        "example",
        "elasticsearch",
        "standard"
    ],
    "ephemeral_storage": true,
    "operations": [
        {
            "_op": "data_generator",
            "size": 5000000
        },
        {
            "_op": "example-op"
        },
        {
            "_op": "delay",
            "ms": 30000
        },
        {
            "_op": "elasticsearch_bulk",
            "index": "terak8s-example-data",
            "type": "events",
            "size": 5000
        }
    ],
    "pod_spec_override": {
        "hostAliases": [
          {
             "ip": "127.0.0.1",
             "hostnames": [
                "foo.local",
                "bar.local"
             ]
          }
       ]
    }
}

The order of precedence for the object merge operations should be

• create spec with named settings first
• then merge in config level settings
• then merge in job level settings

So job level Overlay settings will always override everything else. I will worry about conflicts between the overlay and the "named settings" later. Conflicts at this point can probably be avoided at job submission time.

I know I had problems last time I tried to add something nested here:

#3157 (comment)

But I think convict has been upgraded now and I won't try validating the podSpecOverride property, so maybe this will work?

[godber]

This was closed in #3348