Loading custom profiles per user

[JackUrb]

Overview

Following #494, we're now able to do recursive defaults and imports along the search path. This means we can set up profile's which can be used across different tasks.

Usage:

By defining a config in ~/.mephisto/hydra_configs/profile/, you can later invoke all of the configuration from that with just one command. For instance, with ~/.mephisto/hydra_configs/profile/sandbox.yaml as follows:

# @package _global_
defaults:
 - override /mephisto/architect: heroku
 - override /mephisto/provider: mturk_sandbox

mephisto:
  architect:
    _architect_type: heroku
    use_hobby: true
  provider:
    requester_name: MY_REQUESTER_ACCOUNT_sandbox
  log_level: debug

I can run python run_task +profile=sandbox for any task, and pull in the options both for the sandbox requester to use, as well as the heroku-specific argument to launch on the hobby tier. This means one can store profiles for common launch patterns that are shared between tasks.

Testing:

Using the above file, successfully launched a task to sandbox.

[EricMichaelSmith]

Okay, seems reasonable - so I suppose that this profile setup can also be used to specify live mode with a single arg, instead of doing mephisto.provider.requester_name=${REQUESTER_NAME} mephisto/architect=heroku as currently?

[JackUrb]

Yup, the basic case will likely just be outlining a requester name and the architect to use, but it can be more specialized for each profile.