[Feature] Add an --input-file option to run-operation

[edgarrmondragon]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

The run-operation command currently supports a CLI option --args with a YAML string value. For complex and/or large YAML objects as inputs, passing the whole thing as a string in the CLI is mildly inconvenient. A good alternative for such use cases would be to support an --input-file option that expects a path to a YAML file.

Describe alternatives you've considered

A workaround documented in Discourse is to use Bash's capabilities to pass the contents of the file like:

 dbt run-operation create_pipe --args "$(cat path/to/file.yml)"

This works in environments where dbt is directly executed as a shell command, but in dbt Cloud the Bash magic doesn't work.

Who will this benefit?

Users of run-operation with complex inputs.

Are you interested in contributing this feature?

Yes!

Anything else?

No response

[yoitsbenc]

wow, what a great idea! maybe this type of macro would then work in dbt cloud

[jtcohen6]

@edgarrmondragon Thanks for opening!

The workaround described in that discourse post makes sense to me. So I sense the real motivation behind this issue as being:

in dbt Cloud the Bash magic doesn't work

maybe this type of macro would then work in dbt cloud

Totally fair; I'd like to see us add better support for the use case. I sense that this falls under the more general-purpose feature of dbt Cloud jobs as code (specifically, YAML). In a future where that's possible, it would also be possible to define the arguments for your run operation in a version-controlled file:

# NOTE: this is made up, and does not exist today
jobs:
  - name: my_job
    execution:
      steps:
        - dbt run-operation create_pipe --args '{"arg1": "value1", "arg2": ["long", "list", "of", "values"], "arg3": ...}'

Or perhaps even:

# for demonstration only
jobs:
  - name: my_job
    execution:
      steps:
        - name: Run operation to create pipe
          subcommand: run-operation
          macro_name: create_pipe
          args:
            arg1: value1
            arg2:
              - long
              - list
              - of
              - values
            arg3: ...

@schottj What do you think? Does something like that make sense to you as part of a future initiative?

---

I noticed that your macro is named create_pipe, so just an addendum: In addition to providing all run-operation arguments on the CLI at runtime, your operation macro also has the ability to grab rich metadata from your project, by accessing the graph context variable.

If you were to define your pipe as a dbt source, you could store contextual information in that source's meta or external properties, and then access them from the parsed manifest at runtime. That's the approach taken by the dbt-external-tables package, enabling users to run just:

$ dbt run-operation stage_external_sources --args "select:my_external_source"

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.