feat: exec, run commands implementation

[levkohimins]

Description

1. exec, run commands implementation.
2. Sorting and renaming CLI flags.
3. Automatic generation of environment variables.
4. Code refactoring.

Relates to #3445 .

TODOs

Read the Gruntwork contribution guidelines.

• Update the docs.
• Run the relevant tests successfully, including pre-commit checks.
• Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
• Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added / Removed / Updated [X].

Migration Guide

[levkohimins]

Changes:

(1) For all flags with value validation (--log-level, --strict-control, --experiment, ...) the place (flag, env) where they were specified is now correctly indicated. This is especially important for environment variables, when the user may forget about previously set global env vars, and TG will tell that the flag value (which the user did not specify) is invalid.

before:

after:

(2) New strict controls deprecated-flags, deprecated-env-vars, deprecated-commands instead of specifying each flag or command separately plan-all,apply-all, etc., that makes the strict control is not usability. The user should be able to specify strict control for all flags/commands/envs and forget about it, rather than adding for example deprecated flags to the strict list each time.

(3) skip-dependencies-inputs has moved to experiments, I believe improvement performance has little to do with strict control.

Confirmation that these error/warn are not displayed anywhere,

terragrunt/internal/strict/strict.go

Lines 129 to 132 in 1676211

	SkipDependenciesInputs: {
	Error: errors.Errorf("The `%s` option is deprecated. Reading inputs from dependencies has been deprecated and will be removed in a future version of Terragrunt. To continue using inputs from dependencies, forward them as outputs.", SkipDependenciesInputs),
	Warning: fmt.Sprintf("The `%s` option is deprecated and will be removed in a future version of Terragrunt. Reading inputs from dependencies has been deprecated. To continue using inputs from dependencies, forward them as outputs.", SkipDependenciesInputs),
	},

instead we have this

terragrunt/config/config_partial.go

Lines 460 to 463 in 1676211

	_, _, skipInputs := control.Evaluate(ctx.TerragruntOptions)
	if skipInputs != nil {
	ctx.TerragruntOptions.Logger.Warnf("Skipping inputs reading from %v inputs for better performance", file.ConfigPath)

[yhakbar]

@levkohimins , the reason skip-dependencies-inputs is in strict control is that there will eventually be a breaking change to avoid parsing inputs of dependencies by default.

[yhakbar]

Nice

[yhakbar]

Note to self: How do we resolve the issue we were facing before with needing strict controls defined before we evaluate them?

[yhakbar]

What's making this synchronize with the EnableStrictMode calls above? Do these flags get evaluated in series?

[levkohimins]

The difference lies in the calling methods:

terragrunt/cli/flags/global.go

Lines 143 to 144 in 653b999

	Setter: func(_ bool) error {
	opts.StrictControls.EnableStrictMode()

terragrunt/cli/flags/global.go

Lines 169 to 174 in 653b999

	Action: func(_ *cli.Context, _ bool) error {
	opts.LogFormatter.SetPlaceholders(format.NewKeyValueFormatPlaceholders())
	newFlagName := LogCustomFormatFlagName + "=" + format.KeyValueFormatName
	if err := opts.StrictControls.Evaluate(opts.Logger, strict.DeprecatedFlags, TerragruntDisableLogFormattingFlagName, newFlagName); err != nil {

Setter is called before Action

terragrunt/internal/cli/flag.go

Lines 20 to 32 in 653b999

	// FlagSetterFunc represents function type that is called when the flag is specified.
	// Unlike `FlagActionFunc` where the function is called after the value has been parsed and assigned to the `Destination` field,
	// `FlagSetterFunc` is called earlier, during the variable parsing.
	// if `FlagSetterFunc` returns the error, it will be wrapped with the flag or environment variable name.
	// Example:
	// `fmt.Errorf("invalid value \"invalid-value\" for env var TG_ENV_VAR: %w", err)`
	// Therefore, using `FlagSetterFunc` is preferable to `FlagActionFunc` when you need to indicate in the error from where the value came from.
	// If the flag has multiple values, `FlagSetterFunc` will be called for each value.
	type FlagSetterFunc[T any] func(T) error
	// FlagActionFunc represents function type that is called when the flag is specified.
	// Executed after flag have been parsed and assigned to the `Destination` field.
	type FlagActionFunc[T any] func(ctx *Context, value T) error

[yhakbar]

What happened here?

Each experiment should have a write-up here. Not just the name.

[levkohimins]

Ah, I did not notice, there are different "How to provide feedback", "What it does". But I can't figure out how this md file passed the run markdownlint check and appeared in the main branch.

docs/_docs/04_reference/experiments.md:96 MD024/no-duplicate-heading Multiple headings with the same content [Context: "What it does"]
docs/_docs/04_reference/experiments.md:100 MD024/no-duplicate-heading Multiple headings with the same content [Context: "How to provide feedback"]
docs/_docs/04_reference/experiments.md:108 MD024/no-duplicate-heading Multiple headings with the same content [Context: "Criteria for stabilization"]
docs/_docs/04_reference/experiments.md:117 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]

terragrunt/docs/_docs/04_reference/experiments.md

Lines 70 to 105 in 44c3c81

	### symlinks
	Support symlink resolution for Terragrunt units.
	#### What it does
	By default, Terragrunt will ignore symlinks when determining which units it should run. By enabling this experiment, Terragrunt will resolve symlinks and add them to the list of units being run.
	#### How to provide feedback
	Provide your feedback on the [Experiment: Symlinks](https://github.com/gruntwork-io/terragrunt/discussions/3671) discussion.
	#### Criteria for stabilization
	To stabilize this feature, the following need to be resolved, at a minimum:
	- [ ] Ensure that symlink support continues to work for users referencing symlinks in flags. See [#3622](https://github.com/gruntwork-io/terragrunt/issues/3622).
	- [ ] Add integration tests for all filesystem flags to confirm support with symlinks (or document the fact that they cannot be supported).
	- [ ] Ensure that MacOS integration tests still work. See [#3616](https://github.com/gruntwork-io/terragrunt/issues/3616).
	- [ ] Add integration tests for MacOS in CI.
	### stacks
	Support for Terragrunt stacks.
	#### What it does
	Enable `stack` command to manage Terragrunt stacks.
	#### How to provide feedback
	Share your experience with the `stack` command in the [Stacks](https://github.com/gruntwork-io/terragrunt/issues/3313) RFC.
	Feedback is crucial for ensuring the feature meets real-world use cases. Please include:
	- Any bugs or issues encountered (including logs or stack traces if possible).
	- Suggestions for additional improvements or enhancements.

[yhakbar]

That check was disabled in CI:
https://github.com/gruntwork-io/terragrunt/blob/main/.circleci/config.yml#L130

[yhakbar]

When we redesign docs, I'll give each of these their own page.

[yhakbar]

I didn't realize you got rid of these controls too. I wouldn't do that, @levkohimins .

It's more annoying to maintain, but these controls are only useful if users can enable them to test out particular changes in an upcoming release. For example "In the new release, Terragrunt made it so that the apply-all command is no longer supported. I should enable the strict-mode check for this in all my lower environments so that I can check that I'm safe to upgrade."

We should never remove a strict control, or an experiment, just mark it as complete.

[levkohimins]

We still won't add all the flags and commands here, and it turns out that for some we will be able to disable one by one and for others there will be one common flag, this will only puzzle the user, like for example for those renamed during the cli redesign. Well, even if we spend a lot of time adding and removing, this will turn into a horror for both mantaince and its use

To be realistic, it will not happen that we remove one apply-all and leave all the others plan-all, sign-up...

If a user wants to make sure that he no longer uses some deprecated flags, all he has to do is make sure that the warning message is no longer displayed. This will not be more difficult than listing about ten flags in the cli.

The problem is also in the implementation, I can't reference to these constants from this package

terragrunt/cli/commands/deprecated.go

Lines 18 to 24 in cf44ec0

	CommandNameSpinUp = "spin-up"
	CommandNameTearDown = "tear-down"
	CommandNamePlanAll = "plan-all"
	CommandNameApplyAll = "apply-all"
	CommandNameDestroyAll = "destroy-all"
	CommandNameOutputAll = "output-all"
	CommandNameValidateAll = "validate-all"

this leads to a loop. I can't reference to flag constants in warn/error messages for the same reason.
This leads to the fact that we have to duplicate constants or write the name of the flags as plain text.

I can say 100%, we will forget to keep them up to date. Why? Because I added these warn/err messages two months ago, and when I renamed the flags, I completely forgot that somewhere there is a text that points to the old name.

What we end up with is a huge amount of unnecessary work to maintain the text up to date, deterioration of code readability, all for the sake of the user getting an error instead of paying attention to the first lines of the logs.

[levkohimins]

I try to imagine myself in the user's shoes. There are two options for how I would behave:

1. I try to keep all flag or command names up-to-date, checking with the --strict-control deprecated-commands --strict-control deprecated-flags options, or paying attention to warning messages.
2. I don't care at all, until the flag or command is completely removed and I get an error.

[levkohimins]

You may have thought that the logs only indicate the strict control name, but no, logs list each deprecated flag and command separately.

Green arrow