Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve hocon paths #5644

Merged
merged 8 commits into from
Feb 15, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
171 changes: 4 additions & 167 deletions docs/articles/configuration/hocon.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,9 +253,9 @@ Substitution processing is performed as the last parsing step, so a substitution

If a key has been specified more than once, the substitution will always evaluate to its latest-assigned value (that is, it will evaluate to the merged object, or the last non-object value that was set, in the entire document being parsed including all included files).

If a substitution does not match any value present in the configuration and is not resolved by an external source, then it is undefined. An undefined _required substitution_ is invalid and will generate an error.
If a substitution does not match any value present in the configuration, then it is undefined. An undefined _required substitution_ is invalid and will generate an error.

[!code-csharp[ConfigurationSample](../../../src/core/Akka.Docs.Tests/Configuration/ConfigurationSample.cs?name=UnsolvableSubstitutionWillThrowSample)]
[!code-csharp[SerializationSetupDocSpec](../../../src/core/Akka.Docs.Tests/Configuration/SerializationSetupDocSpec.cs?name=UnsolvableSubstitutionWillThrowSample)]

If an _optional substitution_ is undefined:

Expand All @@ -271,168 +271,5 @@ A substitution is replaced with any value type (number, object, string, array, t

Examples:

[!code-csharp[ConfigurationSample](../../../src/core/Akka.Docs.Tests/Configuration/ConfigurationSample.cs?name=StringSubstitutionSample)]
[!code-csharp[ConfigurationSample](../../../src/core/Akka.Docs.Tests/Configuration/ConfigurationSample.cs?name=ArraySubstitutionSample)]
[!code-csharp[ConfigurationSample](../../../src/core/Akka.Docs.Tests/Configuration/ConfigurationSample.cs?name=ObjectMergeSubstitutionSample)]

#### Using Substitution to Access Environment Variables

For substitutions which are not found in the configuration tree, it will be resolved by looking at system environment variables.

[!code-csharp[ConfigurationSample](../../../src/core/Akka.Docs.Tests/Configuration/ConfigurationSample.cs?name=EnvironmentVariableSample)]

An application can explicitly block looking up a substitution in the environment by setting a value in the configuration, with the same name as the environment variable. You could set `HOME : null` in your root object to avoid expanding `${HOME}` from the environment, for example:

[!code-csharp[ConfigurationSample](../../../src/core/Akka.Docs.Tests/Configuration/ConfigurationSample.cs?name=BlockedEnvironmentVariableSample)]

It's recommended that HOCON keys always use lowercase, because environment variables generally are capitalized. This avoids naming collisions between environment variables and configuration properties. (While on Windows `Environment.GetEnvironmentVariable()` is generally not case-sensitive, the lookup will be case sensitive all the way until the env variable fallback lookup is reached).

Environment variables are interpreted as follows:

* Env variables set to the empty string are kept as such (set to empty string, rather than undefined)
* If `Environment.GetEnvironmentVariable()` throws SecurityException, then it is treated as not present
* Encoding is handled by C# (`Environment.GetEnvironmentVariable()` already returns a Unicode string)
* Environment variables always become a string value, though if an app asks for another type automatic type conversion would kick in

##### Note on Windows and Case Sensitivity of Environment Variables

HOCON's lookup of environment variable values is always case sensitive, but Linux and Windows differ in their handling of case.

Linux allows one to define multiple environment variables with the same name but with different case; so both "PATH" and "Path" may be defined simultaneously. HOCON's access to these environment variables on Linux is straightforward; ie just make sure you define all your vars with the required case.

Windows is more confusing. Windows environment variables names may contain a mix of upper and lowercase characters, eg "Path", however Windows does not allow one to define multiple instances of the same name but differing in case.
Whilst accessing env vars in Windows is case insensitive, accessing env vars in HOCON is case sensitive.

So if you know that you HOCON needs "PATH" then you must ensure that the variable is defined as "PATH" rather than some other name such as "Path" or "path".
However, Windows does not allow us to change the case of an existing env var; we can't simply redefine the var with an upper case name.
The only way to ensure that your environment variables have the desired case is to first undefine all the env vars that you will depend on then redefine them with the required case.

For example, the the ambient environment might have this definition ...

```cmd
set Path=A;B;C
```

... we just don't know. But if the HOCON needs "PATH", then the start script must take a precautionary approach and enforce the necessary case as follows ...

```cmd
set OLDPATH=%PATH%
set PATH=
set PATH=%OLDPATH%
```

You cannot know what ambient environment variables might exist in the ambient environment when your program is invoked, nor what case those definitions might have. Therefore the only safe thing to do is redefine all the vars you rely on as shown above.

#### Self-Referential Substitutions

The idea of self-referential substitution is to allow a new value for a field to be based on the older value.

```text
path : "a:b:c"
path : ${path}":d"
```

is equal to:

```text
path : "a:b:c:d"
```

Examples of self-referential fields:

[!code-csharp[ConfigurationSample](../../../src/core/Akka.Docs.Tests/Configuration/ConfigurationSample.cs?name=SelfReferencingSubstitutionWithString)]
[!code-csharp[ConfigurationSample](../../../src/core/Akka.Docs.Tests/Configuration/ConfigurationSample.cs?name=SelfReferencingSubstitutionWithArray)]

Note that an object or array with a substitution inside it is **not** considered self-referential for this purpose. The self-referential rules do **not** apply to:

* `a : { b : ${a} }`
* `a : [${a}]`

These cases are unbreakable cycles that generate an error.

Examples:

[!code-csharp[ConfigurationSample](../../../src/core/Akka.Docs.Tests/Configuration/ConfigurationSample.cs?name=CircularReferenceSubstitutionError)]

#### The `+=` Field Separator

Fields may have `+=` as a separator rather than `:` or `=`. A field with `+=` transforms into a self-referential array
concatenation, like this:

a += b

becomes:

a = ${?a} [b]

`+=` appends an element to a previous array. If the previous value was not an array, an error will result just as it would in the long form `a = ${?a} [b]`. Note that the previous value is optional (`${?a}` not `${a}`), which allows `a += b` to be the first mention of `a` in the file (it is not necessary to have `a = []` first).

Example:

[!code-csharp[ConfigurationSample](../../../src/core/Akka.Docs.Tests/Configuration/ConfigurationSample.cs?name=PlusEqualOperatorSample)]

#### Examples of Self-Referential Substitutions

In isolation (with no merges involved), a self-referential field is an error because the substitution cannot be resolved:

foo : ${foo} // an error

When `foo : ${foo}` is merged with an earlier value for `foo`, however, the substitution can be resolved to that earlier value. When merging two objects, the self-reference in the overriding field refers to the overridden field. Say you have:

foo : { a : 1 }
foo : ${foo}

Then `${foo}` resolves to `{ a : 1 }`, the value of the overridden field.

It would be an error if these two fields were reversed, so:

foo : ${foo}
foo : { a : 1 }

Here the `${foo}` self-reference comes before `foo` has a value, so it is undefined, exactly as if the substitution referenced a path not found in the document.

Because `foo : ${foo}` conceptually looks to previous definitions of `foo` for a value, the optional substitution syntax `${?foo}` does not create a cycle:

foo : ${?foo} // this field just disappears silently

If a substitution is hidden by a value that could not be merged with it (by a non-object value) then it is never evaluated and no error will be reported. So for example:

foo : ${does-not-exist}
foo : 42

In this case, no matter what `${does-not-exist}` resolves to, we know `foo` is `42`, so `${does-not-exist}` is never evaluated and there is no error. The same is true for cycles like `foo : ${foo}, foo : 42`, where the initial self-reference are simply ignored.

A self-reference resolves to the value "below" even if it's part of a path expression. So for example:

foo : { a : { c : 1 } }
foo : ${foo.a}
foo : { a : 2 }

Here, `${foo.a}` would refer to `{ c : 1 }` rather than `2` and so the final merge would be `{ a : 2, c : 1 }`.

Recall that for a field to be self-referential, it must have a substitution or value concatenation as its value. If a field has an object or array value, for example, then it is not self-referential even if there is a reference to the field itself inside that object or array.

Substitution can refer to paths within themselves, for example:

bar : { foo : 42,
baz : ${bar.foo}
}

Because there is no inherent cycle here, the substitution will "look forward" (including looking at the field currently being defined). To make this clearer, in the example below, `bar.baz` would be `43`:

bar : { foo : 42,
baz : ${bar.foo}
}
bar : { foo : 43 }

Mutually-referring objects would also work, and are not self-referential (so they look forward):

// bar.a will end up as 4 and foo.c will end up as 3
bar : { a : ${foo.d}, b : 1 }
bar.b = 3
foo : { c : ${bar.b}, d : 2 }
foo.d = 4

One tricky case is an optional self-reference in a value concatenation, in this example `a` would be `foo` not `foofoo` because the self reference has to "look back" to an undefined `a`:

a = ${?a}foo
[!code-csharp[SerializationSetupDocSpec](../../../src/core/Akka.Docs.Tests/Configuration/SerializationSetupDocSpec.cs?name=StringSubstitutionSample)]
[!code-csharp[SerializationSetupDocSpec](../../../src/core/Akka.Docs.Tests/Configuration/SerializationSetupDocSpec.cs?name=ArraySubstitutionSample)]
14 changes: 0 additions & 14 deletions src/core/Akka.Docs.Tests/Configuration/ConfigurationSample.cs
Original file line number Diff line number Diff line change
Expand Up @@ -204,19 +204,5 @@
// // </BlockedEnvironmentVariableSample>
// }

// [Fact]
// public void UnsolvableSubstitutionWillThrowSample()
// {
// // <UnsolvableSubstitutionWillThrowSample>
// // This substitution will throw an exception because it is a required substitution,
// // and we can not resolve it, even when checking for environment variables.
// var hoconString = "from_environment = ${MY_ENV_VAR}";

// Assert.Throws<Exception>(() =>
// {
// Config config = hoconString;
// }).Message.Should().StartWith("Unresolved substitution");
// // </UnsolvableSubstitutionWillThrowSample>
// }
// }
//}
Original file line number Diff line number Diff line change
Expand Up @@ -125,5 +125,45 @@ public void SerializationSetupShouldWorkAsExpected()
}
// </Verification>
}
[Fact]
public void UnsolvableSubstitutionWillThrowSample()
{
// <UnsolvableSubstitutionWillThrowSample>
// This substitution will throw an exception because it is a required substitution,
// and we can not resolve it.

var hoconString = "from_environment = ${does-not-exist}";

Assert.Throws<FormatException>(() =>
{
Config config = hoconString;
}).Message.Should().StartWith("Unresolved substitution");
// </UnsolvableSubstitutionWillThrowSample>
}
[Fact]
public void StringSubstitutionSample()
{
// <StringSubstitutionSample>
// ${string_bar} will be substituted by 'bar' and concatenated with `foo` into `foobar`
var hoconString = @"
string_bar = bar
string_foobar = foo${string_bar}
";
var config = ConfigurationFactory.ParseString(hoconString); // This config uses ConfigurationFactory as a helper
config.GetString("string_foobar").Should().Be("foobar");
// </StringSubstitutionSample>
}
[Fact]
public void ArraySubstitutionSample()
{
// <ArraySubstitutionSample>
// ${a} will be substituted by the array [1, 2] and concatenated with [3, 4] to create [1, 2, 3, 4]
var hoconString = @"
a = [1,2]
b = ${a} [3, 4]";
Config config = hoconString; // This Config uses implicit conversion from string directly into a Config object
(new[] { 1, 2, 3, 4 }).Should().BeEquivalentTo(config.GetIntList("b"));
// </ArraySubstitutionSample>
}
}
}