[native_assets_cli][native_assets_builder] Supporting version skew between SDK and build.dart

[dcharkes]

With #26 we added a version number to to the BuildConfig and BuildOutput.
The way this works is that the SDK passes the version of the protocol that has rolled in to the Dart (and Flutter) SDK. If a user's build.dart cannot deal with that, the build is supposed to fail.
Similarly, the build.dart passes its value of the protocol in the build output. The Dart (and Flutter) SDK check the version and error if the output cannot be used.

@jonasfj suggested that we could use the Dart SDK version instead of the a version number in the protocol and use the SDK lower bound on the package being build to determine what version of the BuildConfig to provide to a package. The main benefit of this approach would be that if we need to do a breaking change to the build configuration, we don't end up in a situation where all dependencies with native assets need to be migrated before developers can update their Dart / Flutter SDK.

If the resolved package:native_assets_cli dependency in the package uniquely determines the protocol version (e.g. all protocol version bump also are a package version bump) we can also support the same behavior by looking at version of that package a package requires.

(With both approaches, the implementation in package:native_asset_cli needs to have a version number in the build config construction so it can keep constructing older versions.)

[jonasfj]

In short, I would suggest that you version the protocol based on the default languageVersion from package_config.json.

So specify input/output like:

• Inputs
  • out_dir (since Dart 3.1): Path where files referenced in assets must be placed.
  • package_root (since Dart 3.1): Path to root package. This is the user package/project the native assets are built for.
  • preferred_link_mode (since Dart 3.2): Either static or dynamic...
• Outputs
  • assets (since Dart 3.1): List of objects on the form:
    • name (since Dart 3.1): Name of the native asset when referenced from @Native attribute, must start with package:<my_package>/path/to/file.dart
    • link_mode (since Dart 3.1): Either static or dynamic...
  • supporting_asset (since Dart 3.3): Path to file that is included as a blob...

If my_package has has native assets and I declare:

name: my_package
environment:
  sdk: ^3.1.0   # This will make the default languageVersion in package_config `3.1`

Then my build script would :

• never be supplied the preferred_link_mode field, and,
• forbidden from outputting the supporting_asset field.

Thus, if my_package wants access to the preferred_link_mode field, I have to bump the SDK constraint to at-least ^3.2.0.
There is no risk that my build script assumes preferred_link_mode is always there, and then when built with an older Dart SDK it crashes.

Similarly, if my_package wants to output the supporting_asset field, then I must bump the SDK constraint to at-least ^3.3.0.
There is no risk my_package outputs the supporting_asset field and relies on the Dart SDK copying this, but when used with an older Dart SDK my package can't find the "supporting asset" and fails in weird manners.

---

It's a bit of work, but a table with inputs and output, field name, description and minimum language version would go far.

[dcharkes]

After more discussions with @jonasfj, relying on the language versions of a package doesn't work.

A package with language version 3.4 could use a helper package to parse BuildConfig and that helper package could have language version 3.3.

If we would like to support version skew, we should do protocol negotiation. build.dart (and link.dart) should support a mode in which they only return a protocol version. (And the native assets builder should be able to speak every version of the protocol indefinitely.)

If we would like to completely avoid version skew, we could consider not exposing a JSON protocol, but only BuildConfig and BuildOutput in a dart: library. Importing a dart: library means by construction a package works with the version of the Dart / Flutter that is running. (We could break the protocol in arbitrary ways, but we could never break the public API.)

A poor mans version of a dart: library would be to use a helper package, but pin the SDK constraint on a single dart version sdk: '>=3.3.0 <3.4.0' and release a new version of that package with every Dart release. This seems rather brittle.

[dcharkes]

Update on the approach:

1. We don't do any breaking changes in the BuildConfig ever. So newer Dart SDKs don't break older build hooks.
2. All new additions to BuildConfig have default values in Dart if they are missing in the JSON. So newer build hooks are not broken on older SDKs.
3. The BuildOutput will be serialized based on the version number from BuildConfig, so older SDKs are not broken by newer build hooks.
4. The BuildOutput can be parsed based on older JSON formats, so older build hooks are not broken with newer SDKs.

This means maintaining an implementation for older versions of BuildConfig and BuildOutput, which we do and cover with tests.

A changelog for both can be found in:

• https://github.com/dart-lang/native/blob/main/pkgs/native_assets_cli/lib/src/model/build_config_CHANGELOG.md
• https://github.com/dart-lang/native/blob/main/pkgs/native_assets_cli/lib/src/model/build_output_CHANGELOG.md

If we ever want to support breaking "1.", we should implement a handshake. E.g. hook/build.dart --version should return only back the version and then the SDK should talk with that version of BuildConfig.

[dcharkes]

1. We don't do any breaking changes in the BuildConfig ever. So newer Dart SDKs don't break older build hooks.

We run into a similar issue with versioning resources.json. And it would be nice if we could use the same solution in multiple place.

Ways to handle this more gracefully:

1. Protocol negotiation. Do a process call to the build and link hook to ask them what version they speak. The link hook should then also return a version for resources.json format.
   Downside: More process invocations.
   Upside: True and tried solution.
2. Write multiple .json files and pass them all in with a map from format version to file URI.
   Downside: The parse API of a package starts taking a Map<Version, Uri> instead of Uri. If the file path comes from a command-line invocation (e.g. --config=path/to/config.json) then that needs to be a multi argument.
3. Write a JSON that has a map with { "v1" : ..., "v2" : ... }.
   Downside: huge file.
   Upside: version skew between an older reader reading a newer file is completely hidden in the API of the package. The package itself deals with version skew both ways. (Newer readers reading older formats already works due to the parsers always being able to parse older versions.)

For all three options, we still want to aim to not do major version bumps, so these are all in case we really need to in the future.
For all three options, if we ever have to do a major version bump, this enables the eco system to migrate. We would then at some point do a breaking change announcement to remove an older major version.
For all three options, we will break the current build and link hook implementations.

From an encapsulation POV, I believe option 3 to be the cleanest. The package itself deals with version skew internally, it doesn't leak out into the API, and the callers don't have to know about it.

@mosuem Did I miss any pros/cons for the different options? You were advocating for option 2, some pros I missed there?

[jonasfj]

A package with language version 3.4 could use a helper package to parse BuildConfig and that helper package could have language version 3.3.

Actually, this could probably be made to work. But it might not be easy.

Suppose we have a world with:

• myapp, an application that uses foo.
• foo, a package with native assets that uses the helper bar package to create a build script.
• bar, a helper package that provides utilities for creating a build script.

When the build system has to build myapp, then the build system will:

• Find that foo has native assets that must be built
• Lookup the default language version of foo in package_config.json.
  Suppose this is 3.4.
• Communicate with build script from foo using native build protocol version 3.4.

For this to work, then bar (the helper package), will (once invoked) need to, either:

• (A) Take 3.4 as a parameter indicating the protocol version.
• (B) Read package_config.json to find the default language version of foo,
  which indicates the protocol version to be used.
  (This is probably the easiest thing to do)

Obviously not all versions of bar can support all versions of the native build protocol. Which notably could cause issues like:

• Author of foo bumps their default language version from 3.4 to 3.5, and if 3.5 protocol is not supported by bar then this will break the build.
  • Though this will probably be discovered in testing before a new version offoo is published.
  • This is probably solved by waiting for a new version of bar to be released.
• If a new version of bar is released that supports newer protocol versions, and foo bumps default language version, but forgets to bump the constraint on bar, then a consumer of foo might get an old version of bar which can't build the latest version of foo.
  • While sad, not bumping lower-bound constraints on a dependency is not a new issue.
    (This happens all the time, indeed pana has plans to do downgrade testing).
  • This is unlikely to happen, if most package authors use a caret-constraint bar: ^1.2.3.
  • Maybe this could be mitigated with lints or downgrade testing in pana.

---

Maybe, I'm missing something, and maybe it's too complicated.

But it kind of feel like writing the native build protocol version in the pubspec.yaml is a natural solution. Even if it means that a helper package like bar will have to support said version of the native build protocol.

And the best way to write the native protocol version in the pubspec.yaml is to use the default language version (which is derived from the SDK constraint, environment.sdk).

Or maybe the native build protocol version should be a completely different field in pubspec.yaml, and use a different version number than the SDK / language version.

[jonasfj]

Talking to Daco, I can certainly see how (3) is just much simpler all around 😄

[dcharkes]

Some notes from discussion with @jonasfj:

Exploring tying protocol version to Dart SDK version

• Tying the protocol version the Dart SDK (dev) version would require adding an entry to a map mapping SDK dev versions to protocol versions when rolling in dart-lang/native into the Dart SDK. One would have to guess the dev version based on what the next roll is going to be and hope that you guess the right one.
• Any reverts to the protocol version would create a non-monotonic mapping.
• One thing that we could do if we go for { "v1" : ... , "v2" : ... } is at some point stop adding v1 if we see all packages in the package graph have a language version of which the SDK always at least speaks v2.
  • Your main app is going to have a language version, but we can't use that for the hooks.
  • The native_assets_cli has a language version but the resolution of that package is likely going to be one of the newest ones (assuming we didn't do any breaking API changes), so we can't use that for the hooks.
  • Then we could try to use the package that contains the hook languages' version. But that doesn't work either because it could be simply only using package:native_assets_cli and package:native_toolchain_c. And if both of those packages support some new feature in the protocol, your build/link hook should automatically update to use that feature after a pub get.
  • But using the language versions of all packages in the whole dependency graph works.
  • But it's probably simpler to just do a breaking change announcement to intend to remove v1 a year or two after we've introduced v2 instead of all the magic.

[dcharkes]

Ways to handle this more gracefully:

4. Postfix existing keys in maps with _v2 when wanting to make a breaking change. This would prevent duplication inside the JSON for all other parts of JSON that are not going v2. Old parsers would just ignore the _v2s, just like they would have ignored a toplevel v2 in option 3. (Thanks @mkustermann for the input!)

For both option 3 and 4, the version: in the format would basically never be bumped to 2.x. Technically speaking, when removing at some point a xxx when a xxx_v2 has been there for a while, would be the actual breaking change. But, we don't want version comparing to fail. Instead the logic should be the same as dart: libs, a breaking change announcement, and then at some point just the removal. Not every breaking change to dart: libs triggers a major version bump to Dart. (Side note major version bumps to Dart are not actually indicative of breaking changes either, one can have two Dart packages that target a different major version e.g. 2.12 and 3.4 still work together!)

Then the question is should these formats have a version at all? Shouldn't they simply be versioned with the SDK? I believe it's always safe to have a version. The question is more what kind of logic is tied to that version. Should users ever have to worry about it? Does it show up in error messages etc?

I think our goal could be for Dart & Flutter stable releases that these versions are never really visible for users.

On dart dev releases and Flutter master channel they might show up while things are rolling. (Version skew between dart and flutter in a Flutter SDK for the build and link config and output.)

(From an implementation point of view, writing the parser and serializer from a certain version is easier than just assuming everything is nullable. It simplifies writing unit tests for checking that version skew works. Of course this is not really a concern for users.)

[dcharkes]

I believe we have explored enough alternatives and will stick with the current approach:

• Version in build/link config and output.
• The hook will output an older version if the config is older.
• The SDK will never break it's input (hooks can have an older version).
• Any breaking changes will be made non-breaking by using _2 keys.