Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Single file apps in .net 5 #90

Merged
merged 10 commits into from
Feb 20, 2020
Merged

Single file apps in .net 5 #90

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
fb69d53
Single file apps in .net 5
swaroop-sridhar Jan 30, 2020
11912ba
Address feedback from @jkotas, and @vitek-karas
swaroop-sridhar Feb 6, 2020
3409697
Address additional feedback
swaroop-sridhar Feb 14, 2020
c20c821
Fix probe ordering
swaroop-sridhar Feb 14, 2020
10a95f6
Make clarification about Appcontext.BaseDirectory
swaroop-sridhar Feb 14, 2020
315be14
Use standard convention for writing .NET Core
swaroop-sridhar Feb 15, 2020
9450e53
Address feedback by @lpereira
swaroop-sridhar Feb 15, 2020
ca4395c
Add a note about bundle relative paths
swaroop-sridhar Feb 15, 2020
c594b3a
Address feedback by @vitek-karas and @jkotas.
swaroop-sridhar Feb 20, 2020
c1255e0
Update accepted/single-file/design.md
swaroop-sridhar Feb 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 28 additions & 29 deletions accepted/single-file/bundler.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# The Single-file Bundler

### Requirements
## Requirements

Ideally, the bundler should be:

Expand All @@ -9,46 +9,45 @@ Ideally, the bundler should be:
* Deterministic (generate the exact same single-file on multiple runs)
* Amenable to post-processing (ex: signing tools)

### Bundle Transformation
## Bundle Layout

Given a .net core app published with respect to a specific runtime, the bundler transforms the `AppHost` binary to a single-file bundle by:
| Bundle Layout (ver 2.0) |
| ------------------------------------------------------------ |
| **AppHost**<br />`(Bundle-Marker)` |
| **Embedded Files**<br />`app.dll` <br />`app.deps.json`<br />`app.runtimeconfig.json`<br />`dependency.managed.dll`<br />`...`<br /> |
| **Bundle Header** <br />`2.0` (Version #)<br /> `#Number of Embedded Files`<br />`Bundle-ID` <br />`Needs Extraction?`<br />`deps.json file location (offset, size)`, if any.<br />`runtimeconfig.json file location (size,offset)`, if any. |
| **Bundle Manifest**<br />For each bundled file:<br /> `Location (Offset, Size)`<br /> `Type: IL, ReadyToRun, deps.json, runtimeconfig.json`, or `other` (extract) |

* Appending the managed app and its dependencies as a binary-blob at the end of the `AppHost` executable.
* Writing meta-data to identify the binary as a single-file bundle, and a manifest of the embedded files.
### Bundle Marker

#### Bundle Layout
Every `AppHost` has a static variable that identifies the location of the single-file header (if any). By default, this value is zero, which indicates that the `AppHost` is not a bundle. The bundler tool rewrites this value with the location of the bundle header.

The bundling tool will append the following contents to the `AppHost` binary:
Using a special marker for recognizing bundle-header (instead of simply writing the header at the end of the file) enables compatibility with other post-processing tools (such as `signtool`) which require their own content to be at the end of the file.

* The actual files to be published into the single file (including the managed app)
* A bundle header containing:
* The bundler tool version
* A bundle identifier: which is a *path-compatible* cryptographically strong name.
* This identifier is used to distinguish bundles for different versions of the same app.
* This identifier is used as part of the bundle extraction mechanism as described in [this document](extract.md).
* Currently, a new bundle identifier is generated for each bundle transformation.
In future, the bundle identifiers should be generated by hashing the contents of the bundle -- so that bundle transformation is deterministic.
* Offset of the bundle manifest
### Bundle Identifier

* A bundle manifest that describes:
* The location of embeded files (offset and size)
* The type of the embedded files: MSIL assemblies, ready-to-run assemblies, configuration files (ex: `app.deps.json` `app.runtimeconfig.json`), or others.
The bundle identifier is a *path-compatible* cryptographically strong name.

Every `AppHost` has a static variable that identifies the location of the single-file header (if any).
By default, this value is zero, which indicates that the `AppHost` is not a bundle.
The bundler tool rewrites this value with the location of the bundle header.
This ensures that the bundle-header is not position constrained (ex: at the end of the file), so that tools like `signtool` can post-process the bundle file.
* This identifier is used to distinguish bundles for different versions of the same app.
* This identifier is used as part of the bundle extraction mechanism as described in [this document](extract.md).
* A new bundle identifier is generated for each bundle transformation.

### Implementation
## Bundle Transformation

The bundler should ideally be located close to the core-host, since their implementation is closely related. Therefore, the bundler will be implemented in the `core-setup` repo.
Given a .net core app published with respect to a specific runtime, the bundler transforms the `AppHost` binary to a single-file bundle through the following actions:

* Append the managed app and its dependencies as a binary-blob at the end of the `AppHost` executable.
* Write meta-data headers and manifest that help locate the contents of the bundle.
* Set the bundle-indicator in the `AppHost` to the offset of the bundle-header.

## Implementation

The bundling tool is implemented as a library in the [Microsoft.NET.HostModel](https://www.nuget.org/packages/Microsoft.NET.HostModel/) package. This library is used by the SDK in order to publish a .net core app as a single-file.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was looking at it just recently. Is there a reason for why we have this split into Microsoft.NET.HostModel and a task that lives in the SDK? It may be useful to write down the reason for the split here. (I was expecting to have one .dll with the msbuild task that does everything.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-exhaustive list of reasons:

  1. msbuild tasks need to be multi-targeted as they must carry all their dependencies

  2. cli uses hostmodel to generate apphosts for global tools without running a build

  3. we can keep all the localized error messages in the sdk repo where we have loc infrastructure

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also it was deemed desirable to have reading and writing code in the same repo to more easily evolve the format and unit test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @nguerrera. Some of these reasons are documented in the Bundler's readme file. I'll add the details here too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have updated the ReadMe file, now that a few repos have changed:
dotnet/runtime#32583


### Further work
The bundler should generate the correct format of single-file bundles based on the target framework.

#### Codesign on Mac
## Limitations

Codesign tool on mac-os performs strict consistency checks, and cannot tolerate the additional files appended at the end of the `AppHost` executable.
[Further work](https://github.com/dotnet/core-setup/issues/7065) is necessary to update the binary headers to make the bundle file compatible for signing.
* Currently, a new bundle identifier is generated for each bundle transformation. In future, the bundle identifiers should be generated by hashing the contents of the bundle -- so that bundle transformation is deterministic.
* `Codesign` tool on mac-os performs strict consistency checks, and cannot tolerate the additional files appended at the end of the `AppHost` executable. [Further work](https://github.com/dotnet/core-setup/issues/7065) is necessary to update the binary headers to make the bundle file compatible for signing.

Loading