Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wasm] No deterministic way to produce additional outputs for both building and publishing a WASM project #94576

Open
JakeYallop opened this issue Nov 9, 2023 · 6 comments
Assignees
Labels
arch-wasm WebAssembly architecture area-Build-mono needs-further-triage Issue has been initially triaged, but needs deeper consideration or reconsideration
Milestone

Comments

@JakeYallop
Copy link
Contributor

Issue details

I have a console WASM project set up, with my main file written in typescript. For a better user experience, I created a custom target that ran after WasmBuildApp, to run the TypeScript compiler and copy the built files to the AppBundle directory.

  <Target Name="_TypeScriptCompileAferBuild" AfterTargets="WasmBuildApp">
    <Exec Command="tsc -p ./tsconfig.json -outDir $(WasmAppDir)"></Exec>
  </Target>

In the above case, $(WasmAppDir) resolves to "bin/{Configuration}/net8.0/browser-wasm/AppBundle".

However this poses a problem when publishing. When a WASM project is published, there is a step that copies a bunch of files to the AppBundle directory. This step removes files that currently exist in the AppBundle directory. So, the published output for the project is now missing the files compiled in an earlier target.

And I can't just use the WasmNestedPublishApp target

  <Target Name="_TypeScriptCompileAferBuild" AfterTargets="WasmNestedPublishApp">
    <Exec Command="tsc -p ./tsconfig.json -outDir $(WasmAppDir)"></Exec>
  </Target>

As this does not run during a normal build!

Is there a target I can use that runs during build and after the bundle generation step, but does not run twice?

Workarounds

  • Run the step after BOTH targets - not ideal as the step runs twice
  <Target Name="_TypeScriptCompileAferBuild" AfterTargets="WasmBuildApp;WasmNestedPublishApp">
    <Exec Command="tsc -p ./tsconfig.json -outDir $(WasmAppDir)"></Exec>
  </Target>
  • Detect if in a publish step using $(_IsPublishing) and do something with that to dynamically configure when the target runs (its not something I've tried, so not 100% sure if this is possible). Again not ideal, as apparently $(_IsPublishing) is not set during a Visual Studio publish, and apparently detecting publish is not something we are meant to do - How to check if the current invocation is running publish? sdk#26324
  • Rerun the target only if expected outputs are not found - similar to the first workaround, but potentially slightly better - if all the inputs and outputs are the same then _WasmGenerateAppBundle is not run and files are preserved, meaning there is sometimes no need to rerun the target. This is still not ideal, as in most cases the target is still going to need to be executed twice.
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Nov 9, 2023
@lewing lewing added this to the 9.0.0 milestone Nov 9, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Nov 9, 2023
@radical radical added the arch-wasm WebAssembly architecture label Nov 10, 2023
@ghost
Copy link

ghost commented Nov 10, 2023

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Issue details

I have a console WASM project set up, with my main file written in typescript. For a better user experience, I created a custom target that ran after WasmBuildApp, to run the TypeScript compiler and copy the built files to the AppBundle directory.

  <Target Name="_TypeScriptCompileAferBuild" AfterTargets="WasmBuildApp">
    <Exec Command="tsc -p ./tsconfig.json -outDir $(WasmAppDir)"></Exec>
  </Target>

In the above case, $(WasmAppDir) resolves to "bin/{Configuration}/net8.0/browser-wasm/AppBundle".

However this poses a problem when publishing. When a WASM project is published, there is a step that copies a bunch of files to the AppBundle directory. This step removes files that currently exist in the AppBundle directory. So, the published output for the project is now missing the files compiled in an earlier target.

And I can't just use the WasmNestedPublishApp target

  <Target Name="_TypeScriptCompileAferBuild" AfterTargets="WasmNestedPublishApp">
    <Exec Command="tsc -p ./tsconfig.json -outDir $(WasmAppDir)"></Exec>
  </Target>

As this does not run during a normal build!

Is there a target I can use that runs during build and after the bundle generation step, but does not run twice?

Workarounds

  • Run the step after BOTH targets - not ideal as the step runs twice
  <Target Name="_TypeScriptCompileAferBuild" AfterTargets="WasmBuildApp;WasmNestedPublishApp">
    <Exec Command="tsc -p ./tsconfig.json -outDir $(WasmAppDir)"></Exec>
  </Target>
  • Detect if in a publish step using $(_IsPublishing) and do something with that to dynamically configure when the target runs (its not something I've tried, so not 100% sure if this is possible). Again not ideal, as apparently $(_IsPublishing) is not set during a Visual Studio publish, and apparently detecting publish is not something we are meant to do - How to check if the current invocation is running publish? sdk#26324
  • Rerun the target only if expected outputs are not found - similar to the first workaround, but potentially slightly better - if all the inputs and outputs are the same then _WasmGenerateAppBundle is not run and files are preserved, meaning there is sometimes no need to rerun the target. This is still not ideal, as in most cases the target is still going to need to be executed twice.
Author: JakeYallop
Assignees: radical
Labels:

arch-wasm, area-Build-mono

Milestone: 9.0.0

@radical
Copy link
Member

radical commented Nov 14, 2023

However this poses a problem when publishing. When a WASM project is published, there is a step that copies a bunch of files to the AppBundle directory. This step removes files that currently exist in the AppBundle directory. So, the published output for the project is now missing the files compiled in an earlier target.

What step is that? Can you share binlog of a build where this happens?

I would have suggested using WasmBuildAppDependsOn, and WasmNestedPublishAppDependsOn, but the workload manifest targets are imported after the project, and thus overwrite these properties. Yes, you would need to specify the target to run in both the cases, and you can differentiate with $(WasmBuildingForNestedPublish).

One solution would be to allow using these two *DependsOn properties. It makes sense, at least right now, for these two to be separate because build, and publish can be different. And you should not be seeing WasmBuildApp run after Build if you are publishing. But please share more information if you are running into such a case.

@JakeYallop
Copy link
Contributor Author

And you should not be seeing WasmBuildApp run after Build if you are publishing. But please share more information if you are running into such a case.

I have a basic reproduction here: https://github.com/JakeYallop/WasmBuildRepro - this is just the WASM console template with some targets added. You can see the target set to run after WasmBuildApp definitely gets run during a publish,

Additionally, testfile.txt does not exist in the AppBundle directly after doing a publish, when the bundle is actually generated.

What step is that? Can you share binlog of a build where this happens?

Its the WasmTriggerPublish step, and then inside that, I think its the _WasmGenerateAppBundle step, but I'm not 100% certain of that.

binlogs are zipped to allow sharing to github - just unzip them to access them.

binlog where testfile.txt does not exist in the AppBundle directory:
binlog.zip

binlog for the situation spoken about above, where the _WasmGenerateAppBundle target is skipped due to all inputs being the same. In this case, the testfile.txt does exist, as I'm assuming its been created during build then never removed during bundle generation.
image
binlog2.zip

@ilonatommy
Copy link
Member

ilonatommy commented May 31, 2024

@JakeYallop, let me summarize @radical's answer into code:

<Project Sdk="Microsoft.NET.Sdk">
	<PropertyGroup>
		<TargetFramework>net8.0</TargetFramework>
		<RuntimeIdentifier>browser-wasm</RuntimeIdentifier>
		<OutputType>Exe</OutputType>
		<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
		<WasmMainJSPath>main.mjs</WasmMainJSPath>
		<WasmRuntimeAssetsLocation>./</WasmRuntimeAssetsLocation>
	</PropertyGroup>

	<Target Name="InitializeWasmAppDir">
		<PropertyGroup>
			<WasmAppDir>$([MSBuild]::NormalizePath('$(OutputPath)/AppBundle'))</WasmAppDir>
		</PropertyGroup>
	</Target>

	<Target Name="__WriteMyFile" DependsOnTargets="InitializeWasmAppDir">
		<Message Importance="high" Text="Running WriteMyFile target after WasmBuildApp. WasmAppDir = $(WasmAppDir). This message appears when doing either a build or a publish." />
		<Exec Command="mkdir $(WasmAppDir)" Condition="!Exists('$(WasmAppDir)')" />
		<Exec Command="echo someText &gt; $(WasmAppDir)/testfile.txt" />
	</Target>

	<Target Name="__WriteMyFilePublish" AfterTargets="Publish" DependsOnTargets="__WriteMyFile" Condition="'$(_IsPublishing)' == 'true'">
		<Message Importance="high" Text="Message from publish!! This message does not appear when doing a build." />
	</Target>

	<Target Name="__WriteMyFileBuild" AfterTargets="Build" DependsOnTargets="__WriteMyFile" Condition="'$(_IsPublishing)' != 'true' AND '$(WasmBuildingForNestedPublish)' != 'true' " >
		<Message Importance="high" Text="Message from build!! This message does not appear when doing a publish." />
	</Target>

	<Target Name="__CheckDeps1" AfterTargets="__WriteMyFile" Condition="!Exists('$(WasmAppDir)/testfile.txt')">
		<Message Importance="high" Text="testfile.txt does not exist." />
	</Target>

	<Target Name="__CheckDeps2" AfterTargets="__WriteMyFile" Condition="Exists('$(WasmAppDir)/testfile.txt')">
		<Message Importance="high" Text="testfile.txt does exist." />
	</Target>
</Project>

Build:

> dotnet build -v d /bl:log.binlog
Restore complete (0.6s)
    Determining projects to restore...
    All projects are up-to-date for restore.
You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  WasmBuildRepro succeeded (1.0s) → bin/Debug/net8.0/browser-wasm/WasmBuildRepro.dll
    Running WriteMyFile target after WasmBuildApp. WasmAppDir = /workspaces/WasmBuildRepro/bin/Debug/net8.0/browser-wasm/AppBundle. This message appears when doing either a build or a publish.
    testfile.txt does exist.
    Message from build!! This message does not appear when doing a publish.
    Generated app bundle at /workspaces/WasmBuildRepro/bin/Debug/net8.0/browser-wasm/AppBundle/

Publish:

> dotnet publish -c Release -v d
Restore complete (0.6s)
    Determining projects to restore...
    All projects are up-to-date for restore.
You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  WasmBuildRepro succeeded (19.3s) → bin/Release/net8.0/browser-wasm/publish/
    Compiling native assets with emcc with -Oz. This may take a while ...
    [1/3] pinvoke.c -> pinvoke.o [took 0.132s]
    [2/3] corebindings.c -> corebindings.o [took 0.171s]
    [3/3] driver.c -> driver.o [took 0.259s]
    Linking for initial memory $(EmccInitialHeapSize)=16777216 bytes. Set this msbuild property to change the value.
    Linking with emcc with -O2. This may take a while ...
     "/workspaces/runtime/.dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.34.Node.linux-x64/8.0.5/tools/bin/node" /workspaces/runtime/.dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.34.Sdk.linux-x64/8.0.5/tools/emscripten/src/compiler.js /tmp/tmpx6dkob08.json --symbols-only
     "/workspaces/runtime/.dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.34.Sdk.linux-x64/8.0.5/tools/bin/wasm-ld" -o /workspaces/WasmBuildRepro/obj/Release/net8.0/browser-wasm/wasm/for-publish/dotnet.native.wasm /workspaces/WasmBuildRepro/obj/Release/net8.0/browser-wasm/wasm/for-publish/pinvoke.o /workspaces/WasmBuildRepro/obj/Release/net8.0/browser-wasm/wasm/for-publish/driver.o /workspaces/WasmBuildRepro/obj/Release/net8.0/browser-wasm/wasm/for-publish/corebindings.o /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libicudata.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libicui18n.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libicuuc.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libmono-component-debugger-stub-static.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libmono-component-diagnostics_tracing-stub-static.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libmono-component-hot_reload-stub-static.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libmono-component-marshal-ilgen-static.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libmono-ee-interp.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libmono-icall-table.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libmono-profiler-aot.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libmono-profiler-browser.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libmono-wasm-eh-wasm.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libmono-wasm-simd.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libmonosgen-2.0.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libSystem.Globalization.Native.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libSystem.IO.Compression.Native.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/libSystem.Native.a /workspaces/runtime/.dotnet/packs/Microsoft.NETCore.App.Runtime.Mono.browser-wasm/8.0.5/runtimes/browser-wasm/native/wasm-bundled-timezones.a -L/workspaces/runtime/.dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.34.Cache.linux-x64/8.0.5/tools/emscripten/cache/sysroot/lib/wasm32-emscripten -lGL -lal -lhtml5 -lstubs -lnoexit -lc -ldlmalloc -lcompiler_rt-wasm-sjlj -lc++-except -lc++abi-except -lunwind-except -lsockets -mllvm -combiner-global-alias-analysis=false -mllvm -wasm-enable-sjlj -mllvm -disable-lsr -mllvm -wasm-enable-eh -mllvm -exception-model=wasm --allow-undefined-file=/tmp/tmpv1jmykxr.undefined --export-if-defined=free --export-if-defined=htons --export-if-defined=malloc --export-if-defined=memalign --export-if-defined=memset --export-if-defined=ntohs --export-if-defined=stackAlloc --export-if-defined=stackRestore --export-if-defined=stackSave --export-if-defined=fmod --export-if-defined=atan2 --export-if-defined=fma --export-if-defined=pow --export-if-defined=fmodf --export-if-defined=atan2f --export-if-defined=fmaf --export-if-defined=powf --export-if-defined=asin --export-if-defined=asinh --export-if-defined=acos --export-if-defined=acosh --export-if-defined=atan --export-if-defined=atanh --export-if-defined=cbrt --export-if-defined=cos --export-if-defined=cosh --export-if-defined=exp --export-if-defined=log --export-if-defined=log2 --export-if-defined=log10 --export-if-defined=sin --export-if-defined=sinh --export-if-defined=tan --export-if-defined=tanh --export-if-defined=asinf --export-if-defined=asinhf --export-if-defined=acosf --export-if-defined=acoshf --export-if-defined=atanf --export-if-defined=atanhf --export-if-defined=cbrtf --export-if-defined=cosf --export-if-defined=coshf --export-if-defined=expf --export-if-defined=logf --export-if-defined=log2f --export-if-defined=log10f --export-if-defined=sinf --export-if-defined=sinhf --export-if-defined=tanf --export-if-defined=tanhf --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export=stackSave --export=stackRestore --export=stackAlloc --export=__errno_location --export=malloc --export=free --export=__trap --export=__wasm_call_ctors --export=ntohs --export=htons --export=__dl_seterr --export=emscripten_builtin_memalign --export=htonl --export-table --growable-table -z stack-size=5242880 --initial-memory=16777216 --no-entry --max-memory=2147483648 --global-base=1024
     "/workspaces/runtime/.dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.34.Node.linux-x64/8.0.5/tools/bin/node" /workspaces/runtime/.dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.34.Sdk.linux-x64/8.0.5/tools/emscripten/src/compiler.js /tmp/tmpllf_slq2.json
     "/workspaces/runtime/.dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.34.Sdk.linux-x64/8.0.5/tools/bin/wasm-opt" --strip-dwarf --post-emscripten -O2 --low-memory-unused --zero-filled-memory --pass-arg=directize-initial-contents-immutable --strip-debug --strip-producers /workspaces/WasmBuildRepro/obj/Release/net8.0/browser-wasm/wasm/for-publish/dotnet.native.wasm -o /workspaces/WasmBuildRepro/obj/Release/net8.0/browser-wasm/wasm/for-publish/dotnet.native.wasm -g --mvp-features --enable-exception-handling --enable-mutable-globals --enable-sign-ext --enable-simd
     "/workspaces/runtime/.dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.34.Node.linux-x64/8.0.5/tools/bin/node" /workspaces/runtime/.dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.34.Sdk.linux-x64/8.0.5/tools/emscripten/tools/acorn-optimizer.js /tmp/emscripten_temp_4vl40fj0/dotnet.native.js JSDCE minifyWhitespace --exportES6 -o /tmp/emscripten_temp_4vl40fj0/dotnet.native.jso1.js
     "/workspaces/runtime/.dotnet/packs/Microsoft.NET.Runtime.Emscripten.3.1.34.Sdk.linux-x64/8.0.5/tools/bin/wasm-opt" --print-function-map --quiet /workspaces/WasmBuildRepro/obj/Release/net8.0/browser-wasm/wasm/for-publish/dotnet.native.wasm --mvp-features --enable-exception-handling --enable-mutable-globals --enable-sign-ext --enable-simd
    Stripping symbols from dotnet.native.wasm ...
    Generated app bundle at /workspaces/WasmBuildRepro/bin/Release/net8.0/browser-wasm/AppBundle/
  WasmBuildRepro succeeded (3.9s) → bin/Release/net8.0/browser-wasm/publish/
    Optimizing assemblies for size. This process might take a while.
    Running WriteMyFile target after WasmBuildApp. WasmAppDir = /workspaces/WasmBuildRepro/bin/Release/net8.0/browser-wasm/AppBundle. This message appears when doing either a build or a publish.
    testfile.txt does exist.
    Message from publish!! This message does not appear when doing a build.

Message from build!! does not print on publish, so it should solve the issue.

@ilonatommy ilonatommy added the needs-author-action An issue or pull request that requires more info or actions from the author. label May 31, 2024
@ilonatommy ilonatommy self-assigned this May 31, 2024
@JakeYallop
Copy link
Contributor Author

Its been a while since I last looked at this project, but I believe my original issue still stands. My real issue requires all the files that are created after the build or bundle generation steps to already exist in $(WasmAppDir).

In the example code you've provided, I don't think the files created by bundle generation (or the build) will be present inside $(WasmAppDir) when the __WriteMyFile target runs.

I might revisit this project to get it working on .NET 9 once the WASM console templates have moved to the SDK, but at least for now its not something I'm looking at revisiting any time soon.

@dotnet-policy-service dotnet-policy-service bot added needs-further-triage Issue has been initially triaged, but needs deeper consideration or reconsideration and removed needs-author-action An issue or pull request that requires more info or actions from the author. labels May 31, 2024
@ilonatommy ilonatommy modified the milestones: 9.0.0, Future Jun 3, 2024
@maraf
Copy link
Member

maraf commented Jun 4, 2024

The WasmAppBuilder SDK is being replaced, for browser template already done, for console template tracked in #102743.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-wasm WebAssembly architecture area-Build-mono needs-further-triage Issue has been initially triaged, but needs deeper consideration or reconsideration
Projects
None yet
Development

No branches or pull requests

5 participants