-
Notifications
You must be signed in to change notification settings - Fork 694
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Source maps applied to wasm binaries #1051
base: main
Are you sure you want to change the base?
Conversation
Couldn't the wasm be disassembled to wast textual format and the links to the original source be attached to lines in the wast (rather than an offset into the wasm binary)? Or perhaps both? |
Web.md
Outdated
generate a stack trace). However existing developer tools implementations are | ||
designed around the source map model of passing a URL from the VM to the dev | ||
tools code, which fetches and interprets the information. Therefore it is | ||
epxected that it will be much simpler to use "real" source maps than to design |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sp: expected
Web.md
Outdated
|
||
Source maps specify how to map a (zero-based) line and column position in | ||
generated code to a file, line, and column location in the source. For | ||
wasm binary locations, the line number is always 0, and the column number |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could use the generated "line number" as the function index instead (or perhaps just the defined function index, skipping imported functions), and the generated "column" as the instruction offset in the function. But I suppose it's best to align w/ what's being discussed here.
Is it worth mentioning this in some rationale?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, On one hand the source map itself is not really meant to be read by humans. But if some tool displays it, then yeah I think it would make sense to display the offsets the way they'd be displayed in tools or browsers, yeah. And yeah it probably is worth mentioning. Maybe I'll do a PR for #990 too and add that to the rationale.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using column number (vs line number) in JS source maps gives a really good savings, to be precise the size of the wasm bytes (the ;
used to separate every encoded line and since we will have only one line, smaller number of column separators will be used)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We cannot use line number == 0 due to map encoding specifics. And 'source-map' library throws when line == 0. We need to change that to "the line number is always 1"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed. I had thought I used 0 in my original prototype, but it was actually 1.
@kanaka IIRC In Chrome and Firefox, currently when displaying wast (instead of original source) the JS engine disassembles the wasm binary and passes the wast text to the devtools. Using source maps would mean the source would be displayed instead of the wast. So in that case you wouldn't need the wast at all. |
I was thinking a binary source map format designed for WASM specifically would be more desirable than re-using the JavaScript format. But I'm not a browser vendor so I'm not sure how troublesome that might be. |
@RyanLamansky I agree that in principle a source-mapping format meant for wasm would not look like JavaScript source maps. However the intention of this convention is just to capture some really low-hanging fruit, because what we ultimately want is something much more capable than sourcemaps, and that will either have to be designed independently, or re-used from some other existing format (e.g. DWARF). So in one sense, this isn't worth doing at all if it isn't really simple, because it will be superseded, hopefully before too long. Therefore it should fit best with what browsers or devtools are already doing with source maps, which means that it would be nice not to require them to parse a format which is new, but also not the final form that we want. |
Web.md
Outdated
the URL is defined as in [RFC3986](https://tools.ietf.org/html/rfc3986) (e.g. | ||
it must be percent-encoded if necessary) and it may be a data URI. The URL | ||
is also resolved according to the source map spec, and may also be specified | ||
with the `SourceMap:` HTTP header. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a spec/reference to how HTTP headers will be associated with wasm module (assuming it's coming from Response)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would think this would only work for the new Response
-taking compile
and instantiate
overloads.
Web.md
Outdated
section named `"sourceMappingURL"` contains the URL. | ||
As with source maps, | ||
the URL is defined as in [RFC3986](https://tools.ietf.org/html/rfc3986) (e.g. | ||
it must be percent-encoded if necessary) and it may be a data URI. The URL |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This isn't sufficiently precise for web APIs; a real spec will need to reference http://url.spec.whatwg.org/
…, and reference WHATWG URL spec
Web.md
Outdated
it must be percent-encoded if necessary) and it may be a data URI. The URL | ||
is also resolved according to the source map spec, and may also be specified | ||
with the `SourceMap:` HTTP header. | ||
The URL is defined as in the the WHATWG |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the the
Aligning with WebAssembly/design#1051
Pinging this back to life :-) |
OK, I think have addressed the comments, although I'm not 100% sure if @domenic had anything more in mind beyond what I did about URLs and resolution. |
It seems reasonable given the general level of detail here. Thanks! |
Demo compiled using emscripten that is using source maps https://yurydelendik.github.io/sqlite-playground/src/playground.html (source at https://github.com/yurydelendik/sqlite-playground) |
@codehag please invite @yurydelendik to this call, too. |
I expect wasm debug info and remote debug prototcols to be largely orthogonal concerns. For example, the Firefox debugger used to apply source maps on the server side (ie the debuggee device) and has switched to applying them on the client side (ie the desktop displaying the UI). This tells me that they needn't be intertwined. Did you have something in mind that requires them to be developed together? |
On Chrome DevTools side, we think that JavaScript engine should provide low
level inspection and debugging tools using protocol, something like
Debugger and Runtime protocol domain for JavaScript but specific for
WebAssembly. Using this protocol tools would be able to do super basic
stuff, if tool needs something more fancy then it should fetch debugging
information from somewhere and use it to make everything nicer.
We think that this debugging information should be available for IDE not as
static format which should be parsed and consumed but as dynamic service
which will expose required API - we are working on proposal how this API
may look like right now. With this additional layer WebAssembly may
represent debugging information in any suitable for original language
format, maybe DWARF for C/C++ and SourceMap for something close to
JavaScript. Dynamic format will allow to resolve some well-known
limitations of existing sourcemap format, such as evaluating expressions in
original languages.
So we have basic protocol (works remotely as well) to do basic WebAssembly
specific inspection and debugging and we have separate service which IDE
may connect to and provide rich capabilities. This service internally uses
some sourcemap format which is suitable for original language and may be
developed by original language community.
I would be happy to join!
Thanks,
Aleksey.
…On Wed, Apr 25, 2018 at 8:51 AM Nick Fitzgerald ***@***.***> wrote:
As wasm can be run on mobile devices and we can't run DevTool there, so
wasm should support some kind remote debug protocol, that allows debug wasm
modules on mobile devices.
Will remote debug protocol be part of the wasm debug specification or it
will be handled by each browser by own remote debugging protocol?
I expect wasm debug info and remote debug prototcols to be largely
orthogonal concerns.
For example, the Firefox debugger used to apply source maps on the server
side (ie the debuggee device) and has switched to applying them on the
client side (ie the desktop displaying the UI). This tells me that they
needn't be intertwined.
Did you have something in mind that requires them to be developed together?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1051 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAaBsoyJbyKOWbYsemdqDTH0tjzm2ubYks5tsJtwgaJpZM4NNmuA>
.
|
I've created a doodle, please let me know of those times work / if we should alter the times a bit to make it easier for more to attend: https://doodle.com/poll/w7brfrgcbehv9zre |
Hi everyone. I will close the doodle on wednesday, so please make sure to vote for a date. In the meantime -- I have created an agenda, please comment if you want to add or remove something from the discussion: https://docs.google.com/document/d/1mMFN-OksiJZm1PD3XmBpAVq0vhT0df1rGKwYq2jfn58/edit# |
Hi everyone, based on the votes we have a tie between Monday and Wednesday. I would propose that we do Wednesday so that we have a little more time to prepare, @chicoxyzzy would you be able to make it? |
@codehag I'm not sure, but I can try. |
@chicoxyzzy In case you cant make it, and if there is anything you want to ask / propose please add it to the doc and we will make sure to cover it, there will be notes hopefully you can make it! |
I'd love to join the call and Wednesday works great for me. I'm currently working on doing C# debugging on top of WebAssembly and our current design aligns with the API proposals. |
Hi Yulia, Sorry I've been out of office for a couple of weeks. I'm back now and any of those dates are fine for me. |
Hello everyone :) In preparation for our meeting next week, I've compiled a list of requirements for wasm debugging capabilities. I've tried to focus solely on the kinds of tooling and queries we would like to support, and completely divorce these requirements from their eventual implementation (static format vs language server vs combo of those vs ...) Rather than proposing a particular solution, this is an attempt to enumerate the requirements and constraints that any solution must satisfy. This is what I've come up with: https://github.com/fitzgen/requirements-for-wasm-debugging-capabilities/blob/master/README.md Let me know what I missed ;) Looking forward to talking to y'all next week! |
Hey everyone, just a reminder -- meeting is at 7 pm CET (or 10:00am Pacific Time (PT)) on May 9th, there was a wrong date in the doc that i linked. |
Sorry again, its 10 AM* pt |
Thanks again to everyone who came to the meeting today :) I have completed my action item, a comparison between the requirements document and the "First version of DebuggingFramework.md" pull request. The comparison is here: fitzgen/wasm-debugging-capabilities#2 |
I just summarized my ideas about API instead of static format: here. |
I'm late to the game in this thread. Kenneth here from VS Code. Our JavaScript debuggers are heavily dependent on sourcemaps, and we'd love to be a part of this conversation, and future meetings. |
@auchenberg hey kenneth! great! we will set up a dedicated communication channel for this. if you want to catch up with the meeting notes from wednesday, you can find them here: https://docs.google.com/document/d/1mMFN-OksiJZm1PD3XmBpAVq0vhT0df1rGKwYq2jfn58/edit# |
Thanks for sharing @ak239! Before we go too deep exploring different designs, I think we should focus on defining requirements before proposing potential architectures and solutions. This way we all have a shared, common understanding of the problems we intend to solve, and we can weigh any proposed solution or architecture against these requirements. Everyone: please read through the requirements and file issues or send PRs for things that you believe are missing! :)
It seems that you are grouping browsers' developer tools (stepping debuggers, consoles, time profilers, etc) fall under the IDE umbrella, correct? I've enumerated a non-exhaustive, but hopefully representative set of tools we would like to support. I don't currently have IDEs listed there, mostly because I'd like to get a better sense of which specific capabilities they need and what their requirements are, but I filed an issue for that here. If there are any folks developing IDEs on this thread, it'd be great to get your comments in that issue! Here are some fairly unstructured thoughts while reading this, and thinking about the relationship between debugging capabilities and protocols:
|
@jfbastien @flagxor @binji |
@Becavalier the spec hasn't been implemented in firefox yet, however we do have debugging capability for wasm (I believe it has been since late last year / early this year). This is distinct from the spec. With regards to the spec, @dschuff will present a proposal for a subworking group to the web assembly working group and there will probably be a repository opened for iterating on the specification there. |
This PR proposes applying source maps to wasm binaries. Source maps obviously have a lot of known limitations and are not a long-term debugging solution. However they are widely supported (including by emscripten for asm.js code) and based on conversations with Chrome and Firefox dev tools folks, my hope is that it will be easy and straightforward to adapt the existing dev tools code to apply to wasm binaries in the meantime, and bring wasm to parity with asm.js.