Encode codemap and span information in crate metadata. #22235

michaelwoerister · 2015-02-12T19:28:52Z

This allows to create proper debuginfo line information for items inlined from other crates (e.g. instantiations of generics). Only the codemap's 'metadata' is stored in a crate's metadata. That is, just filename, positions of line-beginnings, etc. but not the actual source code itself.

Crate metadata size is increased by this change because spans in the encoded ASTs take up space now:

                BEFORE    AFTER
libcore         36 MiB    39.6 MiB    +10%
libsyntax       51.1 MiB  60.5 MiB    +18.4%
libcollections  11.2 MiB  12.8 MiB    +14.3%

This only affects binaries containing metadata (rlibs and dylibs), executables should not be affected in size.

Fixes #19228 and probably #22226.

alexcrichton · 2015-02-16T23:08:56Z

Do you know if it would be easy enough to track which spans are associated with generic functions and which aren't? It'd be nice to just cut down on the metadata size slightly (4mb for libcore is pretty big)

michaelwoerister · 2015-02-18T10:47:26Z

I'll see what I can do to omit span data for types and other stuff that is not used later anyway. I'll also experiment with encoding spans in LEB128 and see if that helps in addition to the LZ77 encoding we are doing anyway.

I'll be on vacation for the rest of the week, however, so please don't expect any updates before next week.

Manishearth · 2015-02-18T13:27:39Z

Needs a rebase, sorry.

michaelwoerister · 2015-02-24T21:52:24Z

So, I've been investigating different options of reducing space usage for spans but I've not been really successful.

At the moment it's not possible to omit some spans from metadata but not others. The reason is that spans occur in the AST and we use derive-generated Encode/Decode implementations for these data structures. If we had a #[dont_serialize_me] attributes for fields, we could do something here. However, I didn't want to add this feature in a rudimentary, ad-hoc fashion, just for this one use case.
I made the compiler store spans in the variable-sized LEB128 format also used in DWARF. This however didn't give the expected space reduction because our RBML writer is super-wasteful, tagging each u8 I emit with a type-tag and a byte storing the size of the number. This results in at least 3 bytes and up to 12 bytes per BytePos and between 6 and 24 bytes per Span which totally defeats the purpose of storing things in LEB128.

A maybe acceptable intermediate solution is to store spans as one u64 in order to cut down on RBML tagging overhead. In a preliminary test this worked quite well under the circumstances. I'll push a cleaned up version of this the day after tomorrow.

The real problem though is how metadata is encoded in general (an issue already recorded in #21482). It's super-verbose and I doubt that all this tagging really has much of a benefit. Coming up with a better data format will be necessary sooner or later.

alexcrichton · 2015-02-25T06:07:01Z

Wow that is... depressing. It's not necessarily the end of the world to inflate metadata size right now, it's just something to think about :). If you've got a smaller format working though, that sounds great!

michaelwoerister · 2015-02-25T19:22:21Z

Wow that is... depressing.

That's one way to see it. I see the current metadata encoding as a lustrous garden of low-hanging fruit, overripe and ready for the picking. How often does one get the chance to shave off a third of the space consumption at no cost at all.

We'll have to pin down the requirements for the metadata format though before the feast can start.

michaelwoerister · 2015-02-26T14:14:35Z

OK, this is updated now. Numbers are better than before.

                BEFORE    AFTER
libcore         39   MiB  41   MiB    +5%
libsyntax       54.6 MiB  61.2 MiB    +12%
libcollections  14.3 MiB  15.3 MiB    +7%

alexcrichton · 2015-02-27T02:34:20Z

Ok, this is all looking good to me, nice work @michaelwoerister!

I'd like a second set of eyes on this, however.

r? @nrc or @brson

nrc · 2015-02-27T02:44:58Z

I need to give this a proper read, but I think the idea is good.

One question: what about macro expansions? I guess we don't use them in debug info? Is it ever possible we'd want to have this information cross-crate?

michaelwoerister · 2015-02-28T11:23:44Z

Yes, macro expansion information is not used in debuginfo. DWARF has some provisions for recording C-macro definitions (not expansions) but that's not even supported by LLVM. I guess this won't be a topic for us in the foreseeable future.

About having this information cross-crate in general: It would be easy to implement, I think, but it costs additional space and at the moment I don't know of a use case. For now, I'd just leave it out.

nrc · 2015-03-01T22:39:09Z

src/librustc/metadata/creader.rs

+            }
+        }
+
+        return true;


nit: unnecessary return

nrc · 2015-03-01T23:23:42Z

r+ with the nits addressed

michaelwoerister · 2015-03-02T19:50:10Z

Thanks for the review. I work in your comments on Wednesday. Don't worry about nittiness :)

lifthrasiir · 2015-03-03T07:07:36Z

The space optimization for filemap may have to be revisited after #22971 lands. The relevant commit is fe73d38, which makes the tag represent the size of following integers. If you keep individual tags in the filemap then you don't need separate bytes_per_diff; otherwise you will need Opaque to elide tags at all.

bors · 2015-03-03T10:45:18Z

☔ The latest upstream changes (presumably #22971) made this pull request unmergeable. Please resolve the merge conflicts.

This allows to create proper debuginfo line information for items inlined from other crates (e.g. instantiations of generics). Only the codemap's 'metadata' is stored in a crate's metadata. That is, just filename, line-beginnings, etc. but not the actual source code itself. We are thus missing the opportunity of making Rust the first "open-source-only" programming language out there. Pity.

michaelwoerister · 2015-03-04T13:03:49Z

I think I've addressed all comments. The encoding of the line data in filemaps is not optimal yet. I started implementing a better version using emit_opaque() as @lifthrasiir suggested but it got so ugly and unwieldy that I decided to do this in a separate PR later.

michaelwoerister · 2015-03-04T13:03:54Z

@bors r=nrc

bors · 2015-03-04T13:03:54Z

@bors r=michaelwoerister 2f88655

bors · 2015-03-04T14:47:52Z

⌛ Testing commit 2f88655 with merge 3b3bb0e...

…oerister This allows to create proper debuginfo line information for items inlined from other crates (e.g. instantiations of generics). Only the codemap's 'metadata' is stored in a crate's metadata. That is, just filename, positions of line-beginnings, etc. but not the actual source code itself. Crate metadata size is increased by this change because spans in the encoded ASTs take up space now: ``` BEFORE AFTER libcore 36 MiB 39.6 MiB +10% libsyntax 51.1 MiB 60.5 MiB +18.4% libcollections 11.2 MiB 12.8 MiB +14.3% ``` This only affects binaries containing metadata (rlibs and dylibs), executables should not be affected in size. Fixes #19228 and probably #22226.

bors · 2015-03-04T17:18:47Z

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-x-android-t, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-win-32-nopt-t, auto-win-32-opt, auto-win-64-nopt-t, auto-win-64-opt

michaelwoerister mentioned this pull request Feb 12, 2015

Parameterized struct doesn't get debug symbols for any of its methods #22226

Closed

michaelwoerister force-pushed the cross-crate-spans branch 6 times, most recently from 15d0206 to 855fe9f Compare February 26, 2015 13:24

rust-highfive assigned nrc Feb 27, 2015

nrc reviewed Mar 1, 2015
View reviewed changes

src/librustc/metadata/creader.rs

}

}

return true;

Copy link

Member

nrc Mar 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: unnecessary return

michaelwoerister force-pushed the cross-crate-spans branch from 855fe9f to 2f88655 Compare March 4, 2015 11:24

bors merged commit 2f88655 into rust-lang:master Mar 4, 2015

This was referenced Mar 4, 2015

std: Deprecate std::old_io::fs #22873

Merged

Minor refactoring/modernization/reintegration/enhancement of --pretty. #23017

Closed

Refactor supertrait handling #23026

Merged

alexcrichton mentioned this pull request Mar 5, 2015

dep-info includes all transitvely linked files #23089

Closed

Manishearth mentioned this pull request Mar 7, 2015

Overflow in span subtraction / astencode #23115

Closed

dtolnay mentioned this pull request Jun 2, 2017

Badly behaved Encode and Decode impls for FileMap #42374

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encode codemap and span information in crate metadata. #22235

Encode codemap and span information in crate metadata. #22235

michaelwoerister commented Feb 12, 2015

alexcrichton commented Feb 16, 2015

michaelwoerister commented Feb 18, 2015

Manishearth commented Feb 18, 2015

michaelwoerister commented Feb 24, 2015

alexcrichton commented Feb 25, 2015

michaelwoerister commented Feb 25, 2015

michaelwoerister commented Feb 26, 2015

alexcrichton commented Feb 27, 2015

nrc commented Feb 27, 2015

michaelwoerister commented Feb 28, 2015

nrc Mar 1, 2015

nrc commented Mar 1, 2015

michaelwoerister commented Mar 2, 2015

lifthrasiir commented Mar 3, 2015

bors commented Mar 3, 2015

michaelwoerister commented Mar 4, 2015

michaelwoerister commented Mar 4, 2015

bors commented Mar 4, 2015

bors commented Mar 4, 2015

bors commented Mar 4, 2015

Encode codemap and span information in crate metadata. #22235

Encode codemap and span information in crate metadata. #22235

Conversation

michaelwoerister commented Feb 12, 2015

alexcrichton commented Feb 16, 2015

michaelwoerister commented Feb 18, 2015

Manishearth commented Feb 18, 2015

michaelwoerister commented Feb 24, 2015

alexcrichton commented Feb 25, 2015

michaelwoerister commented Feb 25, 2015

michaelwoerister commented Feb 26, 2015

alexcrichton commented Feb 27, 2015

nrc commented Feb 27, 2015

michaelwoerister commented Feb 28, 2015

nrc Mar 1, 2015

Choose a reason for hiding this comment

nrc commented Mar 1, 2015

michaelwoerister commented Mar 2, 2015

lifthrasiir commented Mar 3, 2015

bors commented Mar 3, 2015

michaelwoerister commented Mar 4, 2015

michaelwoerister commented Mar 4, 2015

bors commented Mar 4, 2015

bors commented Mar 4, 2015

bors commented Mar 4, 2015