-
-
Notifications
You must be signed in to change notification settings - Fork 905
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate Nokogiri::XML::Document to the TypedData API #2807
Migrate Nokogiri::XML::Document to the TypedData API #2807
Conversation
Oh I forgot to mention, compared to #2806, given we unwrap Also the memsize and memsize_node can be made static, will push an amended commit. |
63b18c3
to
2e25ff1
Compare
Thanks for starting this! One note: I don't think we need to implement memsize, Nokogiri forces libxml2 to use |
That's unrelated. Using The |
Also IMHO this PR is good to go. The |
@byroot Thanks for the additional context, I'm not familiar at all with Any reason we didn't implement |
Yup. That's right.
That's a good question. The reason is you want to avoid double accounting, so if you have multiple objects that ultimately point to the same native structures, you don't want to report the same memory regions twice. So given that So when |
Urk:
I'm not 100% sure, but that looks like a Ruby bug, nokogiri is nowhere to be seen in the backtrace. cc @peterzhu2118, what do you think? |
Full crash for posterity: https://gist.github.com/byroot/150db7e3eb5900da552dd795791fca90 I see it's ruby 2.7.0-p0, it may make sense to test on latest 2.7 instead. |
I get that crash with ruby 3.2.0 as well. |
On other branches? |
Here's the stack walkback:
|
Yes, the absence is an implicit |
Oh nice:
So that was the original issue, and while trying to print that warning, it allocated an caused an actual crash 🤣 |
Ok, my understanding of that stack trace is:
This is very intriguing, I'll dig a bit more. But just an observation that this crash wouldn't happen if LibXML was using regular |
I added that feature in ruby/ruby#6921, it's probably not a good idea to use |
Just give me a bit of time to dig into what's going on, thanks. The memory model here is complicated and a bit fragile, unfortunately. |
Ok, so the culprit is here: nokogiri/ext/nokogiri/xml_document.c Line 20 in 5ec39e6
I'm pretty sure the issue already exist on
Yeah, no rush at all. |
Seems like the origin of this And the reason is that these nodes are no longer in the document, but I think they could just be directly freed with |
What's happening here is that two text nodes in a row are being "deallocated" in Let's sit on this for a bit and do the rest of the classes. I need to think about how to work around this. |
Ah, @casperisfine suggested removing the |
Co-authored-by: Jean Boussier <[email protected]>
Co-authored-by: Jean Boussier <[email protected]>
2e25ff1
to
7216317
Compare
OK, I've rebased onto |
This is a tricky one to explain, but in summary, when dealloc_node_i2 parents unparented nodes, libxml2 may merge two adjacent text nodes, which causes additional memory allocations during GC. If the FREE_IMMEDIATELY flag is set, this will generate warnings. But generating those warnings during GC will lead to a segfault for reasons I haven't dug into yet. Anyway, let's leave this flag off for now.
7216317
to
0804380
Compare
I've added a commit to improve the fidelity of the memsize calculation, and added some test coverage. |
9ea11fc
to
b026f60
Compare
and include test coverage for it
b026f60
to
6b23461
Compare
Thanks, @etiennebarrie and @casperisfine ! |
See #2808 Note that I chose to make a new public function, `noko_xml_node_set_unwrap`, to allow other files to unwrap node sets. This differs from the tactic adopted in #2807 / cb1557d which was to make public the data type. We should probably decide which is the better approach and standardize on it.
Ref: #2806
Paired with @byroot. First we created a
rb_data_type_t
and then used it withTypedData_Get_Struct
everywhere we're unwrapping anxmlDoc
.Then we added a limited version of
dsize
, which counts all the children nodes of the document recursively.This allows
ObjSpace.memsize_of
to be closer to reality:After the second commit:
We're missing other members in the struct, not sure which ones would the most important to have a correct size, but this is a good start.