Open question: how to report iframe performance?

[LarsKumbier]

Feature request summary

When lighthouse runs against a page with an iframe, the iframe is not adequately considered in the report.

If an iframe is slowing down a page, this should be reflected in the metrics and in the audits of lighthouse.

Affected results

Although we are in 2019 and there are much better options to include data from other pages, a lot of architectures still use iframes to include services - especially if the services are split across different companies.

What is the motivation or use case for changing this?

Lighthouse gives a wrong (or at least misleading) report, if an iframe has a very bad performance. Consider the attached zip-file. I've created a very small index.html, which references a ridiculously huge iframe. In addition, the iframe is artificially slowed down from nginx and limited to 50kbps. When I run lighthouse against the index.html, I get around 70 points performance. If I run lighthouse against the huge-dom.html, I get only 40 points.

In other words: although I load more on the index.html (since it includes the bad huge-dom.html) I get a better score than just evaluating the (bad) huge-dom.html. This is highly misleading.

Solution suggestion

I'm unsure on how to properly solve this, but my suggestion would be to gather the iframe along with all information and pass them through a complete evaluation round in lighthouse. Then the result needs to be aggregated by the parents page's audit report.

Example

This utilizes nginx and docker-compose. The huge-dom-html is artificially slowed down in nginx. After starting, get the port of the container via docker-compose port web 80 and run lighthouse once against http://localhost:<port>/index.html and then against http://localhost:<port>/huge-dom.html to get to similar results as described before.

lighthouse-iframe-example.zip

[patrickhulce]

Thanks for filing @LarsKumbier!

tl;dr - For those jumping in, if you have an iframe that is basically your primary content, yes, just audit it directly without the iframe wrapper. For everyone else, we're (mostly) happy with the balance that has been struck for now.

There are indeed some interesting challenges when handling iframes and you've stumbled onto a few here :) You might also find the discussion in #9566 very relevant around how we're thinking about some of these issues.

Yes, there are challenges with iframe performance especially when the role of iframes varies wildly (they can be out-of-process, offscreen with little main thread impact or be sizable portions of screen real estate and interactive content). We are thinking about them, but also fairly happy with the current status quo and likely won't make major changes anytime soon (see points below).

In other words: although I load more on the index.html (since it includes the bad huge-dom.html) I get a better score than just evaluating the (bad) huge-dom.html. This is highly misleading.

You are loading more overall, but the UX of index.html is also better, so the score should be higher. When you load index.html you get a paint of all of the primary content almost immediately compared to waiting several seconds for the huge-dom primary content. Given that the overwhelmingly common iframe use case on the web is ads that are not primary content and should not be assumed to be the primary content, I wouldn't call it "highly misleading" in the general case. Just unfortunate if you happen to care a lot about your iframe first paint performance compared to average.

I don't have the traces for those pages available to see why TTI is higher in the huge-dom case, and it's possible there's some improvement there to make those more similar, but overall I would expect the primary difference to remain paint timing which we stand behind as a reasonable tradeoff to make.

Given those challenges, I'm curious what you would hope to have happen in these iframe cases. Maybe a warning when an iframe is taking up too much of the visual viewport that the user should audit it directly instead?

[LarsKumbier]

@patrickhulce thanks for your insights! I'll try to make my case a little clearer - please read completely before answering and sorry for the long post.

tl;dr - For those jumping in, if you have an iframe that is basically your primary content, yes, just audit it directly without the iframe wrapper. For everyone else, we're (mostly) happy with the balance that has been struck for now.

It's a little bit more complicated than that. Consider this example: https://www.verivox.de/kreditkarten-vergleich/vergleich/
The outer page is generated from the CMS of Verivox, while the content itself inside the iframe is generated by an external page (financeads). I encounter this type of integration quite often on pages.
If I want to perform an audit, the score should reflect the combination of both, since the main content is actually loaded in the iframe.

There are indeed some interesting challenges when handling iframes and you've stumbled onto a few here :) You might also find the discussion in #9566 very relevant around how we're thinking about some of these issues.

Ah, seem to have missed that one - I had searched the issue list for a discussion and was surprised to not find much.

I read through the thread and yes, it's quite complex and unclear what the best path to solve this is. But I believe the issue is broader than what is being discussed there.

You are loading more overall, but the UX of index.html is also better, so the score should be higher. When you load index.html you get a paint of all of the primary content almost immediately compared to waiting several seconds for the huge-dom primary content. Given that the overwhelmingly common iframe use case on the web is ads that are not primary content and should not be assumed to be the primary content, I wouldn't call it "highly misleading" in the general case. Just unfortunate if you happen to care a lot about your iframe first paint performance compared to average.

To be honest, I did not consider advertisment-iFrames at all (thanks to an adblocker). But even considering them, I would argue the opposite case: If you blow up your page with ads and trackers, you should get a worse score. It's not like loading tons of "secondary content" won't affect your performance, so you should have a worse score in order to have a discussion on the performance impact that your ads and trackers impose on your users.

In addition, the other use case of iframes - loading primary content in a wrapper from a third party - does exist and is not reflected well at the moment. Both arguments would tip towards considering the impact of iframes more.

The testcase I attached is extreme, because I wanted to see, how lighthouse handles iframe content. But it shows what can happen, if you have the unfortunate situation where content from a third party has a bad performance and that lighthouse does not reflect that situation adequately in the score for the moment.

I don't have the traces for those pages available to see why TTI is higher in the huge-dom case, and it's possible there's some improvement there to make those more similar, but overall I would expect the primary difference to remain paint timing which we stand behind as a reasonable tradeoff to make.

It's not just the TTI and not just the huge-dom case, but I see why it's misleading. I'll cook up a better showcase to illustrate my problem, but the linked Verivox page above is already quite illustrative (for the moment).

If we compare the report above, we see huge differences in all metrics. FCP is obviously better in the (fast) outer frame, because it does not rely on the third party. SI, TTI and FCI are ~25% better than when loading the iframe alone, so they do not consider the iframe content adequately. The FMP result is misleading, since the testcase is inadequate (all content beside the iframe is indeed above the fold, so the layout-based approach FMP uses will fail in this case).

Given those challenges, I'm curious what you would hope to have happen in these iframe cases. Maybe a warning when an iframe is taking up too much of the visual viewport that the user should audit it directly instead?

I think there's two ways to solve this - depending on if you agree to my arguments or not (I also apologize if some of the parts suggested are already present - I am not as deep into the inner workings of lighthouse, lantern, etc. as I would like to be at the moment).

If you do not consider iframes as "should be regarded as performance-impact", I'd suggest an audit to identify iframes and inform in the report, that those have not been taken into account.

My preferred solution (if you do follow my train of thought and consider html-based iframes in contrast to javascript-injected iframes to be important for a website's performance), the traces of the parent page and the iframe should be interpreted together as a whole - with the iframe's content being considered as another render-blocking resource, loading other resources, etc. Performance impacts in the iframe would therefore be considered by the gatherers (at least those taking the trace as basis) and therefore seamlessly by all audits without any change (e.g. a long-loading request in the iframe would show up in the audit, which it doesn't at the moment).

This would not directly impact your current concerns on iframe-advertisements, because those are predominately loaded asynchronously by javascript.

As for the impact on FMP: I would consider the content of iframes with the same layout-based approach, but would change how they are interpreted. AFAIK, an iframe is considered as a block or non-interpretable data in the layout, similar to e.g. an image or a video element - without taking into consideration anything happening inside the iframe. If this is the case, I would suggest to interpret the iframe more like a div with child elements instead (in the context of FMP and the calculation of the number of objects, etc. - Reference: https://docs.google.com/document/d/1BR94tJdZLsin5poeet0XoTW60M0SjvOJQttKT-JK8HI/view).

A small advertisment would not have many child objects and not take up much space and would not change the score with the change. A bigger advertisment blocking or moving content would have an effect on the FMP with the change. In my specific case of a content-wrapper around an iframe, the iframe's data would take up considerable space in terms of (node size and layout space) and would adequately change the FMP score.

What are your thoughts on this?

[patrickhulce]

I think there are some crossed wires here.

Taking a step back, iframe content does impact all of the interactivity and main-thread based metrics for performance and the score will be worse in real-world examples with ads and other main-thread impacting junk provided they are not OOPIFs. i.e. your preferred solution has already been implemented. The specific example you were calling out in the original issue centers largely around first paint metrics which are special cased in the spec to explicitly exclude iframes.

SI, TTI and FCI are ~25% better than when loading the iframe alone, so they do not consider the iframe content adequately.

AFAICT, in this specific example it's only because of a single bug in --throttling-method=simulate regarding linking of frames to their layout events (thanks for raising 😃), but not due to some broad failure with handling of iframes in general (other throttling methods do not differentiate those metrics and in other examples with script execution instead of layout tasks, simulated throttling does not differentiate either). Agreed this bug should be fixed.

If the only discrepancy left between the two would be the first paint metrics, are your primary concerns addressed?

[LarsKumbier]

In this case it was my missing knowledge - thanks for clarifying! Good we found a bug that way. :)