Suggestions for new report features

[benmccann]

Hi, I maintain a somewhat popular web framework that's included in the reports. I'd like to use the data here to understand how we can have faster sites built with our framework and if there are features we need to add, evangelize, etc. And I'd also like to understand why some frameworks have higher CWVs and whether there's anything we can learn from what they're doing.

Some ideas that would be helpful:

• It would be incredibly helpful to have a list of origins or at least a random sampling of a few dozen each time I run the report. If I can see some actual sites it will give me a lot more insight as I can manually check a number of the things below for a small sample of sites.
• I'd love to see aggregate Lighthouse suggestions. E.g. what percentage of origins are given the feedback "serve images in next-gen formats", "defer offscreen images", "properly size images", "serve static assets with an efficient cache policy", etc.
• Things like number of images, number of DOM elements, number of cookies, website category might help understand things about the types of sites being built with a framework.

[rviscomi]

Thanks @benmccann, those are great suggestions. Would it be helpful if we got you that info in something like a custom report first? That way you can get answers to what you're looking for sooner and we can apply what worked best for you to the dashboard.

[benmccann]

Sure! That would be amazing

[rviscomi]

Cool. Want to set up a time to chat and work out the details? You can email me at [email protected].

[rviscomi]

Here's the sample of 100 SvelteKit and Astro sites, including their origin-level CWV performance: https://docs.google.com/spreadsheets/d/1YjIEI-52dFkxczhKyih5-SI8CKssQwHUoGXTeFkfm5Y/edit?usp=sharing

Query (2.37 GB)

WITH crux AS (
  SELECT
    CONCAT(origin, '/') AS root_page,
    fast_lcp / (fast_lcp + avg_lcp + slow_lcp) AS good_lcp,
    fast_inp / (fast_inp + avg_inp + slow_inp) AS good_inp,
    small_cls / (small_cls + medium_cls + large_cls) AS good_cls
  FROM
    `chrome-ux-report.materialized.device_summary`
  WHERE
    date = '2023-10-01' AND
    device = 'phone'
),

frameworks AS (
  SELECT
    technology,
    root_page,
    rank
  FROM
    `httparchive.scratchspace.frameworks`
  WHERE
    client = 'mobile' AND
    is_root_page AND
    technology IN ('SvelteKit', 'Astro')
),

sites AS (
  SELECT
    technology,
    root_page,
    rank,
    crux.* EXCEPT (root_page)
  FROM
    frameworks
  JOIN
    crux
  USING
    (root_page)
),

sveltekit AS (
  SELECT
    *
  FROM
    sites
  WHERE
    technology = 'SvelteKit'
  ORDER BY
    RAND()
  LIMIT
    100
),

astro AS (
  SELECT
    *
  FROM
    sites
  WHERE
    technology = 'Astro'
  ORDER BY
    RAND()
  LIMIT
    100
)


SELECT
  *
FROM
  sveltekit
UNION ALL
SELECT
  *
FROM
  astro
ORDER BY
  rank,
  NET.REG_DOMAIN(root_page)

[benmccann]

Thanks Rick!! I think I've found the main cause of SvelteKit slowness by looking at the slow sites from the spreadsheet, so no need to spend any more time on this. I'll investigate some more on my side and confirm.

[rviscomi]

Nice! Let us know if there's anything else you need.

FYI @sarahfossheim seems like sample URLs would be useful for the v2 dashboard.

Stashing this custom/LH audit query here for later...

/* https://github.com/HTTPArchive/cwv-tech-report/blob/main/opportunities/README.md */
CREATE TEMP FUNCTION GET_AUDITS(page STRING, lhr STRING, custom_metrics STRING) RETURNS ARRAY<STRUCT<metric STRING, audit STRING, passing BOOL, savings FLOAT64>> AS (
[
STRUCT(
  'LCP' AS metric,
  'server-response-time' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."server-response-time".score') AS FLOAT64) IS NULL AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."server-response-time".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'render-blocking-resources' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."render-blocking-resources".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."render-blocking-resources".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'redirects' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."redirects".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."redirects".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'uses-text-compression' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."uses-text-compression".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."uses-text-compression".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'uses-rel-preconnect' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."uses-rel-preconnect".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."uses-rel-preconnect".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'uses-rel-preload' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."uses-rel-preload".score') AS FLOAT64) IS NULL AS passing,
  NULL AS savings
),
STRUCT(
  'LCP' AS metric,
  'font-display' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."font-display".score') AS FLOAT64) >= 0.9 AS passing,
  NULL AS savings
),
STRUCT(
  'LCP' AS metric,
  'unminified-javascript' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."unminified-javascript".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."unminified-javascript".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'unminified-css' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."unminified-css".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."unminified-css".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'unused-css-rules' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."unused-css-rules".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."unused-css-rules".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'largest-contentful-paint-element' AS audit,
  NULL AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."largest-contentful-paint-element".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'prioritize-lcp-image' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."prioritize-lcp-image".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."prioritize-lcp-image".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'unused-javascript' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."unused-javascript".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."unused-javascript".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'efficient-animated-content' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."efficient-animated-content".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."efficient-animated-content".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'LCP' AS metric,
  'lcp-lazy-loaded' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."lcp-lazy-loaded".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."lcp-lazy-loaded".metricSavings.LCP') AS FLOAT64) AS savings
),
STRUCT(
  'INP' AS metric,
  'third-party-facades' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."third-party-facades".score') AS FLOAT64) IS NULL AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."third-party-facades".metricSavings.TBT') AS FLOAT64) AS savings
),
STRUCT(
  'INP' AS metric,
  'bootup-time' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."bootup-time".score') AS FLOAT64) IS NULL AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."bootup-time".metricSavings.TBT') AS FLOAT64) AS savings
),
STRUCT(
  'INP' AS metric,
  'mainthread-work-breakdown' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."mainthread-work-breakdown".score') AS FLOAT64) IS NULL AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."mainthread-work-breakdown".metricSavings.TBT') AS FLOAT64) AS savings
),
STRUCT(
  'INP' AS metric,
  'duplicated-javascript' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."duplicated-javascript".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."duplicated-javascript".metricSavings.TBT') AS FLOAT64) AS savings
),
STRUCT(
  'INP' AS metric,
  'legacy-javascript' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."legacy-javascript".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."legacy-javascript".metricSavings.TBT') AS FLOAT64) AS savings
),
STRUCT(
  'INP' AS metric,
  'viewport' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."viewport".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."viewport".metricSavings.INP') AS FLOAT64) AS savings
),
STRUCT(
  'CLS' AS metric,
  'layout-shift-elements' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."layout-shift-elements".score') AS FLOAT64) IS NULL AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."layout-shift-elements".metricSavings.CLS') AS FLOAT64) AS savings
),
STRUCT(
  'CLS' AS metric,
  'non-composited-animations' AS audit,
  JSON_VALUE(lhr, '$.audits."non-composited-animations".scoreDisplayMode') = 'notApplicable' AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."non-composited-animations".metricSavings.CLS') AS FLOAT64) AS savings
),
STRUCT(
  'CLS' AS metric,
  'unsized-images' AS audit,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."unsized-images".score') AS FLOAT64) >= 0.9 AS passing,
  SAFE_CAST(JSON_VALUE(lhr, '$.audits."unsized-images".metricSavings.CLS') AS FLOAT64) AS savings
),

/* Custom audits */
STRUCT(
  'LCP' AS metric,
  'custom-lazy-load' AS audit,
  ARRAY_LENGTH(`httparchive.fn.GET_LCP_LAZY`(JSON_QUERY(custom_metrics, '$.performance')).custom) = 0 AS passing,
  NULL AS savings
),
STRUCT(
  'LCP' AS metric,
  'custom-high-priority' AS audit,
  `httparchive.fn.GET_LCP_PRIORITY`(JSON_QUERY(custom_metrics, '$.performance')) = 'high' AS passing,
  NULL AS savings
),
STRUCT(
  'LCP' AS metric,
  'custom-statically-discoverable' AS audit,
  JSON_VALUE(custom_metrics, '$.performance.is_lcp_statically_discoverable') = 'true' AS passing,
  NULL AS savings
),
STRUCT(
  'LCP' AS metric,
  'custom-is-image' AS audit,
  JSON_VALUE(custom_metrics, '$.performance.lcp_elem_stats.url') != '' AS passing,
  NULL AS savings
),
STRUCT(
  'LCP' AS metric,
  'custom-same-host' AS audit,
  NET.HOST(JSON_VALUE(custom_metrics, '$.performance.lcp_elem_stats.url')) = NET.HOST(page) AS passing,
  NULL AS savings
)
]
);

WITH audits AS (
  SELECT
    technology,
    audit
  FROM
    `httparchive.scratchspace.frameworks`,
    UNNEST(GET_AUDITS(page, lighthouse, custom_metrics)) AS audit
  WHERE
    client = 'mobile' AND
    is_root_page AND
    /* Only look at pages failing the corresponding CWV metric. */
    CASE
      WHEN audit.audit = 'LCP' THEN SAFE_CAST(JSON_VALUE(payload, '$._CrUX.metrics.largest_contentful_paint.percentiles.p75') AS NUMERIC) > 2500
      WHEN audit.audit = 'CLS' THEN SAFE_CAST(JSON_VALUE(payload, '$._CrUX.metrics.cumulative_layout_shift.percentiles.p75') AS NUMERIC) > 0.1
      WHEN audit.audit = 'INP' THEN SAFE_CAST(JSON_VALUE(payload, '$._CrUX.metrics.interaction_to_next_paint.percentiles.p75') AS NUMERIC) > 200
    END
)

SELECT
  technology,
  COUNT(0) AS sites,
  audit.metric,
  audit.audit,
  COUNTIF(audit.passing) / COUNT(0) AS pct_passing,
  APPROX_QUANTILES(IF(audit.passing, NULL, audit.savings), 1000 IGNORE NULLS)[OFFSET(500)] AS median_savings
FROM
  audits
GROUP BY
  technology,
  metric,
  audit
ORDER BY
  sites DESC,
  metric,
  pct_passing

[benmccann]

To share my findings, the performance characteristics of SPA vs SSR apps is very different. While many frameworks only support one or the other, SvelteKit can run in different modes and supports both. It quite hard to compare SvelteKit to an SSR-only framework or SPA-only framework because our numbers are grouped together despite these being totally different deployment modes.

It would also be interesting to split SvelteKit on https://cwvtech.report into SvelteKit SSR and SvelteKit SPA. We can differentiate because the single page apps generally have a very small number of DOM elements before client-side rendering kicks in as pictured below. I'm not sure if that's enough to report on. I could also look into generating a meta tag or something to show what mode it's operating in. I think it would also allow us to highlight to our users the performance impacts of picking one vs the other.

Getting a full list of all SvelteKit URLs that appear on https://cwvtech.report would also be helpful to do some further investigations.

[benmccann]

Just checking in on this, do you think it'd be possible to split SvelteKit in the report into SvelteKit SSR and SvelteKit SPA?

[rviscomi]

Yeah that sounds doable from our side. A meta tag would definitely be a more idiomatic way to differentiate between modes, if possible. When available, you could let us know here or open a PR against the Wappalyzer detections directly to get it implemented.

[benmccann]

Thanks. Will do! It'll probably have to wait a couple months so that we can make the change as part of a major version. For now, I've filed an issue against the 3.0 milestone so that we don't lose track of it: sveltejs/kit#11724

[kevinfarrugia]

@benmccann I wrote a query that compares the LCP of website's using various frontend frameworks for client-side and server-side rendering.

If we look at mobile, SvelteKit client-side rendering has 36% (from 585 origins) of website's with good LCP, compared to 64% (from 3,302 origins) when using server-side. Let me know your feedback. I would be happy to update the query if needed.

Note that the query considers a web page client-side rendered if 75% of the HTML is generated on the client side.

[benmccann]

That is awesome!! Thank you!

My only other request would be if we could get Astro in the chart as well

[kevinfarrugia]

Hi @benmccann . I have added Astro (and Next.js) to the chart. Just a heads up that the number of origins (266) using Astro and client-side rendering is very small—I think this is somewhat expected.

[danielroe]

would love to see data for Nuxt as well if possible 🙏

[kevinfarrugia]

@danielroe I have updated it to include Nuxt.js. Note that you can add any technology that is tracked in HTTPArchive/wappalyzer.

I also changed the query to check page-level CrUX data instead of origin-level. This gives a more meaningful value (and explains the small differences with the charts posted earlier).

Feedback welcome.