SQL and MD folders the 2024 Web Almanac (HTTPArchive#3666)

* upload 2024 * change mds * Test update * Revert test update * Fix line endings --------- Co-authored-by: Barry Pollard <[email protected]>
HakaCode · May 29, 2024 · b4a0839 · b4a0839
1 parent ddcef7d
commit b4a0839
Show file tree

Hide file tree

Showing 53 changed files with 1,061 additions and 0 deletions.
diff --git a/sql/2024/accessibility/README.md b/sql/2024/accessibility/README.md
@@ -0,0 +1,20 @@
+# 2024 Accessibility queries
+
+<!--
+  This directory contains all of the 2024 Accessibility chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/1anCSQk9g_YDfZP6GtjqdC-vCfnCNZAUEQwjSr8AzqTw/edit
+[~google-sheets]: https://docs.google.com/spreadsheets/d/1btB1r9QpdgTyToPhn7glcGAdMFs7eq4UcQSVIHBqiYQ/edit#gid=1778117656
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/accessibility.md
diff --git a/sql/2024/caching/README.md b/sql/2024/caching/README.md
@@ -0,0 +1,20 @@
+# 2024 Caching queries
+
+<!--
+  This directory contains all of the 2024 Caching chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/19nHfMkAxIYBPoPfnH8xmq1KP4mHH59z_RVgtF_UT8KU/edit
+[~google-sheets]: https://docs.google.com/spreadsheets/d/1XxN2xIrJk2YffeqhlWuNjPPrt5rT7uoi4BxM4gCFhvk/edit#gid=1137634974
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/caching.md
diff --git a/sql/2024/capabilities/README.md b/sql/2024/capabilities/README.md
@@ -0,0 +1,20 @@
+# 2024 Capabilities queries
+
+<!--
+  This directory contains all of the 2024 Capabilities chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/1YgvVNPicdrqsxccrk64jdDusbHPjbPcIYKnsmXW9BwM/edit
+[~google-sheets]:https://docs.google.com/spreadsheets/d/1Ig-821tyjr897i8QqPvXiRMY9o444qsFAmZt4AFyBjk/edit#gid=0
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/capabilities.md
diff --git a/sql/2024/cdn/README.md b/sql/2024/cdn/README.md
@@ -0,0 +1,26 @@
+# 2024 CDN queries
+
+<!--
+  This directory contains all of the 2024 CDN chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+Query updates:
+- Dates have been updated
+
+
+
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/11Yz8S-e3ltbYQPdzKX1E3oexfA2PwWLdA5tToDv98BI/edit
+[~google-sheets]:https://docs.google.com/spreadsheets/d/15YXQQjyoQ0Bnfw9KNSz_YuGDiCfW978_WKEHvDXjdm4/edit#gid=745368492
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/cdn.md
diff --git a/sql/2024/cms/README.md b/sql/2024/cms/README.md
@@ -0,0 +1,20 @@
+# 2024 CMS queries
+
+<!--
+  This directory contains all of the 2024 CMS chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/13CxAp7HCcxHHCSuEnXS2rolKskLSlUvLQuqUD6QADYc/edit
+[~google-sheets]: https://docs.google.com/spreadsheets/d/118lwQV_GwFYqIxXvsm57oeadJdjAJEOMCRq1PsTqhfs/edit#gid=355498918
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/cms.md
diff --git a/sql/2024/compression/README.md b/sql/2024/compression/README.md
@@ -0,0 +1,29 @@
+# 2024 Compression queries
+
+<!--
+  This directory contains all of the 2024 Compression chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+Queries:
+* Content types using HTTP compression
+* Server settings for HTTP compression. (http vs https)
+* Trends in HTTP compression (4 year trend)
+* First vs Third Party compression
+* Lighthouse compression byte savings
+* Lighthouse compression scores
+
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/13DsZfvQIWVa668pJMFtW8gL-uAlt9TjGCbgVcCdgN0Y/edit
+[~google-sheets]: https://docs.google.com/spreadsheets/d/1LihF3OD7Uq4A9cynpTntsOeD2ZN7cTSErsyNyWjuy2Y/edit#gid=1369271609
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/compression.md
diff --git a/sql/2024/cookies/README.md b/sql/2024/cookies/README.md
@@ -0,0 +1,20 @@
+# 2024 Cookies queries
+
+<!--
+  This directory contains all of the 2024 cookies chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/1o2AgdsDq_x3OvthZF7Kb50rUKMVLn7UANT9Stz7ku2I/edit#heading=h.ymg495uvm3yx
+[~google-sheets]: https://docs.google.com/spreadsheets/d/1wDGnUkO0rgcU5_V6hmUrhm1pq60VU2XbeMHgYJEEaSM/edit#gid=454016814
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/cookies.md
diff --git a/sql/2024/css/README.md b/sql/2024/css/README.md
@@ -0,0 +1,20 @@
+# 2024 CSS queries
+
+<!--
+  This directory contains all of the 2024 CSS chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/1hMRUNA862CypoKXjRcQtZJpCDpb16jUSAIR6Nm2uWwE/edit
+[~google-sheets]: https://docs.google.com/spreadsheets/d/1R40dZYFdszjciIpS2jFMC3mPpZC-xHJnEE42xTEVY2w/edit#gid=1778117656
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/css.md
diff --git a/sql/2024/ecommerce/README.md b/sql/2024/ecommerce/README.md
@@ -0,0 +1,22 @@
+# 2024 Ecommerce queries
+
+<!--
+  This directory contains all of the 2024 Ecommerce chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/1wWCadW1fIlRhzQWxLFhu6fKbXiwuewr2JdIMijYh2Ds/edit
+[~google-sheets]: https://docs.google.com/spreadsheets/d/1LABlisQFCLjOyEd43tdUb-Hxs6pGuboTresntMk71Lc/edit#gid=1075995078
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/ecommerce.md
+
+
diff --git a/sql/2024/fonts/README.md b/sql/2024/fonts/README.md
@@ -0,0 +1,20 @@
+# 2024 Fonts queries
+
+<!--
+  This directory contains all of the 2024 Fonts chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/1ljEHbDvXComXnW5s_EXZ0nM3_JCLnYr28Xrcf0YYtP8/edit#heading=h.vp0ukb2pxxzp
+[~google-sheets]: https://docs.google.com/spreadsheets/d/1EkdvJ8e0B9Rr42evC2Ds5Ekwq6gF9oLBW0BA5cmSUT4/edit#gid=1778117656
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/fonts.md
diff --git a/sql/2024/http/README.md b/sql/2024/http/README.md
@@ -0,0 +1,20 @@
+# 2024 HTTP queries
+
+<!--
+  This directory contains all of the 2024 HTTP chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/1-vPF7GivpV-fbdj8VE6GaMpb1BJHGxLvK2ElBtc_0ls/edit
+[~google-sheets]:https://docs.google.com/spreadsheets/d/1PfTZkbmgyLA3NmEICeCyxpMWgP7cKY7EsZl9RciE5S4/edit#gid=140668849
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/http.md
diff --git a/sql/2024/interoperability/README.md b/sql/2024/interoperability/README.md
@@ -0,0 +1,20 @@
+# 2024 Interoperability queries
+
+<!--
+  This directory contains all of the 2024 Interoperability chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/15leFm5uN-SE_biA1eyOwzQI69Ybl_2OvcN6u0lysjuo/edit
+[~google-sheets]: https://docs.google.com/spreadsheets/d/1IlG_RbYue60RkdA4-ZeDA3u_mHn9cpM8VM6J66CS4bM/edit#gid=1778117656
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/interop.md
diff --git a/sql/2024/jamstack/README.md b/sql/2024/jamstack/README.md
@@ -0,0 +1,20 @@
+# 2024 Jamstack queries
+
+<!--
+  This directory contains all of the 2024 Jamstack chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/17nSxLz18zVQBLTbo2NTIfaLmEPKrDl0MfltRjE1v6vI/edit
+[~google-sheets]: https://docs.google.com/spreadsheets/d/1wKswSnp8TuN4aZb63ir7hE5eCikIu8zEdQYPp4Xo6O0/edit#gid=807625051
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/jamstack.md
diff --git a/sql/2024/javascript/README.md b/sql/2024/javascript/README.md
@@ -0,0 +1,20 @@
+# 2024 JavaScript queries
+
+<!--
+  This directory contains all of the 2024 JavaScript chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/1pGEUO47dYsob45oic_yXWP5neBHzNVN_B5pwlSg8bqA/edit
+[~google-sheets]: https://docs.google.com/spreadsheets/d/16isMe5_rvmRmJHtK5Je66AhwO8SowGgq0EFqXyjEXw8/edit#gid=1778117656
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/javascript.md
diff --git a/sql/2024/markup/README.md b/sql/2024/markup/README.md
@@ -0,0 +1,20 @@
+# 2024 Markup queries
+
+<!--
+  This directory contains all of the 2024 Markup chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/1jml04hWOXy2RYvlQoE6IbjB946obwX5X_t0ZgTZA9qg/edit
+[~google-sheets]: https://docs.google.com/spreadsheets/d/1TtOMr_w58HvqNBv4RIWX021Lxm6m5ajYOcRykrPdAJc/edit#gid=1778117656
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/markup.md
diff --git a/sql/2024/media/README.md b/sql/2024/media/README.md
@@ -0,0 +1,54 @@
+# 2024 Media queries
+
+<!--
+  This directory contains all of the 2024 Media chapter queries.
+
+  Each query should have a corresponding `metric_name.sql` file.
+  Note that readers are linked to this directory, so try to make the SQL file names descriptive for easy browsing.
+
+  Analysts: if helpful, you can use this README to give additional info about the queries.
+-->
+
+## Notes for 2024
+
+2021’s analysis was largely done on custom metrics, which work on what they can see using JS APIs, in the DOM. As such, we worked primarily looked at `<img>` and `<video>` elements, and there were a bunch of things about the actual image resources that we had to work really hard to discover (e.g., `approximateResourceWidth`), or just couldn’t determine at all (is this GIF animated?)
+
+The best thing that happened in 2024 was getting a bunch of metadata about the image resources themselves. Pat Meenan figured out how to get the crawler to run the actual loaded resource bytes through both ImageMagick’s `identify`, and ExifTool. This is available in `$._image_details`.
+
+The biggest unaccomplished goal in 2024 was being able to join up this amazing information about the resources themselves, with the information we glean from Custom Metrics about how those resources were embedded on pages. We no longer need to jump through hoops to calculate `approximateResourceWidth`, or guess at file formats using heuristics. We can know these things definitively now. But I wasn't able to join up the one dataset to the other.
+
+This might be accomplishable in 2024 by getting `_image_details` into [`$WPT_REQUESTS`](https://github.com/HTTPArchive/custom-metrics/blob/d4cdb38201c6c870589edaeb946950656c8009ca/dist/responsive_images.js#L447), accessible within Custom Metric code. Failing that it seems possible to JOIN `_image_details` JSON with the `responsive_images` custom metric JSON over URLs.
+
+It pains me that I never figured out a way to look at background images (or favicons or image resources loaded by JS and painted to a canvas) within custom metrics. Queries that start based on requests tally these non-`<img>` images; queries that look at `responsive-images` (which looks at `<img>` elements), do not.
+
+In general there is a lot of untapped potential in the new ImageMagick and EXIFTool results, but we found this year that these tools report things frustratingly-inconsistently, and getting good results for all of the web's images requires lots of testing and edge cases. I spent probably half of my analysis time this year trying to get good, cross-format color space and depth data, but the way that image formats deal with colorspaces, and the way that `identify` reports things, make this quite tricky! I scaled back my ambitions this year by only looking at ICC profiles (AVIF and PNG allow direct embedding of colorspace info, which `identify` reports in various ways... I digress). Akshay fell into a similar trap trying to write Animated Image queries that dealt not only with Gifs, but with Animated PNGs and AVIFs. We ended up not being able to do that analysis.
+
+The duplication and separation of custom metrics is unfortunate. `media.js`, `Images.js`, and `responsive-images.js` have a lot of duplication and overlap. They could stand a unification and cleanup, if only for clarity's sake.
+
+I didn't do anything new with video at all -- these are just Doug's queries from 2021 with updated dates. I'm positive more can be done here, not sure I'm the person to do it.
+
+---
+
+Some additional notes, that came out of the document review process:
+
+> I've seen multiple sites using `<picture>` where `srcset`/`sizes` would be enough, and even better, without type switching nor art direction. I don't know if this is something that can be detected, maybe for next year.
+
+– Nicolas Hoizey
+
+> Maybe we could add (next year?) details about which types are used in `<source>`s, in which order. (I've seen type switching with JPEG first… 😅)
+>
+> Also maybe useful:
+> - the number of `<source>`s for art direction
+> - wheither the latest `<source>` is useful or `<img srcset sizes>` would be enough, for both use cases (another frequent mistake)
+
+– Nicolas Hoizey
+
+## Resources
+
+- [📄 Planning doc][~google-doc]
+- [📊 Results sheet][~google-sheets]
+- [📝 Markdown file][~chapter-markdown]
+
+[~google-doc]: https://docs.google.com/document/d/1aIMU2N03Fof01066KuMseVdlVdlEmwUGRsB5LsJIOZE/edit
+[~google-sheets]: https://docs.google.com/spreadsheets/d/1Q2ITOe6ZMIXGKHtIxqK9XmUA1eQBX9CLQkxarQOJFCk/edit#gid=1778117656
+[~chapter-markdown]: https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/src/content/en/2024/media.md