Wappalyzer Technologies table has unexpected entries #1843
Comments
|
Dunno why this happens but can tell you the outliers are rare enough they can pretty much be ignored: SELECT DISTINCT
t1.category,
t1.app,
t1.total,
t2.category,
t2.app,
t2.total
FROM
(SELECT category, app, count(1) AS total FROM `httparchive.technologies.2020_10_01_mobile` GROUP BY category, app) t1,
(SELECT category, app, count(1) AS total FROM `httparchive.technologies.2020_10_01_mobile` GROUP BY category, app) t2
WHERE
REPLACE(t1.category, ' ', '') = REPLACE(t2.category, ' ', '') AND
REPLACE(t1.app, ' ', '') = REPLACE(t2.app, ' ', '') AND
(t1.category != t1.category OR t1.app != t2.app) AND
t1.total >= t2.total
ORDER BY
t1.category,
t1.app
|
|
Thanks @barrypollard . Agree that it's small enough can be ignored. I have ignored for now. |
|
@pmeenan not urgent, but since you were looking at this code there any ideas on this one? Very small numbers but odd that spaces are stripped very rarely. Remember looking at the time and saw same in WPT for those URLs (but not Wappalyzer website) so think a WPT issue was repeatable. Meant to raise and issue but forgot until you jolted my memory! |
|
@bazzadp If you can still repeat it, it would really help to have a repro case. I'm wondering if the Wappalyzer definitions were updated mid-crawl and the spaces were added. Since then the whole wappalyzer engine was updated and changed so it will be more useful if we can see it in the May 2021 crawl. |
|
https://jelly-pop.com/ still shows it. For example: https://webpagetest.org/jsonResult.php?test=210416_BiDcT0_20469058a349eb72f7a0548367c14bd3&pretty=1 has Google Analytics without a space:
Whereas https://almanac.httparchive.org/en/2020/ has Google Analytics with a space: https://webpagetest.org/jsonResult.php?test=210416_BiDcK5_5cd247abc1ef655159c773f30e457da1&pretty=1
|
|
That is bizarre. I wonder if the page itself is overriding some array or other ops because when I take the raw output from wappalyzer and run it through the same code not in the page, it keeps the spaces but if I run it on the console for that page then it strips them out (maybe a code page issue). At least I can reproduce it now though so it should be easier to fix |
|
Ahh, looks like the pages override string.trim() and cause it to remove all of the whitespace. Since the Wappalyzer definitions don't have any trailing whitespace I can just remove the trim operations. Should be fixed now (well, over the next hour as the agent update rolls out).
|
|
Why, why would anyone do this? You get all sorts when you look at 7.5 milllion web pages... Good work nailing it down. |
|
Thanks for tracking that down and fixing @pmeenan! Can we close this? |
|
Was going to give it a quick check after May crawl and then close it. There was also another issue where the technologies were all messed up as discussed on the HttpArchive slack. So I say let's leave this open as a reminder to check the technologies results after May crawl as it's a key data for the Web Almanac so want to make sure it's definitely sorted before our crawl month. |
|
Confirmed as all fixed in May crawl. Same query above gives 0 results. |
|
Reopening this to track a related issue. I noticed that "GoDaddy Website Builder" no longer has any websites detected since February: SELECT
_TABLE_SUFFIX AS suffix,
COUNT(DISTINCT url) AS urls
FROM
`httparchive.technologies.2021_*`
WHERE
app = 'GoDaddy Website Builder'
GROUP BY
suffix
ORDER BY
suffixInterestingly, ismyhostfastyet.com is still able to detect GWB sites because it uses a signal from the HTTP headers and it's detecting 7k desktop pages as of May. So I pulled out a URL from that detection and used the Wappalyzer extension for Chrome to get a true positive detection on https://finsbarandgrill.com/
However, in HA BQ and a plain WPT, we're only detecting Google Analytics: SELECT
*
FROM
`httparchive.technologies.2021_05_01_desktop`
WHERE
url = 'https://finsbarandgrill.com/'https://webpagetest.org/jsonResult.php?test=210616_AiDcFQ_67699b36864cd25851abcfd97c53f792&pretty=1 Wappalyzer uses a meta[name=generator] signal to detect GWB and the meta tag exists on the page as we'd expect: "GoDaddy Website Builder": {
"cats": [
1
],
"cookies": {
"dps_site_id": ""
},
"icon": "godaddy.svg",
"meta": {
"generator": "Go Daddy Website Builder (.+)\\;version:\\1"
},
"pricing": [
"mid"
],
"saas": true,
"website": "https://www.godaddy.com/websites/website-builder"
},// "Starfield Technologies; Go Daddy Website Builder 8.0.0000"
document.querySelector('meta[name=generator]').getAttribute('content')@pmeenan this leads me to believe that there may be an integration bug with Wappalyzer in WPT. Would you be able to look into this? |
|
The Wappalyzer checks weren't including the meta tags. Agent has been updated and will be rolling out over the next hour. Tested it in dev here and it correctly caught Go Daddy. "detected": {
"CMS": "GoDaddy Website Builder 8.0.0000",
"Analytics": "Google Analytics"
},
"detected_apps": {
"GoDaddy Website Builder": "8.0.0000",
"Google Analytics": ""
}, |
|
Thanks @pmeenan! I also noticed that the Wappalyzer extension detected React on that page, but it's not included in the new test results. Could something else be missing? |
|
Possibly the serialized DOM. It serializes the HTML but not the DOM. Taking a look now. |
|
Reached out to the Wappalyzer team to see how best to handle DOM-based detections. They are starting to migrate to it but the current engine doesn't directly support it and they have the extension doing the detections manually. Hoping there is a better way but should have something figured out soon. |
|
Whew. That was somewhat more painful than I expected. Had to rewrite the JS variable detection part which changed pretty significantly when the engine changed a few months back (also added the support for the DOM detections). Here is an updated test. Change is rolling out to prod (and HA) over the next hour. "_detected": {
"CMS": "GoDaddy Website Builder 8.0.0000",
"JavaScript libraries": "React 16.13.1,Lodash 4.17.5",
"Analytics": "Google Analytics"
},
"_detected_apps": {
"GoDaddy Website Builder": "8.0.0000",
"React": "16.13.1",
"Lodash": "4.17.5",
"Google Analytics": ""
} |
|
That's great, thanks for fixing! |
|
Something to keep an eye on to verify that the fix is working. Here's a query to measure the change in origins from January to May for all technologies, using the table for the CWV Technology Report dashboard: SELECT
app,
SAFE_DIVIDE(may.origins - jan.origins, jan.origins) AS pct_change,
may.origins - jan.origins AS num_change,
jan.origins AS jan_origins,
may.origins AS may_origins
FROM (
SELECT
date,
app,
origins
FROM
`httparchive.core_web_vitals.technologies`
WHERE
date = '2021-01-01' AND
client = 'mobile' AND
origins >= 1000) AS jan
JOIN (
SELECT
date,
app,
origins
FROM
`httparchive.core_web_vitals.technologies`
WHERE
date = '2021-05-01' AND
client = 'mobile') AS may
USING (app)
ORDER BY
pct_change ASCTop 20 results:
The bug appears to be more widespread than I'd initially thought and some technologies like Angular were almost entirely wiped out. The June dataset should have the fix partially applied, so we should start to see these rebounding. Eventually everything should be fully counted in the July crawl. @pmeenan WDYT about adding some kind of automated testing to ensure that the Wappalyzer integration is working? Not sure if that'd be implemented on the WPT or HTTP Archive side, or if it's something easy to build as a standalone WPT API app. |
|
Seeing detections for the 20 most affected technologies in the previous comment starting to recover in the June dataset. For example, here's a screenshot from the CWV Technology Report:
Given that the fix was applied late in the June crawl, detections haven't fully recovered, so I'll leave this issue open and continue to monitor this when the July crawl is available. |
|
@rviscomi - Can we close this now? |
|
Yes we can close this now. Tracking improvements to technology detections in HTTPArchive/wappalyzer#70 instead. |
Sometimes I am seeing entries which are not in Wappalyzer apps.json file - https://github.com/WPO-Foundation/Wappalyzer/blob/master/src/apps.json
For example, under eCommerce category, we have duplicate entries (With and without spaces).
You can see this in output of query
SELECT distinct app FROMhttparchive.technologies.2020_10_01_mobileWHERE category = 'Ecommerce' order by appNot sure why this is happening. This is resulting in slight over counting in some queries (For example - Total number of eCommerce platforms analyzed)
We should check why this is happening. Impact of this on 2020 chapter is minimal so I am not spending time to get to bottom of this for now and just raising an issue so that we can look into this later.
Also, If you look at site https://jelly-pop.com/, in technologies table, it shows app as 'SalesforceCommerceCloud' but if you see technologies using Wappalyzer chrome extension, this technology is not shown. Not sure why, this is appearing in technologies table.
Also, noticed this under 'Analytics' category and saw entries like -
The text was updated successfully, but these errors were encountered: