Final Outline Discussions

[SachaG]

LINK TO THE SURVEY PREVIEW

• https://survey.devographics.com/en-US/survey/state-of-html/2023

As we finalize the outline, I'll centralize my feedback/ideas here.

Main Features Issues

• The "Content" section only has a single main item so I'd suggest merging it into a generic "Other Features" section.
• Same with Using/Making Web Components, I think they can be merged into a single "Web Components" section.
• Calling a section "Accessibility & i18n" I think diminishes the importance of Accessibility a bit. I'd suggest moving i18n features to the "Other Features" section instead.
• Also, why does the Accessibility section not have any main features? That feels weird to me if only from a consistency standpoint (and it also feels like a11y is treated as less important than the other sections).
• Generally, I don't know if it's a good idea to include one or more freeform questions per features section, as I probably won't have time to analyze that data myself. Could we get by with a couple freeform questions in the Usage section like for other surveys?
• Overall I think we're probably asking about way too many features? The "External Content" page alone asks about 46 distinct features, which is comparable to what we usually ask about in a single survey. Even though a lot of them are in the checkbox format that still feels like a lot? Unless we haven't actually started the process of paring down the list, in which case ignore my comment.

Other Features Issues

• For selectmenu (and maybe a couple other features) the code example is in the description field.
• The "accordion element" feature's code example does not actually use an accordion element?

Other Tools

• How relevant are the "Templating languages" and "Lightweight markup languages" questions? I feel like they don't quite fit in with the rest of the survey, personally.
• "Macros & Snippets" also seems a bit light on content. We could have a more generic "Other Tools" question within the section if we can manage to find at least 4-5 items.
• The current Interactivity question should probably just be "front-end frameworks".
• If we want to capture people who add interactivity by using "vanilla" JS with no framework we can add a separate "Interactivity" question that targets this more specifically.

Usage

• To stay consistent with other surveys I'll rename the "Missing features and pain points" section to "Usage".
• In other surveys, the "Which existing HTML features are you unable to use?" is entitled "Browser Incompatibilities". Maybe we should keep that for consistency?
• I think we could probably seed the "Which components do you find yourself using across multiple projects?" question with some common predefined options?
• "If you could add 3 elements to HTML, which would you add?" could probably have a more generic "Missing Features" title as we might want to include things that are not elements?
• It seems to be that we'll get very similar answers to these two questions, so I'm not sure we need both? In what scenario would someone answer them differently?
• It's weird that accordion is in that list given that it's also one of the features in a previous section?
• I don't think "drawer" or "menu" are specific enough to give us actionable data?

Other Notes

• I'm not sure the "Most of your projects use…" question about language is that useful?

Checkbox Entities

The features mentioned in checkboxes don't have associated entities, which means we can't include them in the survey currently. These entities can be simpler, we basically just need id and name fields. My main question is:

• Should I start the work of giving them ids and then inputting them manually in the YAML file; or will Lea create a script to automate this process?

Other Missing Entities

(In addition to the features mentioned in the checkbox questions)

• formdata
• badging_api
• isolated_webapps
• web_share_api
• window_controls_overlay
• text_fragments
• borderless_mode

Additional Questions

Other questions we could add:

• which browser do you use? <-not needed
• testing environment <-not needed
• what do you use html for?
• industry sector

[SachaG]

More about checkbox feature entities. The minimum info we need is:

- id: autocomplete_element
  name: '`autocomplete="hint"`'
  tags:
    - features

Note that if the name needs to be translated then the name in the entity YAML won't even be used, so we could even just have:

- id: autocomplete_element
  tags:
    - features

In which case the contents of the name field would be added as a translation string in each locale. I can handle that part if it's more convenient.

That being said, there is no downside to having the full info:

- id: autocomplete_element
  name: '`autocomplete="hint"`'
  description: Provide hints about what kind of information should be autocompleted.
  example:
    language: html
    code: |
      <input name="first_name" autocomplete="given-name" />
  mdn: en-US/docs/Web/HTML/Attributes/autocomplete
  resources:
    - url: https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#attr-fe-autocomplete
      title: HTML Standard
  tags:
    - features

The advantages of doing so are:

• If we decide to "promote" that feature to a "main feature" then no additional work is required
• That extra content will appear when people add the feature to their reading list – even if it's not a "main feature"
• That feature could in theory appear in other parts of the survey or survey results where that extra metadata could be helpful.

Implementation Details

Maybe the easiest way to make sure we keep ids consitent is so specify each feature's id in the Coda doc using a separator? For example:

• <input type="color"> | input_type_color

Non-Entity Options

Not all options need to be entities. In that case the option simply has an id and a translation string, with no entity metadata attached. That being said in the case of this survey I do think almost every option for checkbox questions is an entity.

[SachaG]

Second round of feedback, focusing on the main remaining issues:

General Notes

• If we can decide on these items within the next 24 hours I'm confident the outline can be ready in time for user testing.
• We should use GitHub for feedback from now on (essentially lock down the Coda document) in order to make the decision process more open and public. Also Coda replies are very easy to miss.
• The goal here is to solve these issues to get a version of the outline ready in time for user testing. Doesn't mean we can't make tweaks after that.

Forms & Editing

• "Have you used for autocomplete?" seems too specific to apply to most respondents

Interactivity

OK

Content

• If we ask about i18n at all we should ask about RTL/LTR even if it's via proxy through other questions/features. Maybe also CJK?

External Content

• Resolve which questions to keep/remove (assuming we keep max 3 checkbox lists per section)

Web Components

OK

PWAs

OK

Accessibility

• We need to promote 2-4 questions to "main feature" questions

Other Tools

OK

Usage

• "Which existing HTML features are you unable to use?" should be about interoperability specifically? (cc @atopal)
• "Which components do you find yourself using across multiple projects?" and "If you could add 3 elements to HTML, which would you add?" seem redundant to me as they will almost certainly have a lot of items in common.
• The list in "If you could add 3 elements to HTML, which would you add?" should imo be taken from the components provided by libraries such as Bootstrap, Shadcn/UI, Radix, etc. that do not already exist in HTML as these libraries naturally implement what their user require the most.

Demographics

OK

[LeaVerou]

Right but the current format won't tell us the "why".

The format will likely be a freeform list (since the key-value template won't be ready on time), so how will it not tell us why?!

[SachaG]

So the current question is:

• Which existing HTML features are you unable to use?
  • These are elements that you need, but even though they exist natively, you have to use libraries of custom components to satisfy your needs. For example, due to browser incompatibilities, or limited styling.

First of all it does not ask the respondent to specify a reason why they can't use the features, they are just asked to list them.

But even if the respondents did list them, the resulting data would be hard to use. If it's a freeform textfield*, you could imagine something like:

I find it really hard to make selects look like what the designer wants, and also I would love to use dialog once it's supported everywhere

How do you parse that using regexs? First, how do you know that "make selects look like what the designer wants" refers to styling, and that "supported everywhere" refers to browser compatibility? And second, even if you can parse that how do you know which reason applies to which feature?

*The freeform list only alleviates the second issue, not the first, and it's not implemented yet. As awesome human being and my personal hero Donald Rumsfeld once said (little known fact, he was a web designer before becoming a war criminal), “You run a survey with the UI you have, not the UI you might want or wish to have at a later time.”.

[LeaVerou]

First of all it does not ask the respondent to specify a reason why they can't use the features, they are just asked to list them.

In the key-value template, there is an entire field for the reason. If a freeform list, a) we'll use textareas, which hint at this, and b) we can use suitable text to hint that they can provide more details (but don't have to) and to ground the format a bit. I just took a stab at it ("Please enter one feature per box, and prefix any comments with the the corresponding opening tag (e.g. <element> or <element foo="bar"> for attributes)."), we could also include placeholder text to drive the point home.

That kind of wording seems to address your next concern as well, about the data being hard to use. If people follow this format, at least the feature name should be easy to extract and compute stats about.

If it's a freeform textfield*, you could imagine something like: [...] How do you parse that using regexs?

It is well-known that regexes are a poor way to analyze natural language (or even computer languages!), and you're explaining why very well here :) This doesn't indicate that we should not have freeform questions (we should limit them to minimize survey fatigue but that's orthogonal), but that a better analysis pipeline is needed (probably something that involves AI to some degree).
Even before this happens however, @ShaineRosewel can analyze them. There is nothing preventing you from posting that analysis on the public results (instead or in addition to the one that uses regexes).

We can collect the same data in a more structured way, but that would require UI changes. So it's a tradeoff between upfront UI work vs later analysis work. We could go either way, but my understanding was that you preferred the latter.

Note that there is also the middle ground of collecting the data through a structured UI and storing it as a serialized string with a predefined format. That makes it far easier to implement (no data model changes, no API changes etc), but analysis is almost as easy as structured data (assuming the serialization we pick is idempotent). However, when I suggested this idea, you didn't like it. I’m a bit fuzzy on the reasons; you said that it wouldn't actually make it easier to implement, yet for other UI (e.g. sentiment chips on checkbox questions) you've said that the data model and API changes are the bottleneck. 🤷🏽‍♀️

*The freeform list only alleviates the second issue, not the first, and it's not implemented yet.

@eric-burel has an experimental prototype for it 😄

As awesome human being and my personal hero Donald Rumsfeld once said (little known fact, he was a web designer before becoming a war criminal), “You run a survey with the UI you have, not the UI you might want or wish to have at a later time.”.

As Albert Einstein once said (little known fact, he was a web survey designer before becoming a physicist) "You only get one chance to collect good data, but many to analyze it"

[SachaG]

Good points, I just wanted to make sure we all understand the limitations of our own regex-based analysis capabilities, and the consequences of phrasing the question one way or another. I just wanted to be extra careful about this question since in the past it's been one that @atopal's team values a lot.

However, when I suggested this idea, you didn't like it. I’m a bit fuzzy on the reasons; you said that it wouldn't actually make it easier to implement,

My point was that if we compare the time it takes to implement the serialization + deserialization code vs the time to actually store the data the right way, it's probably comparable. I also don't want to add "weird" systems to handle edge cases like that to our codebase any more than I have to.

[LeaVerou]

My point was that if we compare the time it takes to implement the serialization + deserialization code vs the time to actually store the data the right way, it's probably comparable.

Not necessarily. E.g. you could use a standard serialization/deserialization format like JSON or YAML, then the time to implement the serialization + deserialization code is effectively 0.

I also don't want to add "weird" systems to handle edge cases like that to our codebase any more than I have to.

What if you made it a first-class citizen: A "custom" question template, that takes a blob of HTML (including form elements) as a parameter and serializes the output in an <input type="hidden"> (or parses it, if loading stored data), which is what's actually submitted. Just brainstorming, not sure if something like that would work with your infrastructure, but perhaps a variation might!

[SachaG]

Recapping the main remaining issues so people don't have to read through the whole thread:

Accessibility Section

• How to improve the A11y section? (especially which features to promote to main features status)
  • waiting on feedback from a11y experts.

Usage Section

• How exactly to phrase the "Which existing HTML features are you unable to use?" question and what it should cover
• Missing elements/components question, see Gap analysis and pain points (Which elements are missing or unusable?) #164 (comment)

New UIs Status

Working Implementation

This works inside the actual survey form.

• 5-option feature question
• Radio buttons with sentiment chips

In Progress

We have a prototype inside the survey form but it's not functional yet.

• Freeform list (stand-alone list of multiple freeform items)

Prototype Only

This only exists as a stand-alone prototype outside the survey form.

• Checkboxes with sentiment chips

[LeaVerou]

Sun, Sep 3 feedback

I went through the current outline, here are my comments. High priority items in bold.

Overall

• Can we please remove the pagination feature in multiple choice question answers? Hiding some of the options severely biases the results. I noticed it here:
  • Which of the following form elements have you used?
  • Which of the following internationalization features have you used?
• I still see the old, longer labels in the sentiment chips features, as well as the old emojis
• The labels of the sentiment chips were significant, "positive experience" is not the same as "want to use again". @stubbornella had good arguments about this. Also, we should make sure the 5-answer template uses the similar wording.
• Several answers are appearing as phrase ids, I guess that's temporary?
• Some features are appearing twice, because they were in two sections in the Coda doc. These were alternatives, we should not have features that appear in two places.

Outline

• Assign half pain points questions to use freeform lists (either every other one or the first/second half).
• Assign the first/second half of feature questions to use the 5-answer template and the other half to use the sentiment chips template. This is more or less done, just pointing it out so we can ensure it's half.

Forms

• "Which of the following form elements have you used?" has been split into two questions
• Update autocomplete="hint" metadata
• Which of the following form elements have you used?

Interactivity

• "Which of the following DOM methods have you used?" - Let's turn randomization off here, order of options is significant
• "Which of these methods do you use to add interactivity to websites?" mentioning the DOM was significant. We don't need to say DOM per se, but we do need to somehow clarify that this doesn't just refer to events etc but also DOM modifications that may not be perceived as interactive (e.g. templating, modifying data on the fly when the page loads etc).

Content

• "Which of the following have you used to include machine-readable data in HTML?" Turn off randomization, order of options is significant
• "How often do you work on projects that require supporting more than one language?" This is asking a subset of what was in the outline. It's not just about being able to switch across languages. Many tools are released in English with hardcoded strings and it would be good to be able to quote a statistic like "this causes trouble for X% of developers". Not all of these developers necessarily work with multiple languages, there are plenty of monolingual websites whose developers and designers still face challenges that developers working entirely with English (or even latin alphabet languages) never have to consider (having started my career in Greece I'm intimately familiar with this).
• Missing blast from the past question

External Content

• "What are your biggest pain points around using external content in HTML?" The description underneath was significant ("External content includes <iframe>, graphics and multimedia, math etc."). This addresses one of @chrishtr's priorities.

Web Components

• Shadow DOM is missing its description and code snippet
• "Web Component Features" -> "Other Web Components Features"
• "Making Web Components Pain Points" could you please use the original wording? ("What are your biggest pain points around making web components?")
• There are two features that should have been here but were hidden by filters, we should regenerate features.yml

Accessibility

Most of this section is not there yet, so I couldn't review.

• "Which of these accessibility tools have you used?" This has been split into two.
• "What are your biggest pain points around accessibility?" missing "web". I guess for most it's obvious, but maybe not all

Progressive Web Apps

• URLs to Text Fragments should not be in here, I just moved it.
• Manifest fields: extra paren
• "What are your biggest pain points around progressive web apps?" Please use the original wording ("What are your biggest pain points around making web apps that feel native?")

Usage

• Missing HTML missing features and pain points questions This is critical — these are some of the most important questions in the survey.

High priority UI pending

• Sentiment chips for checkboxes: in progress
• Freeform lists: iron out remaining issues
• Multiple custom options: not started

[SachaG]

The labels of the sentiment chips were significant, "positive experience" is not the same as "want to use again". @stubbornella had good arguments about this. Also, we should make sure the 5-answer template uses the similar wording.

What about the scenario where someone really needs a feature but struggles with it? They would pick "want to use again" even though they had a negative experience. At the end of the day, do we want to measure how useful a feature is; or how good its developer experience is? We should pick one or the other depending on that.

Update autocomplete="hint" metadata

I updated the features YAML, so that should be ok?

"Which of the following form elements have you used?" has been split into two questions

Do we really need to ask about input/select/textarea/etc.?

"How often do you work on projects that require supporting more than one language?" This is asking a subset of what was in the outline.

If the goal is to identify which respondents is working with languages that require extra features that English doesn't need, then we should find a more direct way to ask that. I don't think the previous question phrasing quite achieved that either.

"What are your biggest pain points around using external content in HTML?" The description underneath was significant ("External content includes <iframe>, graphics and multimedia, math etc."). This addresses one of @chrishtr's priorities.

I don't understand the "math" part.

"Making Web Components Pain Points" could you please use the original wording? ("What are your biggest pain points around making web components?")

Whoops I used the wrong i18n key here.

There are two features that should have been here but were hidden by filters, we should regenerate features.yml

I saw that template_element was missing, what was the other one? Btw just FYI regenerating features.yml won't add an element to the survey, it has to be added to questions.yml in the surveys repo.

"What are your biggest pain points around progressive web apps?" Please use the original wording ("What are your biggest pain points around making web apps that feel native?")

I just thought it was inconsistent with the other sections where we don't constrain/orient the freeform answer. What's the motivation for not following the "Section Foo -> What are your pain points with Foo" model?

[LeaVerou]

What about the scenario where someone really needs a feature but struggles with it? They would pick "want to use again" even though they had a negative experience.

That's why I switched from "Would use again" to "Want to use again"

Do we really need to ask about input/select/textarea/etc.?

For sentiment

I don't understand the "math" part.

It refers to <math>, but that's not significant. What was, was to mention <iframe> explicitly in there.

I saw that template_element was missing, what was the other one?

DOM Parts

I just thought it was inconsistent with the other sections where we don't constrain/orient the freeform answer. What's the motivation for not following the "Section Foo -> What are your pain points with Foo" model?

It's not about completeness or following a pattern, it's about getting useful data. Feel free to call that section "Bridging the gap with native" if it makes you feel better about using that wording.

[LeaVerou]

Second round of feedback

Legend: ‼️ High priority 🟢 Low priority

• ‼️ Checkbox sentiment chips appear to be hidden with display: none instead of opacity, making the layout jump around. Fixing this is top priority, otherwise we cannot gauge the UX (we know that UI elements jumping around is annoying for users) [ Edit: I did it ]
• ‼️ We need to harmonize copy across the two feature templates. Details here: Capturing sentiment alongside awareness/usage #183 (comment)
• "Which of these accessibility techniques have you used?"
  • ‼️ color-contrast() is not a thing, and also this refers to the technique, not how it's achieved. This may be a linking error, as I do see the correct phrase in the locale.
  • I still see pagination [ Edit: I fixed it ]
  • Includes things that moved to features (ARIA, tabindex) [ Edit: I fixed it ]
  • Visible focus ring is too specific, the guideline is that focus should be visible, and that it should be clear that it represents focus
  • We could probably skip the sentiment chips here? [ Edit: I did it ]
• "Which of these accessibility tools have you used?" Accessibility tools doesn't make sense, they're accessibility testing tools
• Pain points freeform list should use the longText option [ Edit: I fixed it ]
• ‼️ Please add sentiment chips to all the checkboxes that are supposed to have them. We're not A/B testing that one, we're just evaluating its usability (since the alternative is expanding them into full feature questions, which is hard to compare and contrast unless we were doing a between-subjects study). [ Edit: I did it ]
• The "Other Tools" section includes a question about machine readable data. These are mini-features, not tooling. The proposal was to move them to an Other section if we removed the Content section (or rename External content to Content). If we want to keep them there, we should rename "Other Tools" to "Other" [ Edit: Moved them to External Content, renamed it to Content ]
• 🟢 Blast from the past question still missing.
• ‼️ "Which existing HTML features are you unable to use?" should use a freeform list, this is not part of the A/B testing, we use the pain points question for that which is more uniform [ Edit: I did it ]
• ‼️ "If you could add 5 elements to HTML, which would you add?"
  • Drop pagination
  • Given that we don't have multiple custom answers, this needs to change to a freeform list and the existing elements would be mentioned as suggestions in the description. We don't actually want this to be a multiple choice question, these were there just to prime! [ Edit: I did it ]
• ‼️ Question about which components do you find yourself reusing is missing. This was very important to include! [ Edit: I added some text in the missing elements questions to guide them that this may serve as good inspiration, I think that's enough ]
• 🟢 "Are the libraries, packages, components, etc. that you create used by other developers?" Can "None" be "No" there? Or, better, we can rephrase it as "How many other developers use ..." where "None" makes perfect sense, and the answers can be shorter (<100, Hundreds, Thousands, Millions)
• I don't see a multipleWithSentiment template, we should be able to have questions with sentiment that do not allow freeform answers, these are separate things
• ‼️ Question about proportion of HTML/CSS and JS work missing

Things still displaying with raw ids:

• features.accessibility_screenreaders.question

[SachaG]

🟢 Blast from the past question still missing.

Right, I left it out when we were afraid we'd have too many questions but I think it's fine to add it back now.

"Are the libraries, packages, components, etc. that you create used by other developers?" Can "None" be "No" there? Or, better, we can rephrase it as "How many other developers use ..." where "None" makes perfect sense, and the answers can be shorter (<100, Hundreds, Thousands, Millions)

OK

I don't see a multipleWithSentiment template, we should be able to have questions with sentiment that do not allow freeform answers, these are separate things

We can set allowOther: false/true on any question in the survey outline to override the template. Where do we need to disallow freeform answers?

‼️ Question about proportion of HTML/CSS and JS work missing

Did we discuss that question? I don't think it's in the outline? Also do we want it to be HTML vs CSS+JS, or HTML + CSS vs JS?

[SachaG]

Given that we don't have multiple custom answers, this needs to change to a freeform list and the existing elements would be mentioned as suggestions in the description. We don't actually want this to be a multiple choice question, these were there just to prime!

I thought the main argument against multiple choice questions was precisely the risk of priming? Given that multiple choice questions are easier for respondents and get higher response rates, why not use them here?

I get that you don't like having a single "other…" textfield for multiple potential custom answers, but that seems like an acceptable downside when weighed against the downsides of totally freeform questions in general? (namely: lower participation, less accurate data, more time required on the back-end to normalize and analyze results).

[LeaVerou]

It's a double edged sword, that's why in the original outline I had both an unprimed question (which came first), and a primed one. As @stubbornella explained, if you don't prime people in this type of question, they tend to think quite narrowly, about specific types of elements. Once more, the point of the options here was priming, this was not really a multiple choice question, it just presented as one 😄 This is why it was necessary for it to have multiple custom options.

Anyhow, I think I'm reasonably happy with the current template, considering. If either of us finds time, we could make the suggestions clickable, and have them insert an item (we could make this an option of the TextList template, to have suggestions that are clickable — like a reverse multiple choice).

[foolip]

Is there still time for feedback, or did I miss the party?

[LeaVerou]

There's time for feedback, but very little time for addressing it. Also most of the planning has moved to a Coda doc. Are you on Discord? Feel free to add me and I can fill you in.

[foolip]

I was added to a Discord channel a long time ago but was never signed in. I've jumped in there now and commented on the only real bug I found.

[foolip]

Things I found in a quick skim:

• features.external_content_pain_points.question isn't "translated" into English words
• "Have you ever used any of the following old-school elements?" was fun 👍
• "Which existing HTML features or browser APIs are you unable to use because of lack of support or differences between browsers?" is great, I'll look at the results

Do we have an estimate of how much time the full survey will take? Is it longer or shorter than recent State of JS and State of CSS surveys?

[LeaVerou]

Do we have an estimate of how much time the full survey will take? Is it longer or shorter than recent State of JS and State of CSS surveys?

We do not. We ran a user study this year, but we don't have similar data from other years.
I just made a tool to roughly compare effort across different surveys based on the question shape: https://projects.verou.me/stateof/analyzer/
Weights probably need tweaking but so far it seems to actually be less effort than the others, which is contrary to what I expected.