docs(spec): north star spec reporting

[mattrosno]

Surfacing this type of information isn't dependent on a monorepo, but that would make the reporting a lot easier (wouldn't have to have CI PR into the reporting repo). Concepts from monorepo re-architecting that are included: "add-ons" are now called "community components", every component is independently published, but there are also published packages for each framework as a whole.

Demo

This PR demonstrates what a dynamic component status page could look like, with one example of a detailed component view (button). All content is fictional. This is meant to facilitate discussion around a reporting north star.

https://github.com/carbon-design-system/carbon-spec/blob/ffea894324da8f3a993319e98a4bd652e8145a15/docs/component-status.md

https://github.com/carbon-design-system/carbon-spec/blob/ffea894324da8f3a993319e98a4bd652e8145a15/docs/components/button.md

Assumptions

This assumes that each component has a specification, that includes:

1. Written requirements that describe behavior for every user type (general, keyboard-only, screen reader, etc.)
2. Tests that can be ran on each implementation verifying expected behavior, DOM structure, element types, attributes (pertaining to accessibility), and content

There's also be separate documentation on "what it takes to be a component" which includes how to author your component's spec and maintain it over time.

Scenarios

Questions that we want a "component status" page to answer:

1. What framework packages and component packages are available?
2. Is this component available in framework X?
3. Is this component fully available in framework X (including all the variants, states, modifiers that I need)?
4. How accessible is each component?
5. Who maintains each framework and/or component?
6. Which framework and/or component is used the most?
7. Which components are new this release?
8. Which components are deprecated this release?
9. Which components are not stable (a.k.a. experimental)?
10. Where are there contribution opportunities to either improve existing components, create missing components, or make something more accessible through the spec?

Implementation considerations

downloads and npm badge content is dynamic - those SVGs coming directly from NPM. It'd be great to include dynamic badges for commit activity | 457/month or last commit | today or release date | april 2018, but those badges hit the GitHub API which is per repository. With one monorepo, that's not possible per component package. We could possibly generate that during our build by looking at commit history and file modified dates per component directory... but may not be worth the trouble.

Outstanding questions

1. More related to monorepo in general, but how do we handle breaking changes with community components when something higher-order (utility functions, helper Sass) causes the breaking changes?
2. Related to that, do we have a third group of components called "sunset components" or something once the next major release and previously deprecated items are no longer supported? Or do we purge those components and somebody would have to revert to a previous release in the repo to find that sunset source?
3. How do we show exactly what failed or is throwing warnings in the spec test reporting? Or do we keep it simple and expose such information when locally running the spec tests?
4. How can we provide contribution short cuts to improve spec adherence? Or like above, you locally run the spec tests to identify specifics?
5. Is there a better way to sort the core and community components besides alphabetically? We don't have NPM usage stats at build when the markdown page is generated, so alphabetical may be our only option.
6. For each component, for each framework, how do we link to a demo environment like Storybook? Would we be able to deep link to a specific variant/state/modifier or just to the default component?

[scottnath]

@mattrosno this is very exciting. Put some notes next to a couple spots in the code. Primary question is really around data storage. There's a number of data pieces, some of which are framework-specific, but we tend to keep this kind of data in [component].config.js, but that file is not needed by other frameworks really. So...would this maybe be a carbon-specific file each framework would need?

[scottnath]

This separate section for wcag specs is very exciting.

• Would there be separate charts re: wcag 2.0 vs 2.1?
• General rule of thumb is automated tests can only cover ~30% of a11y tests so this can't be automatically updated after CI runs tests. It would be best if there was some sort of object of data associated with each component that these results could be culled from.

[mattrosno]

@scottnath, @dakahn and I are working through some of those details, but here's where we're at so far. Each component's spec could include 1 to many accessibility classifications. Each classification (e.g. WCAG 2.0 AA) has a subset of spec tests needed to hit that level of accessibility, plus a time-stamped summary of manual test results performed on a component that passes 100% of that subset of tests. That way, should we need to add test cases to that accessibility classification, or change anything, we know to re-run manual tests.

In this scenario, maybe the component's Specification column has two available accessibility badges (WCAG 2.0 A and WCAG 2.0 AA), and then each implementation has at most one accessibility badge (e.g. React component is passing all of the tests needed to be AA so it's AA, Angular component passes all the A tests but has not implemented the changes to pass the AA tests, so it's A).

I'll share an example which will be easier to understand.

[scottnath]

spec numbering

confused on the numbers on these. What does a 1 ( ) mean here vs a 9 ( ) above?

data and inheritance

Thinking about where all this data will come from. For our components, we track component-level and demo-level data and store that data in a [component].config.js file (just like what's in each component in carbon-components). This structure allows the testing-data to be inherited by demos from their parent component.

Example (from our core) image component of how we track browser support at the component level:

const support = {
  browsers: {
    chrome: 'true',
    firefox: 'true',
    safari: 'true',
    edge: 'true',
    ie11: 'true',
  },
...

but then those can be overridden at the demo level:

const picture = {
  name: 'picture',
  label: 'Picture',
  support: {
    browsers: {
      ie11: {
        content:
          'The `picture` scenario does not meet all PAL browser requirements.' +
          ' This method can only be used for applications that do not need to support IE11.',
        url:
          'https://developer.mozilla.org/en-US/docs/Web/HTML/Element/picture#Browser_compatibility',
        support: 'mixed',
      },
    },
  },
};

demo wcag rating

How does the spec rating work vs the wcag rating? ie...if Accordion is rated at wcag-2.0-aa, then by passing all spec tests would a component and/or demo be considered aa? What if a given demo or component had a better rating of aaa?

I'm wondering if the wcag rating could be tracked as a separate data point at the data & component & framework level:

Button:
  Spec: WCAG-aa
  Vanilla: spec-passing (WCAG-aa)
  Vue: primary button @ WCAG-aaa, 
      component @ WCAG-aa means:
      vue-button/primary=wcag-aaa, but vue-button=wcag-aa, but _both_ passing spec

	Type	Vanilla	React	Angular	Vue
General	-
Primary	variant

[mattrosno]

I'm working on component configs (living in package.json files once each component is individually published) and revisiting component directory structure now that we have a better understanding of where we want to go compared to a couple months ago. Let's go over that next sync-up! Still a WIP over here.

[scottnath]

Also confused on this one's meaning:

[mattrosno]

I'll document what those spec test result badges mean. In that scenario, 2 tests pass and there is 1 warning.

What would constitute a warning? Maybe a test that addresses something that takes the component from WCAG 2.0 AA to AAA. The component isn't failing - but it could be better.

[scottnath]

What if a framework doesn't want to deprecate? Would the spec stay forever?

[mattrosno]

I think we could come up deprecation logic at each major release similar to what @joshblack outlined here: carbon-design-system/rfcs#5 (comment)

We'll have to clearly document community component governance so any directory of community components doesn't become a graveyard of unsupported components.

[mattrosno]

Thanks for the feedback @scottnath! I know spec reporting isn't in scope of the original spec proof of concept, but it's refreshing to see what this could be once aggregated, as a means to work backwards to where we started with the test runner.

[mattrosno]

@scottnath, @dakahn and I are working through some of those details, but here's where we're at so far. Each component's spec could include 1 to many accessibility classifications. Each classification (e.g. WCAG 2.0 AA) has a subset of spec tests needed to hit that level of accessibility, plus a time-stamped summary of manual test results performed on a component that passes 100% of that subset of tests. That way, should we need to add test cases to that accessibility classification, or change anything, we know to re-run manual tests.

In this scenario, maybe the component's Specification column has two available accessibility badges (WCAG 2.0 A and WCAG 2.0 AA), and then each implementation has at most one accessibility badge (e.g. React component is passing all of the tests needed to be AA so it's AA, Angular component passes all the A tests but has not implemented the changes to pass the AA tests, so it's A).

I'll share an example which will be easier to understand.

[mattrosno]

I think we could come up deprecation logic at each major release similar to what @joshblack outlined here: carbon-design-system/rfcs#5 (comment)

We'll have to clearly document community component governance so any directory of community components doesn't become a graveyard of unsupported components.

[mattrosno]

I'll document what those spec test result badges mean. In that scenario, 2 tests pass and there is 1 warning.

What would constitute a warning? Maybe a test that addresses something that takes the component from WCAG 2.0 AA to AAA. The component isn't failing - but it could be better.

[mattrosno]

I'm working on component configs (living in package.json files once each component is individually published) and revisiting component directory structure now that we have a better understanding of where we want to go compared to a couple months ago. Let's go over that next sync-up! Still a WIP over here.

[cal-smith]

This looks really great 👍 I'll echo that it's not super clear what "spec 1"/"spec 9" refers to in the component sample (maybe "n passed"/"n warning"/"n errored" instead?), but the overall layout looks solid.

5. Is there a better way to sort the core and community components besides alphabetically? We don't have NPM usage stats at build when the markdown page is generated, so alphabetical may be our only option.

Alphabetical is definitely easiest for finding components in the absence of filtering. If the status is consumed by some other tool sorting and filtering would be awesome additions.