Concurrent aXe calls result in "Expect axe._selectorData to be set up" exceptions

[drewlee]

A somewhat substantial backwards compatibility issue introduced in v3, is the fact that making concurrent aXe.run calls results in JavaScript exceptions (see demo below). Seeing as this was likely an intentional architectural decision, it's still surprising that this wasn't called out anywhere. It would be beneficial to document, and explicitly called out as a caveat, or aspect to avoid (i.e., "you shouldn't be making concurrent calls").

• Link to live demo:
  https://codepen.io/drewlee/full/LBdQOO
  https://codepen.io/drewlee/pen/LBdQOO

The above example is rather contrived, but is based off a valid use case, where we have several components carry out self-auditing (scoped to themselves) upon re-rendering. The primary issue is that the components are completely isolated from one another, and there is not direct control on when they finish rendering to kick off axe.run calls, and no specific order is guaranteed. I've noticed that the same problem has been noted in other SPA libraries that have integrated axe for similar purposes.

I'm wondering if there has been any consideration in implementing a concurrency scheduling system (see examples below) that's internal to axe, that would mitigate the issue entirely, and would save other folks from having to roll their own in their applications.

• Mitigating concurrency demos:
  https://codepen.io/drewlee/full/xJWQVj
  https://codepen.io/drewlee/pen/xJWQVj
  https://codepen.io/drewlee/full/MBVzpo
  https://codepen.io/drewlee/pen/MBVzpo

Thanks, and appreciate your consideration.

axe-core version: 3.0.3

[WilcoFiers]

Hey Andrew. Thanks for reporting. That's a very interesting problem. I don't think I've ever seen axe being run in parallel. I'm not sure I fully understand your use case for it either. Is it something like that you've got multiple react apps on a page and at the root after a componentDidMount you run axe? Can you point me to some other places axe is being used this way? I'd like to understand the use case a little better. It certainly seems legit.

Thinking through the problem a bit, I suspect that a change I proposed a while ago could probably solve this issue #787. Generally axe's reliance on globals should be changed IMO. That would stop different runs from interfering with each other too. In the mean time we can certainly update the documentation.

[drewlee]

Hi Wilco, the specific use case is within the context of an Ember.js addon. The addon augments all components to execute an axe audit when a component re-render occurs (scoped to itself). If a violation is found, an artifact is applied to the component in the UI, so that it is easily distinguishable as non-compliant. The specific features can be read about toward the bottom of the addon's README here: https://github.com/ember-a11y/ember-a11y-testing/blob/master/README.md. The specific axe.run call can be viewed here (line 141): https://github.com/drewlee/ember-a11y-testing/blob/axe-bump/app/instance-initializers/axe-component.js. As you can imagine, if there are 10 components on a page, each one will spin off a concurrent axe call (although once the page has fully rendered, the likelihood of multiple calls diminishes).

I understand that this is somewhat of an unintended use of axe, as parallel executions would likely never occur within a test-running context, hence I'm not making a huge deal of it. Just wanted to make folks aware of the fact that the issue does exist. Thanks, and I appreciate the prompt response!

[WilcoFiers]

Thanks for reporting. It may take us a while to fix it. But knowing axe is used this way will certainly help going forward.

[drewlee]

Thanks Wilco.

[dylanb]

@drewlee have you thought of de-bouncing the updates? When I implemented the react-axe module, I found that multiple competing updates led to a lot of fragments being analyzed that were out of context and led to a bunch of noise, debouncing and de-duping the results solved all these problems for me.

[drewlee]

Thanks for the suggestion @dylanb. Our use case is a little different. Each call is intentional and corresponds to a unique fragment. Debouncing would unintentionally drop expected results. De-duping is automatically handled at the framework level. I've come up with an approach that schedules aXe calls serially, so that the next call doesn't occur until it's certain the previous call has completed. Thanks again!