Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Test to Scan All Pages for Axe Issues #1361

Merged
merged 27 commits into from
Oct 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion UITests/AxeHelper.cs
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ public static void AssertNoAccessibilityErrors()
.Where(rule => rule.Rule.ID != RuleId.SiblingUniqueAndFocusable);
if (testResult.Any())
{
var mappedResult = testResult.Select(result => "Element " + result.Element.Properties["ControlType"] + " violated rule '" + result.Rule.Description + "'.");
var mappedResult = testResult.Select(result =>
"Element " + result.Element.Properties["ControlType"] + " violated rule '" + result.Rule.Description + "'.");
Assert.Fail("Failed with the following accessibility errors \r\n" + string.Join("\r\n", mappedResult));
}
}
Expand Down
116 changes: 116 additions & 0 deletions UITests/Tests/AxeScanAllTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.VisualStudio.TestPlatform;
using OpenQA.Selenium.Appium.Windows;
using System;
using System.Linq;
using System.Text.Json;
using System.IO;
using System.Collections.ObjectModel;
using System.Collections;
using System.Collections.Generic;
using System.Xml;
using System.Reflection;
using Newtonsoft.Json;

namespace UITests.Tests
{
[TestClass]
public class AxeScanAll : TestBase
{
public static readonly string jsonUri = "ControlInfoData.json";
public static new WindowsDriver<WindowsElement> Session => SessionManager.Session;

public static string[] ExclusionList =
{
"WebView2", // 46668961: Web contents from WebView2 are throwing null BoundingRectangle errors.
"Icons" // https://github.com/CommunityToolkit/Windows/issues/240 External toolkit SettingsExpander does not pass Axe testing
};

public class ControlInfoData
{
public List<Group> Groups { get; set; }
}

public class Group
{
[JsonProperty("UniqueId")]
public string UniqueId { get; set; }

[JsonProperty("Items")]
public List<Item> Items { get; set; }
}

public class Item
{
[JsonProperty("UniqueId")]
public string UniqueId { get; set; }
}

private static IEnumerable<object[]> TestData()
{
var testCases = new List<object[]>();
Copy link
Member

@kmahone kmahone Oct 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you are doing your own parsing of the data, would it be simpler to directly parse the ControlInfoData.json file instead of having a separate script to convert ControlInfoData.json into WinUIGalleryTestData.xml and then parsing that xml file?

For the Lifted Xaml repo, I needed to convert from json into an xml format that taef understood. But since you re doing your own parsing here, you don't have to use xml at all. #Closed

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it's easier to parse the .xml file instead of the .json. The json file has quite a lot of info that will need to be parsed, but is not relevant to the tests. This method adds an extra file, but results in cleaner test code.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about using the same parsing code from the WinUI 3 Gallery? Since we are using the same .NET SDK, we could move them in a shared project so both projects can reference the same data and parsing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are already parsing the json file. You are just doing it in powershell as part of the build. You can move that same logic out of powershell and into the test code.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kmahone Done.
@chingucoding That can definitely be a future refactor, but too big of a change for this PR in my opinion.


string jsonContent = System.IO.File.ReadAllText(jsonUri);
var controlInfoData = JsonConvert.DeserializeObject<ControlInfoData>(jsonContent);

foreach (var group in controlInfoData.Groups)
{
var sectionName = group.UniqueId;

// Select all row names within the current table
var items = group.Items;

foreach (var item in items)
{
var pageName = item.UniqueId;

// Skip pages in the exclusion list.
if (ExclusionList.Contains(pageName))
{
continue;
}
testCases.Add(new object[] { sectionName, pageName });
}
}

return testCases;
}

[ClassInitialize]
public static void ClassInitialize(TestContext context)
{
}

[TestMethod]
[DynamicData(nameof(TestData), DynamicDataSourceType.Method, DynamicDataDisplayName = nameof(GetCustomDynamicDataDisplayName))]
[TestProperty("Description", "Scan pages in the WinUIGallery for accessibility issues.")]
public void ValidatePageAccessibilityWithAxe(string sectionName, string pageName)
{
try
{
// Click into page and check for accessibility issues.
var page = Session.FindElementByAccessibilityId(pageName);
page.Click();

AxeHelper.AssertNoAccessibilityErrors();
}
catch
{
// If element is not found, expand tree view as it is nested.
var section = Session.FindElementByAccessibilityId(sectionName);
section.Click();

// Click into page and check for accessibility issues.
var page = Session.FindElementByAccessibilityId(pageName);
page.Click();

AxeHelper.AssertNoAccessibilityErrors();
}
}

public static string GetCustomDynamicDataDisplayName(MethodInfo methodInfo, object[] data)
{
return string.Format("Validate{0}PageAccessibility", data[1]);
}
}
}
6 changes: 0 additions & 6 deletions UITests/Tests/Button.cs
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,6 @@ public static void ClassInitialize(TestContext context)
Assert.IsNotNull(buttonElement);
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void Button_Click()
{
Expand Down
6 changes: 0 additions & 6 deletions UITests/Tests/CheckBox.cs
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,6 @@ public static void ClassInitialize(TestContext context)
Assert.IsNotNull(checkBoxElement2);
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void Click()
{
Expand Down
8 changes: 0 additions & 8 deletions UITests/Tests/ComboBox.cs
Original file line number Diff line number Diff line change
Expand Up @@ -36,17 +36,9 @@ public static void ClassInitialize(TestContext context)
Assert.IsNotNull(comboBoxElement2);
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void Click()
{


// Click comboBoxElement1 to show the list and simply dismiss it
var originalSelectedItem = comboBoxElement1.Text;
comboBoxElement1.Click();
Expand Down
6 changes: 0 additions & 6 deletions UITests/Tests/DatePicker.cs
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,6 @@ public static void ClassInitialize(TestContext context)
Assert.IsNotNull(datePickerElement2);
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void Click()
{
Expand Down
6 changes: 0 additions & 6 deletions UITests/Tests/MediaPlayerElement.cs
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,6 @@ public static void ClassInitialize(TestContext context)
GetElementByName("MediaPlayerElement").Click();
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void PlayMedia()
{
Expand Down
6 changes: 0 additions & 6 deletions UITests/Tests/PersonPicture.cs
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,6 @@ public static void ClassInitialize(TestContext context)
OpenControlPage("PersonPicture");
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void SwitchOptions()
{
Expand Down
6 changes: 0 additions & 6 deletions UITests/Tests/ProgressBar.cs
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,6 @@ public static void ClassInitialize(TestContext context)
Assert.IsNotNull(clickAndHoldButton);
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void Displayed()
{
Expand Down
6 changes: 0 additions & 6 deletions UITests/Tests/RadioButton.cs
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,6 @@ public static void ClassInitialize(TestContext context)
Assert.IsNotNull(radioButtonElement2);
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void Click()
{
Expand Down
6 changes: 0 additions & 6 deletions UITests/Tests/Slider.cs
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,6 @@ public static void ClassInitialize(TestContext context)
Assert.IsNotNull(sliderElement2);
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void Click()
{
Expand Down
6 changes: 0 additions & 6 deletions UITests/Tests/TextBlock.cs
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,6 @@ public static void ClassInitialize(TestContext context)
Assert.IsNotNull(textBlockElement2);
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void Displayed()
{
Expand Down
6 changes: 0 additions & 6 deletions UITests/Tests/TextBox.cs
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,6 @@ public static void ClassInitialize(TestContext context)
Assert.IsNotNull(textBoxElement2);
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void Clear()
{
Expand Down
6 changes: 0 additions & 6 deletions UITests/Tests/ToggleButton.cs
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,6 @@ public static void ClassInitialize(TestContext context)
Assert.IsNotNull(toggleButtonElement);
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void Click()
{
Expand Down
6 changes: 0 additions & 6 deletions UITests/Tests/ToggleSwitch.cs
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,6 @@ public static void ClassInitialize(TestContext context)
Assert.IsNotNull(toggleSwitchElement);
}

[TestMethod]
public void ValidateAccessibilityWithAxe()
{
AxeHelper.AssertNoAccessibilityErrors();
}

[TestMethod]
public void Click()
{
Expand Down
11 changes: 11 additions & 0 deletions UITests/UITests.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,22 @@
<RuntimeIdentifiers>win10-x86;win10-x64;win10-arm64</RuntimeIdentifiers>
</PropertyGroup>

<!-- Build output related paths -->
<PropertyGroup>
<ProjectRoot>$(MSBuildThisFileDirectory)..\</ProjectRoot>
<ArtifactsBinRoot>$(MSBuildThisFileDirectory)\bin\</ArtifactsBinRoot>
<TestBinplaceDestinationPath>\$(Platform)\$(Configuration)\$(TargetFramework)</TestBinplaceDestinationPath>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Appium.WebDriver" Version="4.4.0" />
<PackageReference Include="Axe.Windows" Version="2.1.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.5.0" />
<PackageReference Include="MSTest.TestAdapter" Version="3.0.2" />
<PackageReference Include="MSTest.TestFramework" Version="3.0.2" />
</ItemGroup>

<Target Name="CopyControlInfoData" AfterTargets="AfterBuild">
<Copy SourceFiles="$(ProjectRoot)\WinUIGallery\DataModel\ControlInfoData.json" DestinationFiles="$(ArtifactsBinRoot)\$(TestBinplaceDestinationPath)\ControlInfoData.json" />
</Target>
</Project>
1 change: 1 addition & 0 deletions WinUIGallery/ControlPages/DesignGuidance/IconsPage.xaml
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@
<RowDefinition Height="*" />
</Grid.RowDefinitions>
<TextBlock Text="With the release of Windows 11, Segoe Fluent Icons is the recommended icon font." />

<toolkit:SettingsExpander
Grid.Row="1"
Header="Instructions on how to use Segoe Fluent Icons"
Expand Down
2 changes: 1 addition & 1 deletion WinUIGallery/ControlPages/InfoBadgePage.xaml
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@

<local:ControlExample x:Name="Example3" HeaderText="Placing an InfoBadge Inside Another Control" HorizontalContentAlignment="Stretch" >
<local:ControlExample.Example>
<Button Padding="0" Width="200" Height="60" ToolTipService.ToolTip="Refresh required"
<Button x:Name="Example3Button" AutomationProperties.Name="Example3Button" Padding="0" Width="200" Height="60" ToolTipService.ToolTip="Refresh required"
HorizontalAlignment="Center" HorizontalContentAlignment="Stretch" VerticalContentAlignment="Stretch">
<Grid HorizontalAlignment="Stretch" VerticalAlignment="Stretch" Width="Auto" Height="Auto">
<SymbolIcon Symbol="Sync" HorizontalAlignment="Center"/>
Expand Down
6 changes: 3 additions & 3 deletions WinUIGallery/ControlPages/RichTextBlockPage.xaml
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,11 @@
TextAlignment="Justify" Margin="12,0">
<Paragraph>Linked text containers allow text which does not fit in one element to overflow into a different element on the page. Creative use of linked text containers enables basic multicolumn support and other advanced page layouts.</Paragraph>
<Paragraph>Duis sed nulla metus, id hendrerit velit. Curabitur dolor purus, bibendum eu cursus lacinia, interdum vel augue. Aenean euismod eros et sapien vehicula dictum. Duis ullamcorper, turpis nec feugiat tincidunt, dui erat luctus risus, aliquam accumsan lacus est vel quam. Nunc lacus massa, varius eget accumsan id, congue sed orci. Duis dignissim hendrerit egestas. Proin ut turpis magna, sit amet porta erat. Nunc semper metus nec magna imperdiet nec vestibulum dui fringilla. Sed sed ante libero, nec porttitor mi. Ut luctus, neque vitae placerat egestas, urna leo auctor magna, sit amet ultricies ipsum felis quis sapien. Proin eleifend varius dui, at vestibulum nunc consectetur nec. Mauris nulla elit, ultrices a sodales non, aliquam ac est. Quisque sit amet risus nulla. Quisque vestibulum posuere velit, vitae vestibulum eros scelerisque sit amet. In in risus est, at laoreet dolor. Nullam aliquet pellentesque convallis. Ut vel tincidunt nulla. Mauris auctor tincidunt auctor.
Aenean orci ante, vulputate ac sagittis sit amet, consequat at mi. Morbi elementum purus consectetur nisi adipiscing vitae blandit sapien placerat. Aliquam adipiscing tortor non sem lobortis consectetur mattis felis rhoncus. Nunc eu nunc rhoncus arcu sollicitudin ultrices. In vulputate eros in mauris aliquam id dignissim nisl laoreet.</Paragraph>
Aenean orci ante, vulputate ac sagittis sit amet, consequat at mi. Morbi elementum purus consectetur nisi adipiscing vitae blandit sapien placerat. Aliquam adipiscing tortor non sem lobortis consectetur mattis felis rhoncus. Nunc eu nunc rhoncus arcu sollicitudin ultrices. In vulputate eros in mauris aliquam id dignissim nisl laoreet.</Paragraph>
</RichTextBlock>
<RichTextBlockOverflow x:Name="firstOverflowContainer" Grid.Column="1"
<RichTextBlockOverflow x:Name="firstOverflowContainer" AutomationProperties.Name="firstOverflowContainer" Grid.Column="1"
OverflowContentTarget="{x:Bind secondOverflowContainer}" Margin="12,0" />
<RichTextBlockOverflow x:Name="secondOverflowContainer" Grid.Column="2" Margin="12,0" />
<RichTextBlockOverflow x:Name="secondOverflowContainer" AutomationProperties.Name="secondOverflowContainer" Grid.Column="2" Margin="12,0" />
</Grid>
</local:ControlExample.Example>
<local:ControlExample.Xaml>
Expand Down
5 changes: 5 additions & 0 deletions WinUIGallery/Controls/SampleCodePresenter.xaml.cs
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
using AppUIBasics.Common;
using System.Reflection;
using System.IO;
using Microsoft.UI.Xaml.Automation;

namespace AppUIBasics.Controls
{
Expand Down Expand Up @@ -226,6 +227,10 @@ private void FormatAndRenderSampleFromString(string sampleString, ContentPresent

actualCode = sampleString;

var name = GetSampleLanguageVisualState() == "InlineSample" ? actualCode : SampleHeader.Text;
var automationName = "Copy " + name + " Code";
AutomationProperties.SetName(CopyCodeButton, automationName);


var formatter = GenerateRichTextFormatter();
if (SampleType == SampleCodePresenterType.Inline)
Expand Down
Loading