feat: Add findByImage support to Windows driver #67

mykola-mokhnach · 2020-03-16T14:27:41Z

jsa34 · 2020-03-16T14:30:59Z

Wow - such a quick response! Thank you! This is a big blocker for me at the moment with what I am trying to do, so much appreciated!

mykola-mokhnach · 2020-03-16T14:32:15Z

@jsa34 Could you please check out this code and test it locally? With your help it would go faster

jsa34 · 2020-03-16T14:34:15Z

Of course! Let me see...

jsa34 · 2020-03-16T15:00:17Z

Trying by npm is being difficult, won't let me install git repo package, just stuck at rollbackFailedOptional...

mykola-mokhnach · 2020-03-16T15:21:19Z

You can simply install appium beta and then replace or link (see npm link) the original windows driver with the checked out content. Also make sure the package is transpiled before injecting it (running npm install should do the magic)

jsa34 · 2020-03-16T16:42:55Z

Probably just me as I am much more of a Java kind of guy, but using windows here and npm link just keeps giving me "Access is denied" response, but nothing seems weird with permissions that I can see...

mykola-mokhnach · 2020-03-16T16:47:03Z

appium/appium#13227 (comment) contains more detailed tutorial. Simply replace occurrences of webdriveragent-repo with the windows driver's one

KazuCocoa

🎉
The implementation looks good to me.
(Haven't tested on WinAppDriver though)

jsa34 · 2020-03-16T19:48:14Z

Still seem to get the same response, but think I've followed the above guide correctly:

Code:

        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("deviceName", "WindowsPC");
        capabilities.setCapability("app", "Root");
        capabilities.setCapability("ms:experimental-webdriver", true);
WindowsDriver<WindowsElement> driver = new WindowsDriver<>(getRemoteAddress(), capabilities);
               AppiumFluentWait appiumFluentWait = new AppiumFluentWait(driver);
        appiumFluentWait.withTimeout(Duration.ofSeconds(20))
                .pollingEvery(Duration.ofSeconds(2))
                .ignoring(NoSuchElementException.class);

appiumFluentWait.until(ExpectedConditions.presenceOfElementLocated(MobileBy.image(ImageManager.getImage(imageName))));

Error in IDE:
Caused by: org.openqa.selenium.InvalidSelectorException: Invalid selector: -image (WARNING: The server did not provide any stacktrace information)

Appium:

[HTTP] --> POST /wd/hub/session/8fccee01-f83f-4bdb-8d81-0a78453fbea7/element
[HTTP] {"using":"-image","value":"***"}
[MJSONWP (8fccee01)] Driver proxy active, passing request on via HTTP proxy
[debug] [WD Proxy] Matched '/wd/hub/session/8fccee01-f83f-4bdb-8d81-0a78453fbea7/element' to command name 'findElement'
[debug] [WD Proxy] Proxying [POST /wd/hub/session/8fccee01-f83f-4bdb-8d81-0a78453fbea7/element] to [POST http://127.0.0.1:4724/wd/hub/session/DD258CF1-CE6A-4875-80BA-56D9BA28F94A/element] with body: {"using":"-image","value":"***"}
[WinAppDriver] [STDOUT]
[WinAppDriver] [STDOUT]
[WinAppDriver] [STDOUT] ==========================================
[WinAppDriver] [STDOUT] POST /wd/hub/session/DD258CF1-CE6A-4875-80BA-56D9BA28F94A/element HTTP/1.1
[WinAppDriver] [STDOUT] Accept: application/json, */*
[WinAppDriver] [STDOUT] Connection: close
[WinAppDriver] [STDOUT] Content-Length: 737
[WinAppDriver] [STDOUT] Content-Type: application/json; charset=utf-8
[WinAppDriver] [STDOUT] Host: 127.0.0.1:4724
[WinAppDriver] [STDOUT] User-Agent: appium
[WinAppDriver] [STDOUT]
[WinAppDriver] [STDOUT] {"using":"-image","value":"***"}
[WinAppDriver] [STDOUT] HTTP/1.1 400 Bad Request
[WinAppDriver] [STDOUT] Content-Length: 87
[WinAppDriver] [STDOUT] Content-Type: application/json
[WinAppDriver] [STDOUT]
[WinAppDriver] [STDOUT] {"status":32,"value":{"error":"invalid selector","message":"Invalid selector: -image"}}
[WD Proxy] Got response with status 400: {"status":32,"value":{"error":"invalid selector","message":"Invalid selector: -image"}}
[debug] [MJSONWP (8fccee01)] Encountered internal error running command: ProxyRequestError: Could not proxy command to remote server. Original error: 400 - {"status":32,"value":{"error":"invalid selector","message":"Invalid selector: -image"}}
[debug] [MJSONWP (8fccee01)]     at JWProxy.proxy (C:\Users\automation\AppData\Roaming\npm\node_modules\appium-windows-driver\node_modules\appium-base-driver\lib\jsonwp-proxy\proxy.js:218:13)
[debug] [MJSONWP (8fccee01)]     at runMicrotasks (<anonymous>)
[debug] [MJSONWP (8fccee01)]     at runNextTicks (internal/process/task_queues.js:62:5)
[debug] [MJSONWP (8fccee01)]     at processImmediate (internal/timers.js:429:9)
[debug] [W3C] Matched W3C error code 'invalid selector' to InvalidSelectorError
[HTTP] <-- POST /wd/hub/session/8fccee01-f83f-4bdb-8d81-0a78453fbea7/element 500 33 ms - 111

mykola-mokhnach · 2020-03-16T19:55:02Z

Still seem to get the same response, but think I've followed the above guide correctly:

This is easy to check, simply add something like throw new Error('yolo'); to the findElOrEls method to see whether it is called or not ;)

jsa34 · 2020-03-16T20:10:09Z

Yep, as expected, it isn't calling find.js.

I had noticed this before, though, and it may be the root cause of the problem: No matter what I do, all .findElement calls, etc, return back an element of type WebElement, and I can never get a WindowsElement. In the error stacktrace, it looks like appium is resolving to the native selenium find methods so never looks for find.js...

Caused by: org.openqa.selenium.InvalidSelectorException: Invalid selector: -image (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/invalid_selector_exception.html
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
System info: host: '***', ip: '***', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_221'
Driver info: io.appium.java_client.windows.WindowsDriver
Capabilities {app: Root, deviceName: WindowsPC, experimental-webdriver: true, javascriptEnabled: true, platform: WINDOWS, platformName: Windows}
Session ID: 2afdf5b7-ca93-4ce7-9e20-39bc4ddf04e8
*** Element info: {Using=-image, value=***}
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
	at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
	at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
	at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
	at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
	at io.appium.java_client.remote.AppiumCommandExecutor.execute(AppiumCommandExecutor.java:239)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)
	at io.appium.java_client.DefaultGenericMobileDriver.execute(DefaultGenericMobileDriver.java:41)
	at io.appium.java_client.AppiumDriver.execute(AppiumDriver.java:1)
	at io.appium.java_client.windows.WindowsDriver.execute(WindowsDriver.java:1)
	at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:323)
	at io.appium.java_client.DefaultGenericMobileDriver.findElement(DefaultGenericMobileDriver.java:61)
	at io.appium.java_client.AppiumDriver.findElement(AppiumDriver.java:1)
	at io.appium.java_client.windows.WindowsDriver.findElement(WindowsDriver.java:1)
	at io.appium.java_client.FindsByImage.findElementByImage(FindsByImage.java:42)
	at io.appium.java_client.MobileBy$ByImage.findElement(MobileBy.java:567)
	at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:315)
	at io.appium.java_client.DefaultGenericMobileDriver.findElement(DefaultGenericMobileDriver.java:57)
	at io.appium.java_client.AppiumDriver.findElement(AppiumDriver.java:1)
	at io.appium.java_client.windows.WindowsDriver.findElement(WindowsDriver.java:1)
	at org.openqa.selenium.support.ui.ExpectedConditions$6.apply(ExpectedConditions.java:182)
	at org.openqa.selenium.support.ui.ExpectedConditions$6.apply(ExpectedConditions.java:179)
	at io.appium.java_client.AppiumFluentWait.until(AppiumFluentWait.java:222)

jsa34 · 2020-03-16T20:19:10Z

Just trying with a simple driver.findElementByImage results in the same, all the findElementBy* return WebElements instead of WindowsElements for the following simple code (Tried to make sure at each step drivers are definitely casted to WindowsDriver and elements are casted to WindowsElements if some reason they aren't) ... (Guessing this is a java-client issue?)

        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("deviceName", "WindowsPC");
        capabilities.setCapability("app", "Root");
        capabilities.setCapability("ms:experimental-webdriver", true);

        WindowsDriver<WindowsElement> driver = new WindowsDriver<>(currentDriver.getRemoteAddress(), capabilities);
        WindowsElement windowsElement =  (WindowsElement) driver.findElementByImage(imageName);

Error:

org.openqa.selenium.InvalidSelectorException: Invalid selector: -image (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 0 milliseconds
For documentation on this error, please visit: https://www.seleniumhq.org/exceptions/invalid_selector_exception.html
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
System info: host: '***', ip: '***', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_221'
Driver info: io.appium.java_client.windows.WindowsDriver
Capabilities {app: Root, deviceName: WindowsPC, experimental-webdriver: true, javascriptEnabled: true, platform: WINDOWS, platformName: Windows}
Session ID: bf325fbf-f61b-4cb7-a6e5-873552a4767d
*** Element info: {Using=-image, value=***}
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:214)
	at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:166)
	at org.openqa.selenium.remote.http.JsonHttpResponseCodec.reconstructValue(JsonHttpResponseCodec.java:40)
	at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:80)
	at org.openqa.selenium.remote.http.AbstractHttpResponseCodec.decode(AbstractHttpResponseCodec.java:44)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:158)
	at io.appium.java_client.remote.AppiumCommandExecutor.execute(AppiumCommandExecutor.java:239)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:552)
	at io.appium.java_client.DefaultGenericMobileDriver.execute(DefaultGenericMobileDriver.java:41)
	at io.appium.java_client.AppiumDriver.execute(AppiumDriver.java:1)
	at io.appium.java_client.windows.WindowsDriver.execute(WindowsDriver.java:1)
	at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:323)
	at io.appium.java_client.DefaultGenericMobileDriver.findElement(DefaultGenericMobileDriver.java:61)
	at io.appium.java_client.AppiumDriver.findElement(AppiumDriver.java:1)
	at io.appium.java_client.windows.WindowsDriver.findElement(WindowsDriver.java:1)
	at io.appium.java_client.FindsByImage.findElementByImage(FindsByImage.java:42)

mykola-mokhnach · 2020-03-16T20:30:08Z

Are you sure you use appium@beta as the base? With 1.16 it won't work for sure

jsa34 · 2020-03-16T20:31:38Z

Yes:

[Appium] Welcome to Appium v1.17.0-beta.1 (REV 8cffb5274854c4cc1e639d56fb2e539c6a5c1786)
[Appium] Appium REST http interface listener started on 0.0.0.0:4723

Is logged on startup

mykola-mokhnach · 2020-03-16T20:41:45Z

ok, this is fine. Now it might necessary to check if the endpoint is proxied properly. For that you might add some logging to the appium-base-driver module:

https://github.com/appium/appium-base-driver/blob/8624a8061f836c14e331aeace39a352d31fcb875/lib/basedriver/driver.js#L491 to make sure the /element endpoint does not get proxied immediately
https://github.com/appium/appium-base-driver/blob/8624a8061f836c14e331aeace39a352d31fcb875/lib/basedriver/commands/find.js#L47 to make sure the base driver's findElement is called first

mykola-mokhnach · 2020-03-16T20:43:20Z

the simplest way to add logging would be to just modify the corresponding files inside build folder of the corresponding module. console.log('>>> my log'); would be enough to see the result in appium log

jlipps · 2020-03-16T20:46:21Z

lib/commands/find.js

+
+const commands = {};
+
+commands.findElOrEls = async function findElOrEls (strategy, selector, mult, context) {


seems like we could have a generic findElOrElsByProxy method since we do this in many drivers.?

yes, this improvement could be done in a separate PR

jsa34 · 2020-03-16T22:21:18Z

Managed to find the problem - it was silently failing to update the build after doing an install... Fixed now.

So, I have 2 issues:

1. In the current code, previous requests for locator strategies such as By.name, By.xpath are now throwing invalid locator errors. However, locator -image is fine!  It too throws an error but saying:
`[BaseDriver] Finding image element with match threshold 0.4
[debug] [MJSONWP (f720090e)] Encountered internal error running command: Error: This driver does not support the required 'getWindowSize' command`

UPDATE: Discovered this is because the driver was instantiated at root level, which apparently throws a null pointer for .manage().window().getSize(). After creating a driver linked to a window handle, it no longer throws null pointers and there is no issue. Perhaps a helpful error message would help here?

2.Previously working locators now fail:
[debug] [MJSONWP (f720090e)] Calling AppiumDriver.findElement() with args: ["name","Open:","f720090e-64a9-4cd1-bd83-2fb3b983fcb5"] [debug] [BaseDriver] Valid locator strategies for this request: [debug] [MJSONWP (f720090e)] Encountered internal error running command: InvalidSelectorError: Locator Strategy 'name' is not supported for this session

I tried the RegExs before the last commit you made, and thought it was worth noting that with mykola-mokhnach@f4989e6 commit reversed, the old behaviour returns: all locators other than image are fine, and image throws a "InvalidSelectorError"

mykola-mokhnach · 2020-03-17T07:06:13Z

Cool, this is already something. Let me do some more magic...

mykola-mokhnach · 2020-03-17T07:35:22Z

After 84570b7 the errors above should be gone ;)

jsa34 · 2020-03-17T15:12:05Z

Fab - all locators are now functional!

Tried finding by image, but getting the following error (made sure it has started a driver session with a top level window handle defined):
org.openqa.selenium.WebDriverException: An unknown server-side error occurred while processing the command. Original error: Cannot destructure property 'width' of '(intermediate value)' as it is undefined. (WARNING: The server did not provide any stacktrace information)
appium error:

[debug] [MJSONWP (d31fc9cf)] Calling AppiumDriver.findElement() with args: ["-image","***"]
[BaseDriver] Finding image element with match threshold 0.4
[debug] [WD Proxy] Proxying [GET /window/size] to [GET http://127.0.0.1:4724/wd/hub/session/E9BD5D69-A757-4EAA-8A55-A66A6C080A4C/window/size] with no body
[WinAppDriver] [STDOUT]
[WinAppDriver] [STDOUT]
[WinAppDriver] [STDOUT] ==========================================
[WinAppDriver] [STDOUT] GET /wd/hub/session/E9BD5D69-A757-4EAA-8A55-A66A6C080A4C/window/size HTTP/1.1
[WinAppDriver] [STDOUT] Accept: application/json, */*
[WinAppDriver] [STDOUT] Connection: close
[WinAppDriver] [STDOUT] Content-Type: application/json; charset=utf-8
[WinAppDriver] [STDOUT] Host: 127.0.0.1:4724
[WinAppDriver] [STDOUT] User-Agent: appium
[WinAppDriver] [STDOUT]
[WinAppDriver] [STDOUT]
[WinAppDriver] [STDOUT] HTTP/1.1 200 OK
[WinAppDriver] [STDOUT] Content-Length: 63
[WinAppDriver] [STDOUT] Content-Type: application/json
[WinAppDriver] [STDOUT]
[WinAppDriver] [STDOUT] {"sessionId":"E9BD5D69-A757-4EAA-8A55-A66A6C080A4C","status":0}
[debug] [WD Proxy] Got response with status 200: {"sessionId":"E9BD5D69-A757-4EAA-8A55-A66A6C080A4C","status":0}
[debug] [MJSONWP (d31fc9cf)] Encountered internal error running command: TypeError: Cannot destructure property 'width' of '(intermediate value)' as it is undefined.
[debug] [MJSONWP (d31fc9cf)]     at WindowsDriver.findByImage (***\local_modules\appium\node_modules\appium-base-driver\lib\basedriver\commands\find.js:221:17)
[debug] [MJSONWP (d31fc9cf)]     at runMicrotasks (<anonymous>)
[debug] [MJSONWP (d31fc9cf)]     at runNextTicks (internal/process/task_queues.js:62:5)
[debug] [MJSONWP (d31fc9cf)]     at processImmediate (internal/timers.js:429:9)
[debug] [MJSONWP (d31fc9cf)]     at WindowsDriver.findElement (***\local_modules\appium\node_modules\appium-base-driver\lib\basedriver\commands\find.js:48:12)
[HTTP] <-- POST /wd/hub/session/d31fc9cf-761d-4e0d-a4a3-c1b198e323da/element 500 38 ms - 252

Example of driver creation code:

            DesiredCapabilities capabilities = new DesiredCapabilities();
            capabilities.setCapability("platformName", "Windows");
            capabilities.setCapability("deviceName", "WindowsPC");
            capabilities.setCapability("ms:experimental-webdriver", true);
            capabilities.setCapability("appTopLevelWindow", desiredWindowHandle);
            WindowsDriver<WindowsElement> d = new WindowsDriver<>(getDriver().getRemoteAddress(), capabilities);

Also tried d.manage().window().getSize() and this returned a null pointer... Suspect this is the root cause

mykola-mokhnach · 2020-03-17T15:21:37Z

Also tried d.manage().window().getSize() and this returned a null pointer... Suspect this is the root cause

This API is expected to return {width, height} set to valid window dimension values. In case the returned value is null then it is a highly possible bug inside https://github.com/microsoft/WinAppDriver

Could you please describe this issue to Microsoft and see what their response would be?

mykola-mokhnach · 2020-03-17T15:23:55Z

Btw, they declare the GET /window/size endpoint as supported: https://github.com/microsoft/WinAppDriver/blob/master/Docs/SupportedAPIs.md

I would also make sure we use the most recent server version from them

jsa34 · 2020-03-17T15:44:00Z

Hello - I am using 1.2-RC... I can't see why this is returning as I too saw it listed in their supported endpoints...

jsa34 · 2020-03-17T17:02:08Z

Is there a way of trying different winappdriver versions, as using 1.7.0 beta seems to tie me to 1.2-RC version so I can't verify if other versions of winappdriver support this endpoint?

jsa34 · 2020-03-17T17:09:41Z

No worries - found how to do it... Just trying different winappdriver versions...

jsa34 · 2020-03-17T17:37:11Z

So, tried the following, but no luck:

switching to v1.1: same behaviour,
-making sure the driver.manage().switchTo().window(currentWindowHandle); is set results in same behaviour, too

I have filed an issue with WAD team, but they don't seem to maintaining it from recent activity...

jsa34 · 2020-03-17T18:41:37Z

Just seen a possible workaround would be to use the element of the window instead of the /window/size endpoint as they should be equal according to the test:
https://github.com/microsoft/WinAppDriver/blob/master/Tests/WebDriverAPI/ElementSize.cs (line 55-58)

Not sure if this also returns null or if it is a viable workaround, but just thought it worth mentioning

mykola-mokhnach · 2020-03-17T19:01:06Z

I've added a workaround there. I don't like it, but that's anyway better than waiting until MS fixes the server. Please try it and let me know if it works

jsa34 · 2020-03-17T20:09:14Z

Great! This is working now!

jsa34 · 2020-03-17T20:17:36Z

Just ran it again, not sure what I changed, but MobileBy throws the following...

[debug] [WD Proxy] Got response with status 200: {"sessionId":"79B06E6E-B32F-4954-A204-1078C903C4B6","status":0,"value":"***...
[BaseDriver] Verifying screenshot size and aspect ratio
[BaseDriver] When trying to find an element, determined that the screen aspect ratio and screenshot aspect ratio are different. Screen is 1680x1050 whereas screenshot is 1920x1040.
[BaseDriver] Resizing screenshot to 1901.7142857142858x1030.095238095238 to match screen aspect ratio so that image element coordinates have a greater chance of being correct.
[BaseDriver] Scaling screenshot from 1902x1030 to match screen at 1680x1050
[BaseDriver] Image template matched: {"x":112,"y":324,"width":70,"height":18}
[BaseDriver] Will tap on image element at coordinate [147, 333]
[BaseDriver] Will tap using W3C actions
[BaseDriver] Driver does not seem to implement W3C actions, falling back to TouchActions
[BaseDriver] Will tap using MJSONWP TouchActions
[debug] [MJSONWP (d5e83d88)] Encountered internal error running command: Error: Driver did not implement the 'performTouch' command. For drivers to support finding image elements, they should support 'performTouch' and 'performActions'
[debug] [MJSONWP (d5e83d88)]     at ImageElement.click (C:\Users\automation\Documents\local_modules\appium\node_modules\appium-base-driver\lib\basedriver\image-element.js:207:11)
[debug] [MJSONWP (d5e83d88)]     at Function.execute (C:\Users\automation\Documents\local_modules\appium\node_modules\appium-base-driver\lib\basedriver\image-element.js:231:16)
[debug] [MJSONWP (d5e83d88)]     at commandExecutor (C:\Users\automation\Documents\local_modules\appium\node_modules\appium-base-driver\lib\basedriver\driver.js:328:9)
[HTTP] <-- POST /wd/hub/session/d5e83d88-dab1-4882-8685-44a6818a025b/element/appium-image-element-15224153-153e-40a5-a3d8-e3df69d86e9d/click 500 1068 ms - 326

And trying to not use MobileBy and a native findsByImage throws casting exception:
java.lang.ClassCastException: org.openqa.selenium.remote.RemoteWebElement cannot be cast to io.appium.java_client.windows.WindowsElement

Using the following code (DriverManager.getDriver() already returns an instance of WindowsDriver but I am trying to cover all casting issues...):

        WindowsDriver<WindowsElement> driver = DriverManager.getDriver();
        WindowsElement e = driver.findElementByImage(ImageManager.getImage(imageName));
        e.click();

mykola-mokhnach · 2020-03-17T20:45:31Z

These errors are expected. We'd have to add a wrapper for touch actions to make clicks working same way the wrappers for screenshots/screensize have been added, but not in this PR.
Would you like to try it yourself?

KazuCocoa · 2020-03-18T02:26:38Z

lib/commands/general.js

+  }
+  // workaround for https://github.com/microsoft/WinAppDriver/issues/1104
+  log.info('Cannot retrieve window size from WinAppDriver. ' +
+    'Falling back to WMIC usage');


jsa34 · 2020-03-18T09:05:40Z

Thanks for the explanation - I will have a look at the wrapper solution for touch actions, but I still don't understand why:
WindowsElement e = driver.findElementByImage(ImageManager.getImage(imageName)); Where driver is definitely of type WindowsDriver< WindowsElement > throws the following error:
java.lang.ClassCastException: org.openqa.selenium.remote.RemoteWebElement cannot be cast to io.appium.java_client.windows.WindowsElement
It's something internal where .findElementByImage should return back a WindowsElement but it looks like some internal casting to RemoteWebElement causes issues with returning this type back?

feat: Add findByImage support to Windows driver

7a8fa6e

mykola-mokhnach requested review from jlipps and imurchie March 16, 2020 14:27

Simplify conditions

ef7e2f9

jsa34 mentioned this pull request Mar 16, 2020

findByImage microsoft/WinAppDriver#1102

Closed

mykola-mokhnach added 2 commits March 16, 2020 15:42

Tune comments

847a0a7

Tune regexps

f4989e6

KazuCocoa approved these changes Mar 16, 2020

View reviewed changes

jlipps approved these changes Mar 16, 2020

View reviewed changes

Add the necessary requirements

84570b7

jsa34 mentioned this pull request Mar 17, 2020

GET /window/size returns null microsoft/WinAppDriver#1104

Open

Add workaround

a7f1733

mykola-mokhnach merged commit b6832b8 into appium:master Mar 17, 2020

mykola-mokhnach deleted the find_proxy branch March 17, 2020 20:45

KazuCocoa reviewed Mar 18, 2020

View reviewed changes

mykola-mokhnach mentioned this pull request Jun 8, 2020

Cannot use ImageComparison on MacOS apps appium/appium#14424

Closed


		const commands = {};

		commands.findElOrEls = async function findElOrEls (strategy, selector, mult, context) {

feat: Add findByImage support to Windows driver #67

feat: Add findByImage support to Windows driver #67

Conversation

mykola-mokhnach commented Mar 16, 2020

jsa34 commented Mar 16, 2020

mykola-mokhnach commented Mar 16, 2020

jsa34 commented Mar 16, 2020

jsa34 commented Mar 16, 2020

mykola-mokhnach commented Mar 16, 2020

jsa34 commented Mar 16, 2020

mykola-mokhnach commented Mar 16, 2020 • edited Loading

KazuCocoa left a comment

Choose a reason for hiding this comment

jsa34 commented Mar 16, 2020 • edited Loading

mykola-mokhnach commented Mar 16, 2020

jsa34 commented Mar 16, 2020 • edited Loading

jsa34 commented Mar 16, 2020 • edited Loading

mykola-mokhnach commented Mar 16, 2020

jsa34 commented Mar 16, 2020

mykola-mokhnach commented Mar 16, 2020

mykola-mokhnach commented Mar 16, 2020

jlipps Mar 16, 2020

Choose a reason for hiding this comment

mykola-mokhnach Mar 17, 2020

Choose a reason for hiding this comment

jsa34 commented Mar 16, 2020 • edited Loading

mykola-mokhnach commented Mar 17, 2020

mykola-mokhnach commented Mar 17, 2020

jsa34 commented Mar 17, 2020 • edited Loading

mykola-mokhnach commented Mar 17, 2020 • edited Loading

mykola-mokhnach commented Mar 17, 2020 • edited Loading

jsa34 commented Mar 17, 2020

jsa34 commented Mar 17, 2020

jsa34 commented Mar 17, 2020

jsa34 commented Mar 17, 2020

jsa34 commented Mar 17, 2020

mykola-mokhnach commented Mar 17, 2020

jsa34 commented Mar 17, 2020 • edited Loading

jsa34 commented Mar 17, 2020 • edited Loading

mykola-mokhnach commented Mar 17, 2020

KazuCocoa Mar 18, 2020

Choose a reason for hiding this comment

jsa34 commented Mar 18, 2020 • edited Loading

mykola-mokhnach commented Mar 16, 2020 •

edited

Loading

jsa34 commented Mar 16, 2020 •

edited

Loading

jsa34 commented Mar 16, 2020 •

edited

Loading

jsa34 commented Mar 16, 2020 •

edited

Loading

jsa34 commented Mar 16, 2020 •

edited

Loading

jsa34 commented Mar 17, 2020 •

edited

Loading

mykola-mokhnach commented Mar 17, 2020 •

edited

Loading

mykola-mokhnach commented Mar 17, 2020 •

edited

Loading

jsa34 commented Mar 17, 2020 •

edited

Loading

jsa34 commented Mar 17, 2020 •

edited

Loading

jsa34 commented Mar 18, 2020 •

edited

Loading