Confirm commands for W3C actions #733

KazuCocoa · 2017-12-18T13:17:12Z

This is a

Bug report
Question
Freature Request

Summary

https://github.com/jlipps/simple-wd-spec#perform-actions
For
- XCUITest
- UIA2

W3CActionBuilder: https://seleniumhq.github.io/selenium/docs/api/rb/Selenium/WebDriver/W3CActionBuilder.html

Environment

Appium version (or git revision):
ruby_lib version:
Mobile platform/version/device under test:

Actual behaviour and steps to reproduce

Expected behaviour

Link to Appium/Ruby logs

Create a GIST which is a paste of your full Appium logs, and link them here.

Any additional comments

The text was updated successfully, but these errors were encountered:

KazuCocoa · 2017-12-18T13:19:33Z

Check https://github.com/appium/ruby_lib_core first.
Wrap commands for current TouchAction if we need for the compatibility.
- Hint for current implementation
  - https://github.com/SeleniumHQ/selenium/blob/master/rb/lib/selenium/webdriver/common/touch_action_builder.rb#L18

KazuCocoa · 2017-12-21T13:12:51Z

UIA2

Easy action. Just tapping a particular element.

[debug] [JSONWP Proxy] Proxying [POST /element] to [POST http://localhost:8200/wd/hub/session/b1b98b5a-c70e-4e1e-9365-6377ac922561/element] with body: {"strategy":"accessibility id","selector":"App","context":"","multiple":false}
[debug] [JSONWP Proxy] Got response with status 200: {"sessionId":"b1b98b5a-c70e-4e1e-9365-6377ac922561","status":0,"value":{"ELEMENT":"993b2c92-9a36-490a-be84-32a23b7b7f4f"}}
[debug] [MJSONWP] Responding to client with driver.findElement() result: {"ELEMENT":"993b2c92-9a36-490a-be84-32a23b7b7f4f"}
[HTTP] <-- POST /wd/hub/session/d7968638-90be-4306-b4e0-d15ac0fbf284/element 200 31 ms - 122
[HTTP] --> POST /wd/hub/session/d7968638-90be-4306-b4e0-d15ac0fbf284/actions {"actions":[{"type":"pointer","id":"mouse","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"993b2c92-9a36-490a-be84-32a23b7b7f4f"}},{"type":"pointerDown","button":0},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"mouse"}}]}
[MJSONWP] Driver proxy active, passing request on via HTTP proxy
[debug] [JSONWP Proxy] Proxying [POST /wd/hub/session/d7968638-90be-4306-b4e0-d15ac0fbf284/actions] to [POST http://localhost:8200/wd/hub/session/b1b98b5a-c70e-4e1e-9365-6377ac922561/actions] with body: {"actions":[{"type":"pointer","id":"mouse","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"993b2c92-9a36-490a-be84-32a23b7b7f4f"}},{"type":"pointerDown","button":0},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"mouse"}}]}
[debug] [JSONWP Proxy] Got response with status 200: {"sessionId":"b1b98b5a-c70e-4e1e-9365-6377ac922561","status":0,"value":"OK"}
[JSONWP Proxy] Replacing sessionId b1b98b5a-c70e-4e1e-9365-6377ac922561 with d7968638-90be-4306-b4e0-d15ac0fbf284
[HTTP] <-- POST /wd/hub/session/d7968638-90be-4306-b4e0-d15ac0fbf284/actions 200 306 ms - 76

KazuCocoa · 2017-12-21T13:29:00Z

UIA2

And I also can do the following command:

        el = @@driver.find_element(:accessibility_id, 'Views')
        @@driver.action.click(el).perform

        el = @@driver.find_element(:accessibility_id, 'Custom')
        @@driver.action.click_and_hold(el).move_to_location(0, 500).release.perform

[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, class name, accessibility id, -android uiautomator
[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, class name, accessibility id, -android uiautomator
[debug] [BaseDriver] Waiting up to 0 ms for condition
[debug] [JSONWP Proxy] Proxying [POST /element] to [POST http://localhost:8200/wd/hub/session/022a3c95-7ae6-4dcc-b7ec-bc9469ad6d3a/element] with body: {"strategy":"accessibility id","selector":"Custom","context":"","multiple":false}
[debug] [JSONWP Proxy] Got response with status 200: {"sessionId":"022a3c95-7ae6-4dcc-b7ec-bc9469ad6d3a","status":0,"value":{"ELEMENT":"900590bc-1fae-450e-afc4-be714283cf67"}}
[debug] [MJSONWP] Responding to client with driver.findElement() result: {"ELEMENT":"900590bc-1fae-450e-afc4-be714283cf67"}
[HTTP] <-- POST /wd/hub/session/1b67cce8-2654-4315-afcd-c5fa9fa21438/element 200 32 ms - 122
[HTTP] --> POST /wd/hub/session/1b67cce8-2654-4315-afcd-c5fa9fa21438/actions {"actions":[{"type":"pointer","id":"mouse","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"900590bc-1fae-450e-afc4-be714283cf67"}},{"type":"pointerDown","button":0},{"type":"pointerMove","duration":250,"x":0,"y":500,"origin":"viewport"},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"mouse"}}]}
[MJSONWP] Driver proxy active, passing request on via HTTP proxy
[debug] [JSONWP Proxy] Proxying [POST /wd/hub/session/1b67cce8-2654-4315-afcd-c5fa9fa21438/actions] to [POST http://localhost:8200/wd/hub/session/022a3c95-7ae6-4dcc-b7ec-bc9469ad6d3a/actions] with body: {"actions":[{"type":"pointer","id":"mouse","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"900590bc-1fae-450e-afc4-be714283cf67"}},{"type":"pointerDown","button":0},{"type":"pointerMove","duration":250,"x":0,"y":500,"origin":"viewport"},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"mouse"}}]}
[debug] [JSONWP Proxy] Got response with status 200: {"sessionId":"022a3c95-7ae6-4dcc-b7ec-bc9469ad6d3a","status":0,"value":"OK"}
[JSONWP Proxy] Replacing sessionId 022a3c95-7ae6-4dcc-b7ec-bc9469ad6d3a with 1b67cce8-2654-4315-afcd-c5fa9fa21438
[HTTP] <-- POST /wd/hub/session/1b67cce8-2654-4315-afcd-c5fa9fa21438/actions 200 1750 ms - 76
[BaseDriver] Shutting down because we waited 60 seconds for a command
[debug] [UiAutomator2] Deleting UiAutomator2 session
[Appium] Closing session, cause was 'New Command Timeout of 60 seconds expired. Try customizing the timeout using the 'newCommandTimeout' desired capability'
[Appium] Removing session 1b67cce8-2654-4315-afcd-c5fa9fa21438 from our master session list
[debug] [UiAutomator2] Deleting UiAutomator2 server session

Documentation:

For client, we can call previous touchactions.
Thus, I put some documentation and examples in test case, but I keep the touch methods.

KazuCocoa · 2017-12-21T15:39:47Z

WDA

By default, Ruby client "parameters":{"pointerType":"mouse"} and WDA return error about:

Selenium::WebDriver::Error::UnknownError: Error Domain=com.facebook.WebDriverAgent Code=1 "Only pointer type 'touch' is supported. 'mouse' is given instead for action with id 'mouse'" UserInfo={NSLocalizedDescription=Only pointer type 'touch' is supported. 'mouse' is given instead for action with id 'mouse'}

UIA2 can work with mouse. Can WDA relax to allow mouse?

[debug] [JSONWP Proxy] Proxying [POST /wd/hub/session/96e1f06a-8dc9-4a83-9f07-1322c7ae63ac/actions] to [POST http://localhost:8100/session/24CD3298-9992-4B15-AB5A-B1DF025C2621/actions] with body: {"actions":[{"type":"pointer","id":"mouse","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"F02C3E0A-F230-42B5-8ECD-88E4A381075D"}},{"type":"pointerDown","button":0},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"mouse"}}]}
[debug] [JSONWP Proxy] Got response with status 200: {"value":"Error Domain=com.facebook.WebDriverAgent Code=1 \"Only pointer type 'touch' is supported. 'mouse' is given instead for action with id 'mouse'\" UserInfo={NSLocalizedDescription=Only pointer type 'touch' is supported. 'mouse' is given instead for action with id 'mouse'}","sessionId":"24CD3298-9992-4B15-AB5A-B1DF025C2621","status":13}
[JSONWP Proxy] Replacing sessionId

I could do with touch like the below:

        action = ::Selenium::WebDriver::W3CActionBuilder.new @@driver.send(:bridge), ::Selenium::WebDriver::Interactions::PointerInput.new(:touch, name: 'touch'), Selenium::WebDriver::Interactions::KeyInput.new('keyboard'), false

        el = @@driver.find_element(:accessibility_id, 'Buttons')
        action.click(el).perform

        el = @@driver.find_element(:id, 'XCUIElementTypeApplication')
        action.click_and_hold(el).move_to_location(0, 700).release.perform

        el = @@driver.find_element(:accessibility_id, 'ImageButton')
        assert_equal 'ImageButton', el.name

KazuCocoa · 2017-12-21T15:47:43Z

@mykola-mokhnach
Can you relax the pointerType for WDA?
Currently, 'mouse' is used by default for the parameter because of W3C spec.

To set touch in the action, Ruby client need to the below line and only for WDA, not UIA2, so far.

::Selenium::WebDriver::W3CActionBuilder.new @@driver.send(:bridge), ::Selenium::WebDriver::Interactions::PointerInput.new(:touch, name: 'touch'), Selenium::WebDriver::Interactions::KeyInput.new('keyboard'), false

mykola-mokhnach · 2017-12-21T16:54:11Z

@KazuCocoa It's by design. W3C spec makes a clear distinction between different pointer types (currently 3 types are defined: mouse, pen and touch). Only touch pointers are supported on iOS and Android has support of all types. That is why you see the error below. We could think about additional wrappers/helpers on appium Ruby client side to make the creation on Actions object with touch pointer type easier

KazuCocoa · 2017-12-21T23:34:50Z

https://github.com/jlipps/simple-wd-spec#location-strategies

CSS selector css selector

Link text selector link text

Partial link text selector partial link text

Tag name tag name

XPath selector xpath

Ruby client converts some elements to css celector like:

        def convert_locators(how, what)
          case how
          when 'class name'
            how = 'css selector'
            what = ".#{escape_css(what)}"
          when 'id'
            how = 'css selector'
            what = "##{escape_css(what)}"
          when 'name'
            how = 'css selector'
            what = "*[name='#{escape_css(what)}']"
          when 'tag name'
            how = 'css selector'
          end
          [how, what]
        end

Don't convert to Appium, so far.

KazuCocoa · 2017-12-21T23:47:37Z

📝
Use 'touch' by default

[HTTP] --> POST /wd/hub/session/ddb1d5c6-affa-4b47-aeef-97eb95777093/actions {"actions":[{"type":"pointer","id":"touch","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"3041AA5D-53EF-48D0-80B3-E30EE2012EB8"}},{"type":"pointerDown","button":0},{"type":"pointerMove","duration":250,"x":0,"y":300,"origin":"viewport"},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"touch"}}]}
[MJSONWP] Driver proxy active, passing request on via HTTP proxy
[debug] [XCUITest] Executing command 'proxyReqRes'
[debug] [JSONWP Proxy] Proxying [POST /wd/hub/session/ddb1d5c6-affa-4b47-aeef-97eb95777093/actions] to [POST http://localhost:8100/session/780847CE-1494-4EF5-A2B9-12C2C84F4E42/actions] with body: {"actions":[{"type":"pointer","id":"touch","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"3041AA5D-53EF-48D0-80B3-E30EE2012EB8"}},{"type":"pointerDown","button":0},{"type":"pointerMove","duration":250,"x":0,"y":300,"origin":"viewport"},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"touch"}}]}
[debug] [JSONWP Proxy] Got response with status 200: {"value":{},"sessionId":"780847CE-1494-4EF5-A2B9-12C2C84F4E42","status":0}
[JSONWP Proxy] Replacing sessionId 780847CE-1494-4EF5-A2B9-12C2C84F4E42 with ddb1d5c6-affa-4b47-aeef-97eb95777093
[HTTP] <-- POST /wd/hub/session/ddb1d5c6-affa-4b47-aeef-97eb95777093/actions 200 990 ms - 74

KazuCocoa · 2017-12-22T00:01:02Z

✅ appium/ruby_lib_core@2868488

KazuCocoa · 2017-12-23T05:02:42Z

📝
post timeout's parameters differ between OSS and W3C.
=> https://github.com/jlipps/simple-wd-spec#set-timeouts

KazuCocoa changed the title ~~W3C actions~~ Confirm commands for W3C actions Dec 18, 2017

KazuCocoa self-assigned this Dec 18, 2017

KazuCocoa added this to the v9.8.2 milestone Dec 20, 2017

This was referenced Dec 23, 2017

Update and add description for actions #737

Merged

List incompatible commands #739

Closed

KazuCocoa closed this as completed Dec 25, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Confirm commands for W3C actions #733

Confirm commands for W3C actions #733

KazuCocoa commented Dec 18, 2017 •

edited

Loading

KazuCocoa commented Dec 18, 2017 •

edited

Loading

KazuCocoa commented Dec 21, 2017 •

edited

Loading

KazuCocoa commented Dec 21, 2017 •

edited

Loading

KazuCocoa commented Dec 21, 2017

KazuCocoa commented Dec 21, 2017

mykola-mokhnach commented Dec 21, 2017

KazuCocoa commented Dec 21, 2017

KazuCocoa commented Dec 21, 2017

KazuCocoa commented Dec 22, 2017

KazuCocoa commented Dec 23, 2017

Confirm commands for W3C actions #733

Confirm commands for W3C actions #733

Comments

KazuCocoa commented Dec 18, 2017 • edited Loading

This is a

Summary

Environment

Actual behaviour and steps to reproduce

Expected behaviour

Link to Appium/Ruby logs

Any additional comments

KazuCocoa commented Dec 18, 2017 • edited Loading

KazuCocoa commented Dec 21, 2017 • edited Loading

UIA2

KazuCocoa commented Dec 21, 2017 • edited Loading

UIA2

KazuCocoa commented Dec 21, 2017

WDA

KazuCocoa commented Dec 21, 2017

mykola-mokhnach commented Dec 21, 2017

KazuCocoa commented Dec 21, 2017

KazuCocoa commented Dec 21, 2017

KazuCocoa commented Dec 22, 2017

KazuCocoa commented Dec 23, 2017

KazuCocoa commented Dec 18, 2017 •

edited

Loading

KazuCocoa commented Dec 18, 2017 •

edited

Loading

KazuCocoa commented Dec 21, 2017 •

edited

Loading

KazuCocoa commented Dec 21, 2017 •

edited

Loading