Add (nonstandard) support for Apple iPad keyboards

[krimple]

Issue comes from twitter conversation with @CompuIves who suggested logging it here.

According to this response to the lack of arrow key support on iOS Webkit/Safari, in https://bugs.webkit.org/show_bug.cgi?id=149054#c6 it is actually possible to get events from the Apple iPad keyboard. They are, of course, nonstandard, but making it possible to use arrow cursor keys in an editor box makes it a great scratchpad for thinking out loud in JS.

Here is the relevant comment:

 Willem L. Middelkoop 2018-01-30 22:32:46 PST

Workaround is possible:
- make sure some element has focus (so that key events do fire)
- event.keyCode will be 0 (which is non-desirable)
- however, event.key will have a value "UIKeyInputUpArrow", "UIKeyInputDownArrow", "UIKeyInputLeftArrow" or "UIKeyInputRightArrow"

I have confirmed this to work on iPad (in combination with hardware keyboard, type cover and bluetooth magic keyboard).

I'd be happy to test any builds with my iPad pro.

[CompuIves]

Hey! I think I have a (hacky) fix for this, could you test it for me?

PR is here: #660
And it's hosted here: http://pr660.cs.lbogdan.tk/s/new (will be online 10 minutes after this comment)

[krimple]

I will check this out tonight. Looking forward to it.

…

Sent from my iPhone

On Mar 23, 2018, at 10:39 AM, Ives van Hoorne ***@***.***> wrote: Hey! I think I have a (hacky) fix for this, could you test it for me? PR is here: #660 And it's hosted here: http://pr660.cs.lbogdan.tk/s/new (will be online 10 minutes after this comment) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[krimple]

Ok, if I click the keyboard icon and hold it, the arrow keys become active, but they sort of scroll the view as well as the text, and it zooms in a bit. If I navigate or click somewhere else the arrows stop working. So it’s kind of half-there. What is the significance of the keyboard icon? Does it activate some sort of editing mode where the cursor keys are active? Thanks, Ken

…

Sent from my iPad

On Mar 23, 2018, at 5:43 PM, Ken Rimple ***@***.***> wrote: I will check this out tonight. Looking forward to it. Sent from my iPhone > On Mar 23, 2018, at 10:39 AM, Ives van Hoorne ***@***.***> wrote: > > Hey! I think I have a (hacky) fix for this, could you test it for me? > > PR is here: #660 > And it's hosted here: http://pr660.cs.lbogdan.tk/s/new (will be online 10 minutes after this comment) > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub, or mute the thread. >

[aadsm]

@krimple if you switch the text editor to CodeMirror in the preferences menu the arrow keys work as expected.

However, there’s still a bunch of stuff that is not working as well on the iPad. Key bindings like cmd-shift-P don’t work for instance :(.
Also, is it possible to hide the left panel and increase/decrease the font size or zoom level?

[CompuIves]

Interesting...

Key bindings like cmd-shift-P don’t work for instance :(.

Hmm, maybe the iPad sends out different events?

Also, is it possible to hide the left panel and increase/decrease the font size or zoom level?

Yep! You can hide the side panel by clicking the active icon, you can change font size in preferences.

[aadsm]

Thanks! I guess what I want is the zoom instead :) Which one is the “active icon”?

[krimple]

I see the same things now that I switch to CodeMirror. This thing zipped up into a React Native app with custom keystrokes would be a killer app. I don't see code completion, and CTRL-SPACE shows a keyboard pick list / emoji selector. But this is progress. Ken Ken Rimple Training and Mentoring Services Chariot Solutions 610-608-3935 @RimpleOnTech

…

On Mon, Apr 2, 2018 at 11:26 AM, Ives van Hoorne ***@***.***> wrote: Interesting... Key bindings like cmd-shift-P don’t work for instance :(. Hmm, maybe the iPad sends out different events? Also, is it possible to hide the left panel and increase/decrease the font size or zoom level? Yep! You can hide the side panel by clicking the active icon, you can change font size in preferences. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#507 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAQll-u7wP5ozs-byN2nPEs_X3UBh8Nks5tkkM3gaJpZM4R9bGu> .

[aadsm]

@CompuIves I’ve checked the code, and according to https://github.com/CompuIves/codesandbox-client/blob/a911927458058fa3ca10a84b0f41066f8f3a84b0/packages/app/src/app/store/providers/KeybindingManager.js#L22 and https://github.com/CompuIves/codesandbox-client/blob/41301707af5926de266069074651fc5bc93f9544/packages/common/utils/keybindings.js#L7 the editor.open-quick-actions binding will only match if normalizeKey returns Meta right? This won’t happen, at least on the iPad. You need to check event.{shiftKey,altKey,ctrlKey,metaKey} instead.

[CompuIves]

Ahh, that makes it more complicated. I used event.metaKey before, but noticed that the behaviour was different between multiple browsers when more than 2 keys were pressed at the same time. Hmm, maybe we can do both, or select a way to handle it based on the platform.

I'm curious, does it work on this page? http://wangchujiang.com/hotkeys/

[aadsm]

That’s a pretty cool demo for a key library :D. But unfortunately no, it doesn’t work (it doesn’t even capture the cursor keys).

From what I can see the iPad doesn’t have a problem in capturing multiple keys pressed at once, you can check that here: https://codepen.io/chriscoyier/embed/mPgoYJ.

If you want my opinion I’d personally go with a way to handle keys based on the platform. Key handling is super hard on the web so separating might be better (I wish someone did something about it, but people seem more busy with shiny stuff).

[aadsm]

@CompuIves is it possible to install codesandbox locally so I can run a few tests w.r.t. keyboard?

[CompuIves]

Yep @aadsm! If you clone this repo and run yarn && yarn start you should be able to access it at localhost:3000

[aadsm]

Thanks! I figured it out in the meanwhile by reading the package.json but my raspberry pi said no :D (there’s a point in time where yarn fails). Need to find a real computer to make it work.

[krimple]

Works for me on OS X. Is there a code branch with the iOS changes in it or is it in master? Ken Ken Rimple Training and Mentoring Services Chariot Solutions 610-608-3935 @RimpleOnTech

…

On Mon, Apr 9, 2018 at 11:04 AM, António Afonso ***@***.***> wrote: Thanks! I figured it out in the meanwhile by reading the package.json but my raspberry pi said no :D (there’s a point in time where yarn fails). Need to find a real computer to make it work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#507 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAQlhijW26go0bEhpb_nmPZ157HsVt8ks5tm3iCgaJpZM4R9bGu> .

[CompuIves]

Thanks! I guess what I want is the zoom instead :) Which one is the “active icon”?

Ahh, in that view there is no sidebar. When you're working on your own sandbox you see icons on the left. The one with the color is active:

Zoom can be built!

Works for me on OS X. Is there a code branch with the iOS changes in it or
is it in master?

I was working on this: #660 but it didn't work. That's my progress so far 😄

[aadsm]

@CompuIves what do you mean with “your own sandbox”? I thought that was the case actually :-).
I believe the issue with your PR is that Monaco uses a textarea which will not make the cursor keys trigger keyDown on iOS :(. CodeMirror is a contenteditable on mobile to avoid that issue. You can find more about this here: microsoft/monaco-editor#293.

However, from my own experimentation what I found is that the Meta, Ctrl, etc. keys are not fired on contenteditable as well :(. @NV can you advise here? :P

[aadsm]

Here’s some more thoughts on this: from what I can see you can’t detect any of these keys (shift, alt, ctrl, cmd, cursor) on either textarea or contenteditable. CodeMirror uses contenteditable on mobile because the browser itself handles the cursor.

What is possible though is to detect “Option+” and “Shift+Option+” so you could potentially have different keybindings just for iOS. It wold still be at the expense of some characters since “Option+” corresponds to an actual character, it’s just not widely used. What do you think?

[SirBryan]

It appears that UIKit and WebKit on iOS have finally gotten around to addressing this in the 12.1.3 betas (see
https://bugs.webkit.org/show_bug.cgi?id=149054). I for one am hoping to use CodeSandbox from either the home screen or inside of its own UIWebView to eliminate as much browser chrome as possible. (Unfortunately the Sign in via GitHub pops everything back out to Safari.)

[CompuIves]

!!!!! This is such good news! I'll do some testing as well!

Let's make a list of remaining things we can do to improve the iPad integration.

[aadsm]

@SirBryan this is great! thank you so much for raising, I didn't realize work has been done on it after I flagged it to the team on twitter :)).
I'm concerned with the last comment though, saying that cursor keys still don't work in textareas, which is really the main pain point for vscode :(. I've brought it up on twitter: https://twitter.com/aadsm/status/1081674544434139136?s=12

[lbogdan]

As I don't have an iPad, I didn't really follow the discussion here. Is this better with the recent switch to VSCode?

[aadsm]

It will actually make things worse because vscode uses a textarea and not a contenteditable element (which is able to manage its own cursor). We’ve been discussing this on vscode github’s issue.

On Tue, Mar 26, 2019 at 11:38 Bogdan Luca ***@***.***> wrote: As I don't have an iPad, I didn't really follow the discussion here. Is this better with the recent switch to VSCode? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#507 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAExKoz7eLNMZ5j6CsD5lu-INA03W8xUks5vamkOgaJpZM4R9bGu> .

-- Best regards, António Afonso

[SirBryan]

To summarize:

The problem is in UIKit (see https://bugs.webkit.org/show_bug.cgi?id=149054).

The Apple devs have commented that a radar has been opened for it, and that it "should have" been fixed (somewhere up the line), but it hasn't dropped in recent iOS releases (including 12.2).

Once the bug is fixed, arrow key up/down/left/right events will fire when an external keyboard is used and we'll be able to use Codesandbox on our iPads (esp. iPad Pro).

With a combination of Codesandbox, Now.sh, Github, and others, one could theoretically develop, test, build, and deploy from their iPad anywhere in the world without having to compile or transfer one bit of code to or from the tablet.

[aadsm]

@SirBryan Yes, but you can already do this today with other platforms like www.glitch.com.

[joshcoody]

I recently started trying to use this on my iPad Pro with the smart keyboard. After reading through this thread I was curious about the lack of key shortcuts, and tested it out on here: https://keycode.info/

All of the keys seem to exist and fire the same key codes as my desktop computer. Maybe that’s not the case for other external keyboards, but seems like it should be viable to support regular shortcuts. Or maybe give iPad users the ability to use the desktop interface with those shortcuts? Kind of like an opt-in if you don’t want to support nonstandard events?

[aadsm]

@joshcoody I use the smart keyboard as well, I don’t think there’s any difference from an external keyboard.
The reason why you can see all the events on that page is because the listener is probably on the document or on another element like a div.
The monaco editor is listening for events on the textarea (where the focus is) and on that element we don’t get all the events like we do on a div.

[joshcoody]

@aadsm Ah I see now, you’re right it doesn’t work inside a textarea. So other tools and sites that support the keyboard shortcuts are probably just listening on a wrapping div, or on the document as a whole... I understand why it’s an issue with UIKit now, that’s rather unfortunate...
Thanks for the clarification though!

[jpdriver]

sounds like this will start working correctly with Safari in the upcoming iPadOS 13 release

microsoft/monaco-editor#293 (comment)

[PilotConway]

Yeah, in my limited testing it is working in iPadOS 13.

[eastcoastcoder]

So arrow keys work fine and pasting (cmd + v) works fine but on the latest iOS 13 I’m actually having issues copying anything from codesandbox (cmd + c). So for example, if I have some code from one file I can’t copy it to another. Yet I can indeed copy with cmd + c from any other page and cmd + v will paste it in as expected. I’m not sure if this issue started in iOS13 or if it has been here from prior version but I’ve just noticed it recently.

Additionally I can select a line with cmd + right arrow but long pressing to hit copy briefly shows the “right-click” menu then disappears as the keyboard steals focus. So really there is no reliable copy and paste method on an iPad.

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

[github-actions]

This issue has been automatically closed because there wasn't any activity after the previous notice or the stale label wasn't removed.