readline: Improve Unicode handling #25723

Avi-D-coder · 2019-01-26T07:12:02Z

Makes delete and backspace work upon unicode code points instead of UTF-16 code units.
Prevents moving left or right from placing the cursor in between code units comprising a code point.

In plain english: Deleting and moving the cursor across emoji works now, but grapheme clusters/characters like 👨‍👩‍👧‍👦 still have trailing blank spaces.

Ideally grapheme clusters would be used instead of code points, but until the proposal-intl-segmenter lands node seems to be without built in support for handling them.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
commit message follows commit guidelines

Fixes: nodejs#25693

BridgeAR · 2019-01-27T21:08:44Z

@Avi-D-coder thanks a lot! Would you be so kind to add further tests that verify the correct positions for these two cases:

1. Insert an astral character
2. Press right
3. Add another astral character
1. Insert an astral character
2. Press left
3. Add another astral character

I believe some of these cases could corrupt the input when looking at the code (AFAIK all String#slice commands need the character.length and not the actual cursor position to work properly. If I'm correct we'll likely have to track two things: a) the cursor position and b) the actual character length offset).
[Update] This was a wrong assumption since the cursor handling is still the same as before. [/Update]
[Update2] See #25723 (comment) for further details for the original issue. [/Update]

Please also use the following helper function to retrieve the number of characters in an string:

function characters(str) {
  let count = 0;
  // eslint-disable-next-line no-unused-vars
  for (const char of str) {
    count++;
  }
  return count;
}

In my mini benchmark this function outperforms Array.from(c).length by 20x.

Avi-D-coder · 2019-01-27T22:18:12Z

@BridgeAR Will do

BridgeAR · 2019-01-27T23:02:16Z

@Avi-D-coder you could also calculate the correct length offset with the cursor instead of tracking it as well. That would probably be easier (just a bit more CPU expensive but it should not hurt all that much).

function cursorToLength(cursor, str) {
  let len = 0;
  for (const char of str) {
    len += char.length;
    if (--cursor === 0)
      break;
  }
  return len;
}

This function could be used in all places that use the cursor position to slice something or that is compared to the line length.

Please also note that all occurrences of e.g. this.cursor = this.line.length also have to be updated (and they are almost everywhere).

Avi-D-coder · 2019-01-29T10:03:51Z

@BridgeAR I added the tests and optimizations. I was not able to produce corrupt characters on this commit or the previous.

I renamed your characters function to codePointLen, since the concept of a character is not very well defined and tends to create issues (see the lsp range debacle).

I'm not entirely sure why cursorToLength is needed, but codePointLeftOf and codePointRightOf are I think a similar optimization.

BridgeAR

This is looking really good. I just left a few comments and ctrl + b and ctrl + f should also be changed accordingly.

lib/readline.js

Avi-D-coder · 2019-02-06T07:54:25Z

@BridgeAR I included your code.

BridgeAR · 2019-02-06T14:55:19Z

LGTM, just ctrl + b (back one character) and ctrl + f (forward one character) should also be changed accordingly (we could do that in a separate PR but it feels natural to keep the functionality aligned).

node/lib/readline.js

Lines 836 to 837 in 91adbe1

    
           case 'b': // back one character 
        
             this._moveCursor(-1);

node/lib/readline.js

Lines 840 to 841 in 91adbe1

    
           case 'f': // forward one character 
        
             this._moveCursor(+1);

Avi-D-coder · 2019-02-07T05:01:39Z

@BridgeAR my bad. I could have sworn I adjusted ctrl + f/b

lib/readline.js

BridgeAR · 2019-02-07T16:01:40Z

@Avi-D-coder thanks a lot for sticking to it! I just found the issue that confused me originally and it seems to be the last obstacle. Thanks again for reacting so quickly to everything! ❤️

lib/readline.js

BridgeAR · 2019-02-07T17:55:02Z

CI https://ci.nodejs.org/job/node-test-pull-request/20638/

Avi-D-coder · 2019-02-08T22:49:50Z

Anyone know the reason for the inconsistent ci?

lpinca · 2019-02-09T14:14:09Z

Resume CI: https://ci.nodejs.org/job/node-test-pull-request/20684/

Avi-D-coder · 2019-02-15T22:06:05Z

Do any changes need to be made? Why are those two ci environments failing?

addaleax · 2019-02-17T17:53:24Z

@Avi-D-coder It looks like this is failing tests when Node.js is compiled with ./configure --without-intl?

https://ci.nodejs.org/job/node-test-commit-linux-containered/10565/nodes=ubuntu1604_sharedlibs_withoutintl_x64/testReport/junit/(root)/test/parallel_test_readline_interface/

assert.js:86
  throw new AssertionError(obj);
  ^

AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

1 !== 2

    at /home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-readline-interface.js:674:14
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-readline-interface.js:87:17)
    at Module._compile (internal/modules/cjs/loader.js:735:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:746:10)
    at Module.load (internal/modules/cjs/loader.js:627:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:570:12)
    at Function.Module._load (internal/modules/cjs/loader.js:562:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:798:12)
    at internal/main/run_main_module.js:29:11

Avi-D-coder · 2019-02-17T22:15:52Z

@addaleax This fix requires node to correctly handle unicode. I am not exactly sure what --without-intl does, but if no internationalization means not supporting unicode then these tests should not be run for --without-intl builds.

How can these tests be blacklisted for --without-intl builds.

richardlau · 2019-02-17T23:13:28Z

@addaleax This fix requires node to correctly handle unicode. I am not exactly sure what --without-intl does, but if no internationalization means not supporting unicode then these tests should not be run for --without-intl builds.

How can these tests be blacklisted for --without-intl builds.

common.hasIntl.

addaleax · 2019-02-19T16:04:48Z

CI: https://ci.nodejs.org/job/node-test-pull-request/20882/

Avi-D-coder · 2019-02-19T23:32:42Z

The --without-intl build pases tests on my machine now. Could someone CI this again.

vsemozhetbyt · 2019-02-20T06:08:58Z

CI: https://ci.nodejs.org/job/node-test-pull-request/20901/

lib/readline.js

richardlau · 2019-02-22T03:57:50Z

CI: https://ci.nodejs.org/job/node-test-pull-request/20957/

Avi-D-coder · 2019-02-23T20:59:51Z

The failing tests are under a security embargo. What is the next step here?

richardlau · 2019-02-24T04:02:50Z

The failing tests are under a security embargo. What is the next step here?

Waiting until the CI is unlocked after the security releases are done.

BridgeAR · 2019-03-01T12:52:15Z

Resumed CI https://ci.nodejs.org/job/node-test-pull-request/21093/ ✅ (besides Windows fanned).

Rebuild Windows fanned: https://ci.nodejs.org/job/node-test-commit-windows-fanned/25073/ ✔️

Prevents moving left or right from placing the cursor in between code units comprising a code point. PR-URL: nodejs#25723 Fixes: nodejs#25693 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]>

BridgeAR · 2019-03-01T15:01:42Z

Landed in fedc31b 🎉

@Avi-D-coder congratulations on your first commit to Node.js and a big thanks for sticking to it! ❤️ This was great work.

Avi-D-coder · 2019-03-01T20:30:27Z

@BridgeAR thanks for all your help.

Prevents moving left or right from placing the cursor in between code units comprising a code point. PR-URL: #25723 Fixes: #25693 Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]>

nodejs-github-bot added the readline Issues and PRs related to the built-in readline module. label Jan 26, 2019

readline: improve Unicode handling

26f5fc5

Fixes: nodejs#25693

Avi-D-coder force-pushed the readline-unicode branch from acc443b to 26f5fc5 Compare January 26, 2019 07:19

addaleax approved these changes Jan 27, 2019

View reviewed changes

readline: improve Unicode handling performance

c0c7f58

jasnell approved these changes Jan 29, 2019

View reviewed changes

BridgeAR reviewed Feb 5, 2019

View reviewed changes

lib/readline.js Show resolved Hide resolved

lib/readline.js Show resolved Hide resolved

lib/readline.js Outdated Show resolved Hide resolved

readline: use built in codePointAt

756ebea

readline: fix ctrl + f/b Unicode handling

200560c

BridgeAR reviewed Feb 7, 2019

View reviewed changes

lib/readline.js Outdated Show resolved Hide resolved

BridgeAR approved these changes Feb 7, 2019

View reviewed changes

lib/readline.js Outdated Show resolved Hide resolved

readline: remove redundant codePointLen function

0639c1b

Avi-D-coder force-pushed the readline-unicode branch from 1cc5b79 to 0639c1b Compare February 7, 2019 17:52

BridgeAR added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Feb 7, 2019

addaleax removed the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Feb 17, 2019

Avi-D-coder force-pushed the readline-unicode branch from a0287dc to f4635bd Compare February 19, 2019 21:08

readline: fix tests failing without internationalization

1b9ea93

Avi-D-coder force-pushed the readline-unicode branch from f4635bd to 1b9ea93 Compare February 19, 2019 23:06

BridgeAR reviewed Feb 20, 2019

View reviewed changes

lib/readline.js Outdated Show resolved Hide resolved

readline: handle 𐀀 character correctly

e649085

Avi-D-coder force-pushed the readline-unicode branch from 2cff8e4 to e649085 Compare February 22, 2019 03:42

BridgeAR added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Feb 26, 2019

BridgeAR closed this Mar 1, 2019

BridgeAR mentioned this pull request Mar 4, 2019

v11.11.0 proposal #26322

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

readline: Improve Unicode handling #25723

readline: Improve Unicode handling #25723

Avi-D-coder commented Jan 26, 2019

BridgeAR commented Jan 27, 2019 •

edited

Loading

Avi-D-coder commented Jan 27, 2019

BridgeAR commented Jan 27, 2019 •

edited

Loading

Avi-D-coder commented Jan 29, 2019 •

edited

Loading

BridgeAR left a comment

Avi-D-coder commented Feb 6, 2019

BridgeAR commented Feb 6, 2019

Avi-D-coder commented Feb 7, 2019

BridgeAR commented Feb 7, 2019 •

edited

Loading

BridgeAR commented Feb 7, 2019

Avi-D-coder commented Feb 8, 2019 •

edited

Loading

lpinca commented Feb 9, 2019

Avi-D-coder commented Feb 15, 2019

addaleax commented Feb 17, 2019

Avi-D-coder commented Feb 17, 2019

richardlau commented Feb 17, 2019

addaleax commented Feb 19, 2019

Avi-D-coder commented Feb 19, 2019

vsemozhetbyt commented Feb 20, 2019

richardlau commented Feb 22, 2019

Avi-D-coder commented Feb 23, 2019

richardlau commented Feb 24, 2019

BridgeAR commented Mar 1, 2019 •

edited

Loading

BridgeAR commented Mar 1, 2019

Avi-D-coder commented Mar 1, 2019

readline: Improve Unicode handling #25723

readline: Improve Unicode handling #25723

Conversation

Avi-D-coder commented Jan 26, 2019

Checklist

BridgeAR commented Jan 27, 2019 • edited Loading

Avi-D-coder commented Jan 27, 2019

BridgeAR commented Jan 27, 2019 • edited Loading

Avi-D-coder commented Jan 29, 2019 • edited Loading

BridgeAR left a comment

Choose a reason for hiding this comment

Avi-D-coder commented Feb 6, 2019

BridgeAR commented Feb 6, 2019

Avi-D-coder commented Feb 7, 2019

BridgeAR commented Feb 7, 2019 • edited Loading

BridgeAR commented Feb 7, 2019

Avi-D-coder commented Feb 8, 2019 • edited Loading

lpinca commented Feb 9, 2019

Avi-D-coder commented Feb 15, 2019

addaleax commented Feb 17, 2019

Avi-D-coder commented Feb 17, 2019

richardlau commented Feb 17, 2019

addaleax commented Feb 19, 2019

Avi-D-coder commented Feb 19, 2019

vsemozhetbyt commented Feb 20, 2019

richardlau commented Feb 22, 2019

Avi-D-coder commented Feb 23, 2019

richardlau commented Feb 24, 2019

BridgeAR commented Mar 1, 2019 • edited Loading

BridgeAR commented Mar 1, 2019

Avi-D-coder commented Mar 1, 2019

BridgeAR commented Jan 27, 2019 •

edited

Loading

BridgeAR commented Jan 27, 2019 •

edited

Loading

Avi-D-coder commented Jan 29, 2019 •

edited

Loading

BridgeAR commented Feb 7, 2019 •

edited

Loading

Avi-D-coder commented Feb 8, 2019 •

edited

Loading

BridgeAR commented Mar 1, 2019 •

edited

Loading