str.isspace should use the Unicode White_Space property #62436
Comments
|
ASCII information separators, hex codes 1C through 1F are classified as space: >>> all(c.isspace() for c in '\N{FS}\N{GS}\N{RS}\N{US}')
Truebut int()/float() do not accept strings with leading or trailing separators: >>> int('123\N{RS}')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '123\x1e'This is probably because corresponding bytes values are not classified as whitespace: >>> any(c.encode().isspace() for c in '\N{FS}\N{GS}\N{RS}\N{US}')
False |
|
You stated facts: what is your proposal? The fact that unicode calls characters 'space' does not make then whitespace as commonly understood, or as defined by C, or even as defined by the Unicode database. Unicode apparently has a WSpace property. According to the table in So I think your implied proposal to treat them as whitespace (in strings but not bytes) should be rejected as invalid. For 3.x, the manual should specify that it follows the C definition of 'whitespace' (\r included) for bytes and the extended unicode definition for strings. >>> int('3\r')
3
>>> int('3\u00a0')
3
>>> int('3\u2000')
3
>>> int(b'3\r')
3
>>> int(b'3\u00a0')
Traceback (most recent call last):
File "<pyshell#10>", line 1, in <module>
int(b'3\u00a0')
ValueError: invalid literal for int() with base 10: '3\\u00a0' |
terryjreedy
added
type-feature
There is a bug somewhere. We cannot simultaneously have >>> '\N{RS}'.isspace()
Trueand not accept '\N{RS}' as whitespace when parsing numbers. I believe int(x) should be equivalent to int(x.strip()). This is not the case now: >>> '123\N{RS}'.strip()
'123'
>>> int('123\N{RS}')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '123\x1e'The reason I did not clearly state my proposal is because I am not sure whether bytes.isspace or str.isspace is correct, but I don't see any justification for having them defined differently in the ASCII range. |
|
I see your point now. Since RS is not whitespace by any definition I knew of previously, why is RS.isspace True? Apparent answer: Doc says '''Return true if there are only whitespace characters in the string and there is at least one character, false otherwise. Whitespace characters are those characters defined in the Unicode character database as “Other” or “Separator” and those with bidirectional property being one of “WS”, “B”, or “S”.''' I suspect this is a more expansive definition than WSpace chars, which seems to be the one used by int(), but you could check the int code. Bytes docs says: "Whenever a bytes or bytearray method needs to interpret the bytes as characters (e.g. the is...() methods, split(), strip()), the ASCII character set is assumed (text strings use Unicode semantics)." This says to me that str.isxxx and bytes.isxxx should match on ascii chars and not otherwise. That would happen is the bytes methods check for all ascii and decoded to unicode and used str method. Since they do not match, bytes must do something different. I think there is one definite bug: the discrepancy between str.isspace and bytes.isspace. There is possibly another bug: the discrepancy between 'whitespace' for str.isspace and int/float. After pinning down the details, I think you should ask how to resolve these on py-dev, and which versions to patch. |
terryjreedy
added
type-bug
|
It looks like str.isspace() is incorrect. The proper definition of unicode whitespace seems to include 26 characters: # ================================================ 0009..000D ; White_Space # Cc [5] <control-0009>..<control-000D> # Total code points: 26 http://www.unicode.org/Public/UNIDATA/PropList.txt Python's str.isspace() uses the following definition: "Whitespace characters are those characters defined in the Unicode character database as “Other” or “Separator” and those with bidirectional property being one of “WS”, “B”, or “S”." Information separators are swept in because they have bidirectional property "B": >>> unicodedata.bidirectional('\N{RS}')
'B'See also bpo-10587. |
|
I did a little more investigation and it looks like information separators have been included in whitespace since unicode type was first implemented in Python: guido 11967 Fri Mar 10 22:52:46 2000 +0000: /* Returns 1 for Unicode characters having the type 'WS', 'B' or 'S', (hg blame -u -d -n -r 11967 Objects/unicodectype.c) |
|
Martin v. Löwis wrote at bpo-13391 (msg147634):
I am adding Martin and Ezio to the "nosy." |
|
I stand by that comment: IsWhiteSpace should use the Unicode White_Space property. Since FS/GS/RS/US are not in the White_Space property, it's correct that the int conversion fails. It's incorrect that .isspace() gives true. There are really several bugs here:
I propose to fix this by parsing PropList.txt, and generating _PyUnicode_IsWhitespace based on the White_Space property. For efficiency, it should also generate a fast-lookup array for the ASCII case, or just use _Py_ctype_table (with a comment that this table needs to match PropList White_Space). _Py_ascii_whitespace should go. Contributions are welcome. |
|
I agree with Martin. At the time Unicode was added to Python, there was no single Unicode property for white space, so I had to deduce this from the other available properties. Now that we have a white space property in Unicode, we should start using it. Fortunately, the difference in Python's set of white space chars and the ones having the Unicode white space property are minimal. |
|
I have updated the title to focus this issue on the behavior of str.isspace(). I'll pick up remaining int/float issues in bpo-10581. |
abalkin
changed the title
|
I would like someone review this change: The patch can go in without this optimization, but I think this is the right first step towards removing _Py_ascii_whitespace. I don't think there is a need to generate ASCII optimization in makeunicodedata. While technically Unicode stability policy only guarantees that White_Space property will not be removed from code point s that have it, I think there is little chance that they will ever add White_Space property to another code point in the ASCII range and if they do, I am not sure Python will have to follow. |
abalkin
added
interpreter-core
|
-1 on that patch. It's using trickery to implement the test, and it's not clear that it is actually as efficient as the previous version. The previous version was explicitly modified to use a table lookup for performance reasons. I'd be fine with not generating PyUnicode_IsWhiteSpace at all, but instead hand-coding it. I suspect that we might want to use more of PropList at some point, so an effort to parse it might not be wasted. |
|
For future reference, the code discussed above is in the following portion of the patch: -#define Py_UNICODE_ISSPACE(ch) \
- ((ch) < 128U ? _Py_ascii_whitespace[(ch)] : _PyUnicode_IsWhitespace(ch))
+#define Py_UNICODE_ISSPACE(ch) \
+ ((ch) == ' ' || \
+ ((ch) < 128U ? (ch) - 0x9U < 5U : _PyUnicode_IsWhitespace(ch))) |
|
As uncovered in bpo-12855, str.splitlines() currently considers the FS, GS and RS (1C–1E), but not the US (1F), to be line breaks. It might be surprising if these are no longer considered white space but are still considered line breaks. |
|
I've gone and made a patch for this change: Most of the work happens in the script Tools/unicode/makeunicode.py , and along the way I made several changes there that I found made it somewhat nicer to work on, and I think will help other people reading that script too. So I'd like to try to merge those improvements first. I've filed bpo-37760 for those preparatory changes, and posted several PRs (GH-15128, #59334, #59335) as bite-sized pieces. These PRs can go in in any order. Please take a look! Reviews appreciated. |
|
Greg, could you try this code after your patch? >>> import re
>>> re.match(r'\s', '\x1e')
<re.Match object; span=(0, 1), match='\x1e'> # <- before patch |
|
Good question! With the patch:
In other words, the definition of the regexp r'\s' follows along. Good to know. |
Update:
It's a generally straightforward and boring change that converts the main data structures of makeunicodedata.py from using length-18 tuples as records to using a dataclass, which I think makes subsequent changes that add features to that script much easier both to write and to review.
|
|
I've done a ridiculous amount of research into this in the last week or two. At long last, I've written a script to produce a canonical answer. It parses the (XML version of the) Unicode database, producing a list of all whitespace and line-breaking whitespace characters. It then compares those lists to the list of characters the Python I found three places where there's disagreement. So far this issue has discussed two of those places:
Here's the third, which I don't think so far has been mentioned here:
I've attached my script here: |
|
How to attach a file? |
|
As mentioned a longer while ago (in 2013, some ten years ago), I believe that we should switch Python to stick to the Unicode definitions of whitespace and line breaks. The question is: How can we do this in a way which doesn't break too much code out there ? Larry's investigation shows that the "separator" code points are causing trouble. I wonder how often these are used in real-life data. Looking at the docs, \v and \f were added to the list of line breaks in 3.2 without causing major turmoil. It's likely that the even less commonly used separator codes won't cause much trouble either, so perhaps we could consider the fix a "bug fix".
|
"Attach files by dragging & dropping, selecting or pasting them." Drag & drop worked for me.
I wouldn't characterize it like that. I don't know if they're causing actual trouble for anybody, because I believe nobody uses those characters in real life. As far as I know, the only problem they cause is to make Python's Unicode object disobey the Unicode standard, in a way I believe is harmless for all practical purposes. But I also believe that fixing the Unicode object to bring it into spec will also be harmless, and we'll all sleep that much better at night ;-)
Do you mean, added to the list of line breaks for the By the way, in the intervening years since previous correspondents posted proposed patches, all those patch links have died. Bitbucket is long out of the Mercurial game, and clicking on gnprice's patch links go to 404 pages. |
|
By the way, I did find those .txt files, but I couldn't understand them. My script instead reads the XML version of the Unicode database, which has every annotation for every character. As long as you have a reasonable XML library it's dead easy to deal with. Whitespace is noted as |
|
I think the three issues that @larryhastings identified should be fixed. I think they're bugs, but I don't think they should be backported. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: