-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
/
Copy pathpep-3131.txt
280 lines (207 loc) · 10.9 KB
/
pep-3131.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
PEP: 3131
Title: Supporting Non-ASCII Identifiers
Version: $Revision$
Last-Modified: $Date$
Author: Martin von Löwis <[email protected]>
Status: Final
Type: Standards Track
Content-Type: text/x-rst
Created: 01-May-2007
Python-Version: 3.0
Post-History:
Abstract
========
This PEP suggests to support non-ASCII letters (such as accented characters,
Cyrillic, Greek, Kanji, etc.) in Python identifiers.
Rationale
=========
Python code is written by many people in the world who are not
familiar with the English language, or even well-acquainted with the
Latin writing system. Such developers often desire to define classes
and functions with names in their native languages, rather than having
to come up with an (often incorrect) English translation of the
concept they want to name. By using identifiers in their native
language, code clarity and maintainability of the code among
speakers of that language improves.
For some languages, common transliteration systems exist (in particular, for the
Latin-based writing systems). For other languages, users have larger
difficulties to use Latin to write their native words.
Common Objections
=================
Some objections are often raised against proposals similar to this one.
People claim that they will not be able to use a library if to do so they have
to use characters they cannot type on their keyboards. However, it is the
choice of the designer of the library to decide on various constraints for using
the library: people may not be able to use the library because they cannot get
physical access to the source code (because it is not published), or because
licensing prohibits usage, or because the documentation is in a language they
cannot understand. A developer wishing to make a library widely available needs
to make a number of explicit choices (such as publication, licensing, language
of documentation, and language of identifiers). It should always be the choice
of the author to make these decisions - not the choice of the language
designers.
In particular, projects wishing to have wide usage probably might want to
establish a policy that all identifiers, comments, and documentation is written
in English (see the GNU coding style guide for an example of such a policy).
Restricting the language to ASCII-only identifiers does not enforce comments and
documentation to be English, or the identifiers actually to be English words, so
an additional policy is necessary, anyway.
Specification of Language Changes
=================================
The syntax of identifiers in Python will be based on the Unicode standard annex
UAX-31 [1]_, with elaboration and changes as defined below.
Within the ASCII range (U+0001..U+007F), the valid characters for identifiers
are the same as in Python 2.5. This specification only introduces additional
characters from outside the ASCII range. For other characters, the
classification uses the version of the Unicode Character Database as included in
the ``unicodedata`` module.
The identifier syntax is ``<XID_Start> <XID_Continue>*``.
The exact specification of what characters have the XID_Start or
XID_Continue properties can be found in the DerivedCoreProperties
file of the Unicode data in use by Python (4.1 at the time this
PEP was written), see [6]_. For reference, the construction rules
for these sets are given below. The XID_* properties are derived
from ID_Start/ID_Continue, which are derived themselves.
``ID_Start`` is defined as all characters having one of the general
categories uppercase letters (Lu), lowercase letters (Ll), titlecase
letters (Lt), modifier letters (Lm), other letters (Lo), letter
numbers (Nl), the underscore, and characters carrying the
Other_ID_Start property. ``XID_Start`` then closes this set under
normalization, by removing all characters whose NFKC normalization
is not of the form ID_Start ID_Continue* anymore.
``ID_Continue`` is defined as all characters in ``ID_Start``, plus
nonspacing marks (Mn), spacing combining marks (Mc), decimal number
(Nd), connector punctuations (Pc), and characters carrying the
Other_ID_Continue property. Again, ``XID_Continue`` closes this set
under NFKC-normalization; it also adds U+00B7 to support Catalan.
All identifiers are converted into the normal form NFKC while parsing;
comparison of identifiers is based on NFKC.
A non-normative HTML file listing all valid identifier characters for
Unicode 4.1 can be found at
http://www.dcl.hpi.uni-potsdam.de/home/loewis/table-3131.html.
Policy Specification
====================
As an addition to the Python Coding style, the following policy is
prescribed: All identifiers in the Python standard library MUST use
ASCII-only identifiers, and SHOULD use English words wherever feasible
(in many cases, abbreviations and technical terms are used which
aren't English). In addition, string literals and comments must also
be in ASCII. The only exceptions are (a) test cases testing the
non-ASCII features, and (b) names of authors. Authors whose names are
not based on the Latin alphabet MUST provide a Latin transliteration
of their names.
As an option, this specification can be applied to Python 2.x. In
that case, ASCII-only identifiers would continue to be represented as
byte string objects in namespace dictionaries; identifiers with
non-ASCII characters would be represented as Unicode strings.
Implementation
==============
The following changes will need to be made to the parser:
1. If a non-ASCII character is found in the UTF-8 representation of
the source code, a forward scan is made to find the first ASCII
non-identifier character (e.g. a space or punctuation character)
2. The entire UTF-8 string is passed to a function to normalize the
string to NFKC, and then verify that it follows the identifier
syntax. No such callout is made for pure-ASCII identifiers, which
continue to be parsed the way they are today. The Unicode database
must start including the Other_ID_{Start|Continue} property.
3. If this specification is implemented for 2.x, reflective libraries
(such as pydoc) must be verified to continue to work when Unicode
strings appear in ``__dict__`` slots as keys.
Open Issues
===========
John Nagle suggested consideration of Unicode Technical Standard #39,
[2]_, which discusses security mechanisms for Unicode identifiers.
It's not clear how that can precisely apply to this PEP; possible
consequences are
* warn about characters listed as "restricted" in xidmodifications.txt
* warn about identifiers using mixed scripts
* somehow perform Confusable Detection
In the latter two approaches, it's not clear how precisely the
algorithm should work. For mixed scripts, certain kinds of mixing
should probably allowed - are these the "Common" and "Inherited"
scripts mentioned in section 5? For Confusable Detection, it seems one
needs two identifiers to compare them for confusion - is it possible
to somehow apply it to a single identifier only, and warn?
In follow-up discussion, it turns out that John Nagle actually
meant to suggest UTR#36, level "Highly Restrictive", [3]_.
Several people suggested to allow and ignore formatting control
characters (general category Cf), as is done in Java, JavaScript, and
C#. It's not clear whether this would improve things (it might
for RTL languages); if there is a need, these can be added
later.
Some people would like to see an option on selecting support
for this PEP at run-time; opinions vary on what precisely
that option should be, and what precisely its default value
should be. Guido van Rossum commented in [5]_ that a global
flag passed to the interpreter is not acceptable, as it would
apply to all modules.
Discussion
==========
Ka-Ping Yee summarizes discussion and further objection
in [4]_ as such:
A. Should identifiers be allowed to contain any Unicode letter?
Drawbacks of allowing non-ASCII identifiers wholesale:
1. Python will lose the ability to make a reliable round trip to
a human-readable display on screen or on paper.
2. Python will become vulnerable to a new class of security exploits;
code and submitted patches will be much harder to inspect.
3. Humans will no longer be able to validate Python syntax.
4. Unicode is young; its problems are not yet well understood and
solved; tool support is weak.
5. Languages with non-ASCII identifiers use different character sets
and normalization schemes; :pep:`3131`'s choices are non-obvious.
6. The Unicode bidi algorithm yields an extremely confusing display
order for RTL text when digits or operators are nearby.
B. Should the default behaviour accept only ASCII identifiers, or
should it accept identifiers containing non-ASCII characters?
Arguments for ASCII only by default:
1. Non-ASCII identifiers by default makes common practice/assumptions
subtly/unknowingly wrong; rarely wrong is worse than obviously wrong.
2. Better to raise a warning than to fail silently when encountering
a probably unexpected situation.
3. All of current usage is ASCII-only; the vast majority of future
usage will be ASCII-only.
3. It is the pockets of Unicode adoption that are parochial, not the
ASCII advocates.
4. Python should audit for ASCII-only identifiers for the same
reasons that it audits for tab-space consistency
5. Incremental change is safer.
6. An ASCII-only default favors open-source development and sharing
of source code.
7. Existing projects won't have to waste any brainpower worrying
about the implications of Unicode identifiers.
C. Should non-ASCII identifiers be optional?
Various voices in support of a flag (although there's been debate
over which should be the default, no one seems to be saying that
there shouldn't be an off switch)
D. Should the identifier character set be configurable?
Various voices proposing and supporting a selectable character set,
so that users can get all the benefits of using their own language
without the drawbacks of confusable/unfamiliar characters
E. Which identifier characters should be allowed?
1. What to do about bidi format control characters?
2. What about other ID_Continue characters? What about characters
that look like punctuation? What about other recommendations
in UTS #39? What about mixed-script identifiers?
F. Which normalization form should be used, NFC or NFKC?
G. Should source code be required to be in normalized form?
References
==========
.. [1] http://www.unicode.org/reports/tr31/
.. [2] http://www.unicode.org/reports/tr39/
.. [3] http://www.unicode.org/reports/tr36/
.. [4] https://mail.python.org/pipermail/python-3000/2007-June/008161.html
.. [5] https://mail.python.org/pipermail/python-3000/2007-May/007925.html
.. [6] http://www.unicode.org/Public/4.1.0/ucd/DerivedCoreProperties.txt
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End: