Error running pip install -r <requirements file>

[miloandmilk]

Python 3.4 / Windows 7 / Powershell / pip version 7.0.3

Running

pip install -r .\stable-req.txt

with stable-req.txt containing requests==2.7.0

results in the following exception:

Traceback (most recent call last):
  File "D:\frh\env2\lib\site-packages\pip\basecommand.py", line 223, in main
    status = self.run(options, args)
  File "D:\frh\env2\lib\site-packages\pip\commands\install.py", line 268, in run
    wheel_cache
  File "D:\frh\env2\lib\site-packages\pip\basecommand.py", line 287, in populate_requirement_set
    wheel_cache=wheel_cache):
  File "D:\frh\env2\lib\site-packages\pip\req\req_file.py", line 86, in parse_requirements
    for req in req_iter:
  File "D:\frh\env2\lib\site-packages\pip\req\req_file.py", line 130, in process_line
    wheel_cache=wheel_cache
  File "D:\frh\env2\lib\site-packages\pip\req\req_install.py", line 207, in from_line
    wheel_cache=wheel_cache)
  File "D:\frh\env2\lib\site-packages\pip\req\req_install.py", line 66, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "D:\frh\env2\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2960, in parse
    reqs = list(parse_requirements(s))
  File "D:\frh\env2\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2904, in parse_requirements
    "version spec")
  File "D:\frh\env2\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2869, in scan_list
    raise ValueError(msg, line, "at", line[p:])
ValueError: ('Expected version spec in', 'ÿþr\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00'
, 'at', '\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00')

Reproduced on two pcs and one Windows Server 2008 r2

[rbtcollins]

looks like a big pile of encoding confusion. Is that UCS32 perhaps?

[xavfernandez]

>>> u"ÿþr\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00".encode('utf8')
b'\xc3\xbf\xc3\xber\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00'
>>> u"ÿþr\x00e\x00q\x00u\x00e\x00s\x00t\x00s\x00=\x00=\x002\x00.\x007\x00.\x000\x00".encode('utf8').decode('utf16')
'뿃뻃requests==2.7.0'

and ÿþ seems to corresponds to the UTF-16 (LE) BOM (cf https://en.wikipedia.org/wiki/Byte_order_mark#Representations_of_byte_order_marks_by_encoding), which would match.

So I'd guess the req file was encoded in utf16 but python opened it as utf8 (returned from locale.getpreferredencoding() if I understand correctly https://docs.python.org/3.4/library/functions.html#open).

Not sure what pip could do to improve the user experience on such case ...

[pfmoore]

Yeah, that's a UTF-16 file. Was it created by redirection in Powershell (some_cmd > stable-req.txt) ? As that is typically where I end up with UTF-16 files that I didn't want ;-)

Auto-detecting (guessing) encodings is generally a bad idea, and probably something pip shouldn't get into, but I guess that looking for a BOM when reading the requirements file, and choosing the encoding from that if it's found, would be a useful feature to help with this common Windows issue. (Although it's mainly a Windows issue, the BOM detection can be added for all platforms, as I can't see how it would cause a problem on any other platform).

[xavfernandez]

Duplicate of #733