UnicodeDecodeError when _tty_in=True and output is stdout

[brainpower]

After updating to 1.12.6 I get sometimes the following error when running commands:

Exception in thread STDOUT/ERR thread for pid 2123:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.5/site-packages/sh.py", line 2243, in output_thread
    done = stream.read()
  File "/usr/lib/python3.5/site-packages/sh.py", line 2688, in read
    self.write_chunk(chunk)
  File "/usr/lib/python3.5/site-packages/sh.py", line 2663, in write_chunk
    self.should_quit = self.process_chunk(chunk)
  File "/usr/lib/python3.5/site-packages/sh.py", line 2565, in process
    handler.write(encode(chunk))
  File "/usr/lib/python3.5/site-packages/sh.py", line 2558, in 
    encode = lambda chunk: chunk.decode(handler.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 0: unexpected end of data

I get the same error with python 2.7.

The error also only appears when _in_tty=True is set.
The following simple script triggers this:

#!/usr/bin/python

import sys
from sh import echo

echo("äääääääääääääääääääääääääääääääääääääääääää", _out=sys.stdout, _out_bufsize=0, _tty_in=True)

Maybe the splitting issue described in the following answer is the reason for the error? The name 'chunk' suggests that the string gets split, after all...
http://stackoverflow.com/a/24004343/1563584

[amoffat]

Thanks for reporting. There were actually a couple of bugs knotted up into the behavior you were seeing. One problem, from a user's perspective, is that you shouldn't be able to set _out_bufsize to a value and set _out to be a pipe or tty. The reason is that pipes and ttys already have their own buffering that you have no control over, so sh saying "apply some buffering" is more or less useless. The read ends of pipes and ttys will receive data according to your OS buffering.

I've added an argument validator to prevent people from setting both options at once.

The other issue that you helped discover was an internal issue where some internal processing threads (the ones that did the chunk splitting) were running they shouldn't have been running. So I pushed up a fix for that.

Version 1.12.7 is live and has those fixes.

The thing to keep in mind though in the future is that if you are using _out targets that can be controlled with _out_bufsize buffering, like functions or queues, etc, then you should be careful of your encoding, or you can still end up trying to decode split bytes, which will raise the error you're seeing.

[brainpower]

Thanks for the fix and the background information.