url preview fails with exceptions.TypeError: Data must not be unicode

[johnfoo]

when sending a message containing an URL to preview, the previewing process fails with the following stack trace

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1532, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
result = g.send(result)
File "/usr/local/lib/python2.7/dist-packages/synapse/http/server.py", line 119, in wrapped_request_handler
yield request_handler(self, request)
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1532, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- ---
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
result = g.send(result)
File "/usr/local/lib/python2.7/dist-packages/synapse/rest/media/v1/preview_url_resource.py", line 155, in _async_render_GET
respond_with_json_bytes(request, 200, og, send_cors=True)
File "/usr/local/lib/python2.7/dist-packages/synapse/http/server.py", line 405, in respond_with_json_bytes
request.write(json_bytes)
File "/usr/local/lib/python2.7/dist-packages/twisted/web/server.py", line 234, in write
http.Request.write(self, data)
File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 1086, in write
self.channel.write(data)
File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 2307, in write
self.transport.write(data)
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/_newtls.py", line 191, in write
FileDescriptor.write(self, bytes)
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/abstract.py", line 352, in write
raise TypeError("Data must not be unicode")
exceptions.TypeError: Data must not be unicode

the issue stems from http/server.py, twisted does not seem to appreciate to receive improperly encoded data. simply encoding in utf8 the json_bytes fixes the problem

--versions
environment: docker (from silvio/docker-matrix)
synapse: master (3d5a254)
python: 2.7.13
twisted: 17.9.0

created PR #2916 with the fix

[richvdh]

suspect this is closely related to #2891.

[hawkowl]

This ought to be fixed by #4157 .