URL preview error in the log after upgrade

[Linuxine]

Hi,

I managed to upgrade to version 0.25 today, and it works just fine !

There seem to be an issue with the url preview, though : each time a user displays the preview in Riot, it seems, I get the following error in the log:

Traceback (most recent call last):
  File "/home/matrix/synapse/lib/python2.7/site-packages/synapse/http/server.py", line 118, in wrapped_request_handler
    yield request_handler(self, request)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/home/matrix/synapse/lib/python2.7/site-packages/synapse/rest/media/v1/preview_url_resource.py", line 151, in _async_render_GET
    respond_with_json_bytes(request, 200, og, send_cors=True)
  File "/home/matrix/synapse/lib/python2.7/site-packages/synapse/http/server.py", line 399, in respond_with_json_bytes
    request.write(json_bytes)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/web/server.py", line 234, in write
    http.Request.write(self, data)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/web/http.py", line 1086, in write
    self.channel.write(data)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/web/http.py", line 2307, in write
    self.transport.write(data)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/protocols/tls.py", line 458, in write
    raise TypeError("Must write bytes to a TLS transport, not unicode.")
TypeError: Must write bytes to a TLS transport, not unicode.

The preview seems to work anyway, but this is quite verbose : is there an issue with my configuration ?

[richvdh]

does this happen for all URLs or just a subset?

[Linuxine]

Hi, I have just looked into my logs to check. It does not seems to happen for all url; for example, according to my logs, it is triggered by some url preview of images stored locally on my server, but not everytime : I have just tried to trigger it by posting the url in a room, and no error is displayed...
In the logs, the error is displayed for url previews of very various sites, like twitter, screenshots.firefox.com, local content... the only similarity is that they all seems to be https url, if this can help.

[richvdh]

can you share examples of URLs that give the error?

[Linuxine]

Here is an example :

2017-12-01 14:43:02,331 - synapse.http.server - 139 - ERROR - GET-13883- Failed handle request synapse.rest.media.v1.preview_url_resource._async_render_GET on <synapse.rest.media.v1.preview_url_resource.PreviewUrlResource instance at 0x42b2e18>: <SynapseRequest at 0x6145950 method=GET uri=/_matrix/media/r0/preview_url?url=https%3A%2F%2Fgithub.com%2Fansible%2Fansible%2Fissues%2F22466&ts=1512124571222&access_token=<redacted> clientproto=HTTP/1.1 site=8448>
Traceback (most recent call last):
  File "/home/matrix/synapse/lib/python2.7/site-packages/synapse/http/server.py", line 118, in wrapped_request_handler
    yield request_handler(self, request)
TypeError: Must write bytes to a TLS transport, not unicode.

Or :

2017-12-18 10:16:15,546 - synapse.http.server - 139 - ERROR - GET-99961- Failed handle request synapse.rest.media.v1.preview_url_resource._async_render_GET on <synapse.rest.media.v1.preview_url_resource.PreviewUrlResource instance at 0x4306d88>: <SynapseRequest at 0x776ccb0 method=GET uri=/_matrix/media/r0/preview_url?url=http%3A%2F%2Fwww.liberation.fr%2Fplanete%2F2017%2F12%2F11%2Fviols-en-syrie-la-voix-des-femmes-pour-temoigner-de-l-enfer-du-regime_1616012&ts=1513357314538&access_token=<redacted> clientproto=HTTP/1.1 site=8448>
Traceback (most recent call last):
  File "/home/matrix/synapse/lib/python2.7/site-packages/synapse/http/server.py", line 118, in wrapped_request_handler
    yield request_handler(self, request)
TypeError: Must write bytes to a TLS transport, not unicode.

and also :

2018-05-17 15:07:07,031 - synapse.http.server - 89 - ERROR - GET-3211- Failed handle request via <function _async_render_GET at 0x4a2fe60>: <SynapseRequest at 0x5daa128 method=GET uri=/_matrix/media/r0/preview_url?url=https%3A%2F%2Flinuxine.net%2Ficons%2Ftroll.png&ts=1526562217628 clientproto=HTTP/1.1 site=8448>: Traceback (most recent call last):
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1532, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/home/matrix/synapse/lib/python2.7/site-packages/synapse/http/server.py", line 66, in wrapped_request_handler
    yield h(self, request)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1532, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/home/matrix/synapse/lib/python2.7/site-packages/synapse/rest/media/v1/preview_url_resource.py", line 160, in _async_render_GET
    respond_with_json_bytes(request, 200, og, send_cors=True)
  File "/home/matrix/synapse/lib/python2.7/site-packages/synapse/http/server.py", line 371, in respond_with_json_bytes
    request.write(json_bytes)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/web/server.py", line 236, in write
    http.Request.write(self, data)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/web/http.py", line 1112, in write
    self.channel.write(data)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/web/http.py", line 2352, in write
    self.transport.write(data)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/protocols/tls.py", line 461, in write
    raise TypeError("Must write bytes to a TLS transport, not unicode.")
exceptions.TypeError: Must write bytes to a TLS transport, not unicode.

I don't know if this is related, but more recently (since the opening of this bug) I also get this new error message about the url preview:

2018-05-17 15:28:09,134 - synapse.http.server - 89 - ERROR - GET-5919- Failed handle request via <function _async_render_GET at 0x4a2fe60>: <XForwardedForRequest at 0x5477e18 method=GET uri=/_matrix/media/r0/preview_url?url=https%3A%2F%2Flinuxine.net%2Ficons%2Ftroll.png&ts=1526562217628 clientproto=HTTP/1.1 site=8008>: Traceback (most recent call last):
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1532, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/home/matrix/synapse/lib/python2.7/site-packages/synapse/http/server.py", line 66, in wrapped_request_handler
    yield h(self, request)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1532, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
    result = g.send(result)
  File "/home/matrix/synapse/lib/python2.7/site-packages/synapse/rest/media/v1/preview_url_resource.py", line 160, in _async_render_GET
    respond_with_json_bytes(request, 200, og, send_cors=True)
  File "/home/matrix/synapse/lib/python2.7/site-packages/synapse/http/server.py", line 371, in respond_with_json_bytes
    request.write(json_bytes)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/web/server.py", line 236, in write
    http.Request.write(self, data)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/web/http.py", line 1112, in write
    self.channel.write(data)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/web/http.py", line 2352, in write
    self.transport.write(data)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/internet/_newtls.py", line 191, in write
    FileDescriptor.write(self, bytes)
  File "/home/matrix/synapse/lib/python2.7/site-packages/twisted/internet/abstract.py", line 349, in write
    raise TypeError("Data must not be unicode")

[richvdh]

can you share examples of URLs that give the error?

2018-05-17 15:07:07,031 - synapse.http.server - 89 - ERROR - GET-3211- Failed handle request via <function _async_render_GET at 0x4a2fe60>: <SynapseRequest at 0x5daa128 method=GET uri=/_matrix/media/r0/preview_url?url=https%3A%2F%2Flinuxine.net%2Ficons%2Ftroll.png&ts=1526562217628 clientproto=HTTP/1.1 site=8448>: Traceback (most recent call last):

Right, so the URL in question was https://linuxine.net/icons/troll.png, which seems to work just fine for me, so I have no idea what's going on here.

Just to check: what version of synapse are you on now? if it's anything older than 0.29.0, please do upgrade.

[Linuxine]

Yes, what a marvelous url isn't it ? 😄
My current version of synapse is 0.29.0 indeed.

[kyrias]

Just started noticing the same error with e.g. https://github.com/matrix-org/synapse/releases/tag/v0.31.1 while running 0.31.1.

[hawkowl]

This ought to be fixed by #4157 .