URL linkification includes unicode punycode which causes issues with twitter previews

[HarHarLinks]

Steps to reproduce

1. Post a twitter URL into a room, URL previews enabled
2. Twitter post content is put into double quotes ""
3. in case of media-only posts, the t.co media URL is in quotes
4. element includes the ending quote in the URL, resulting in a broken link

Outcome

What did you expect?

links work

What happened instead?

Demo URLs for reference:
https://twitter.com/FXNetworks/status/1577704289476128771
https://twitter.com/mischiefanimals/status/1576904037449969664

Operating system

arch

Application version

Element Nightly version: 2022100501 Olm version: 3.2.12

How did you install the app?

aur

Homeserver

No response

Will you send logs?

No

[t3chguy]

The quote there is “ or 5o0a in punycode. It is valid as part of URLs.
This is a bug with Synapse or the OGP tags in Twitter. If you think the linkification should ignore it then I suggest an issue on LinkifyJS

To confirm, this issue is not with double quotes, " are unaffected. https://www.compart.com/en/unicode/U+201C however is affected.

[t3chguy]

t3chguy@Michael-t3chguy-MBP ~ [127]> curl "https://publish.twitter.com/oembed?url=https://twitter.com/mischiefanimals/status/1576904037449969664" | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   853  100   853    0     0   3946      0 --:--:-- --:--:-- --:--:--  4140
{
  "url": "https://twitter.com/mischiefanimals/status/1576904037449969664",
  "author_name": "animals going goblin mode",
  "author_url": "https://twitter.com/mischiefanimals",
  "html": "<blockquote class=\"twitter-tweet\"><p lang=\"zxx\" dir=\"ltr\"><a href=\"https://t.co/fVP8YWHS2j\">pic.twitter.com/fVP8YWHS2j</a></p>&mdash; animals going goblin mode (@mischiefanimals) <a href=\"https://twitter.com/mischiefanimals/status/1576904037449969664?ref_src=twsrc%5Etfw\">October 3, 2022</a></blockquote>\n<script async src=\"https://platform.twitter.com/widgets.js\" charset=\"utf-8\"></script>\n",
  "width": 550,
  "height": null,
  "type": "rich",
  "cache_age": "3153600000",
  "provider_name": "Twitter",
  "provider_url": "https://twitter.com",
  "version": "1.0"
}

Looks like Synapse is injecting the quotes - I suggest opening an issue with https://github.com/matrix-org/synapse/