Fix IP URL previews on Python 3 #4215
Conversation
|
Initial fix for #4208 . Needs more tests, mostly around the moved code regarding IP blacklisting. |
Codecov Report
@@ Coverage Diff @@
## develop #4215 +/- ##
===========================================
+ Coverage 73.54% 73.59% +0.04%
===========================================
Files 302 302
Lines 29894 29924 +30
Branches 4891 4898 +7
===========================================
+ Hits 21987 22023 +36
+ Misses 6470 6463 -7
- Partials 1437 1438 +1
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
It is not a given that getaddrinfo's results are cached - in general each separate call can result in a separate request to the DNS server. I'm pretty sure that with a short TTL and by rapidly switching an A record between internal and external addresses, you could get access to a blacklisted IP address in not-very-many attempts, so I'm really quite keen that we find a solution here that doesn't involve two separate DNS lookups.
hawkowl
force-pushed
the
hawkowl/ip-preview
branch
from
1d6352d to
fc1dd48
Compare
|
Further looks at the IP blacklist to be done in #4242 -- reducing the scope of this PR to just the listed bug. |
synapse/config/repository.py
Outdated
| @@ -151,6 +155,12 @@ def read_config(self, config): | |||
| except ImportError: | |||
| raise ConfigError(MISSING_LXML) | |||
|
|
|||
| try: | |||
| import hyperlink | |||
There was a problem hiding this comment.
we'll need to document this in the README (grep for lxml - there's more than one place) as well as the upgrade notes, and put something in the changelog about it, to stop people getting surprised by it suddenly not working.
There was a problem hiding this comment.
This should only affect people on Twisteds older than 17.5, which hard depends on hyperlink. (I guess this affects xenial and so?) Maybe I should add it to our pip dependencies?
| ) | ||
| request_deferred = timeout_deferred( | ||
| request_deferred, 60, self.hs.get_reactor(), |
There was a problem hiding this comment.
all these unrelated formatting changes make this a much bigger diff, so make it much harder to review. pleeeease can we keep formatting and functional changes separate in future?
| def __init__(self, agent, reactor, whitelist=None, blacklist=None): | ||
| """ | ||
| An Agent wrapper which will prevent access to IP addresses being accessed | ||
| directly (without an IP address lookup). |
There was a problem hiding this comment.
I'm not really clear what you mean by "an IP address lookup" here.
Is this just an optimisation? I'm not sure the extra complexity it introduces makes it worthwhile if so.
There was a problem hiding this comment.
Accessing an IP address directly means that it doesn't do a DNS lookup for said IP address.
There was a problem hiding this comment.
right, so it never hits the IPBlacklistingResolver? Is this not a problem if we get a 302 redirect to an IP address?
|
The 302 should reenter through request ()? Let me check. Maybe it doesn't
work how I think.
…On Sat., 22 Dec. 2018, 02:55 Richard van der Hoff ***@***.*** wrote:
***@***.**** commented on this pull request.
------------------------------
In synapse/http/client.py
<#4215 (comment)>:
> @@ -111,20 +128,32 @@ def _callback(addrs):
class BlacklistingAgentWrapper(Agent):
- def __init__(self, agent, reactor, whitelist=None, blacklist=None):
+ """
+ An Agent wrapper which will prevent access to IP addresses being accessed
+ directly (without an IP address lookup).
right, so it never hits the IPBlacklistingResolver? Is this not a problem
if we get a 302 redirect to an IP address?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#4215 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADJ2XHkb3pgiShRqtBj3OkDJBf3Tv1P9ks5u7QRygaJpZM4YsOzv>
.
|
|
Okay, yes, that's how it works. When it gets a redirect, it recalls
request, which then has the filter.
I can add another test if it would make you feel better :)
…On Sat., 22 Dec. 2018, 02:57 Amber Brown ***@***.*** wrote:
The 302 should reenter through request ()? Let me check. Maybe it doesn't
work how I think.
On Sat., 22 Dec. 2018, 02:55 Richard van der Hoff <
***@***.*** wrote:
> ***@***.**** commented on this pull request.
> ------------------------------
>
> In synapse/http/client.py
> <#4215 (comment)>:
>
> > @@ -111,20 +128,32 @@ def _callback(addrs):
>
>
> class BlacklistingAgentWrapper(Agent):
> - def __init__(self, agent, reactor, whitelist=None, blacklist=None):
> + """
> + An Agent wrapper which will prevent access to IP addresses being accessed
> + directly (without an IP address lookup).
>
> right, so it never hits the IPBlacklistingResolver? Is this not a problem
> if we get a 302 redirect to an IP address?
>
> —
> You are receiving this because you modified the open/close state.
> Reply to this email directly, view it on GitHub
> <#4215 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ADJ2XHkb3pgiShRqtBj3OkDJBf3Tv1P9ks5u7QRygaJpZM4YsOzv>
> .
>
|
|
It would make me feel better, but if you're confident it works right, I don't mind too much :) |
Synapse 0.34.1rc1 (2019-01-08) ============================== Features -------- - Special-case a support user for use in verifying behaviour of a given server. The support user does not appear in user directory or monthly active user counts. ([\#4141](#4141), [\#4344](#4344)) - Support for serving .well-known files ([\#4262](#4262)) - Rework SAML2 authentication ([\#4265](#4265), [\#4267](#4267)) - SAML2 authentication: Initialise user display name from SAML2 data ([\#4272](#4272)) - Synapse can now have its conditional/extra dependencies installed by pip. This functionality can be used by using `pip install matrix-synapse[feature]`, where feature is a comma separated list with the possible values `email.enable_notifs`, `matrix-synapse-ldap3`, `postgres`, `resources.consent`, `saml2`, `url_preview`, and `test`. If you want to install all optional dependencies, you can use "all" instead. ([\#4298](#4298), [\#4325](#4325), [\#4327](#4327)) - Add routes for reading account data. ([\#4303](#4303)) - Add opt-in support for v2 rooms ([\#4307](#4307)) - Add a script to generate a clean config file ([\#4315](#4315)) - Return server data in /login response ([\#4319](#4319)) Bugfixes -------- - Fix contains_url check to be consistent with other instances in code-base and check that value is an instance of string. ([\#3405](#3405)) - Fix CAS login when username is not valid in an MXID ([\#4264](#4264)) - Send CORS headers for /media/config ([\#4279](#4279)) - Add 'sandbox' to CSP for media reprository ([\#4284](#4284)) - Make the new landing page prettier. ([\#4294](#4294)) - Fix deleting E2E room keys when using old SQLite versions. ([\#4295](#4295)) - The metric synapse_admin_mau:current previously did not update when config.mau_stats_only was set to True ([\#4305](#4305)) - Fixed per-room account data filters ([\#4309](#4309)) - Fix indentation in default config ([\#4313](#4313)) - Fix synapse:latest docker upload ([\#4316](#4316)) - Fix test_metric.py compatibility with prometheus_client 0.5. Contributed by Maarten de Vries <[email protected]>. ([\#4317](#4317)) - Avoid packaging _trial_temp directory in -py3 debian packages ([\#4326](#4326)) - Check jinja version for consent resource ([\#4327](#4327)) - fix NPE in /messages by checking if all events were filtered out ([\#4330](#4330)) - Fix `python -m synapse.config` on Python 3. ([\#4356](#4356)) Deprecations and Removals ------------------------- - Remove the deprecated v1/register API on Python 2. It was never ported to Python 3. ([\#4334](#4334)) Internal Changes ---------------- - Getting URL previews of IP addresses no longer fails on Python 3. ([\#4215](#4215)) - drop undocumented dependency on dateutil ([\#4266](#4266)) - Update the example systemd config to use a virtualenv ([\#4273](#4273)) - Update link to kernel DCO guide ([\#4274](#4274)) - Make isort tox check print diff when it fails ([\#4283](#4283)) - Log room_id in Unknown room errors ([\#4297](#4297)) - Documentation improvements for coturn setup. Contributed by Krithin Sitaram. ([\#4333](#4333)) - Update pull request template to use absolute links ([\#4341](#4341)) - Update README to not lie about required restart when updating TLS certificates ([\#4343](#4343)) - Update debian packaging for compatibility with transitional package ([\#4349](#4349)) - Fix command hint to generate a config file when trying to start without a config file ([\#4353](#4353)) - Add better logging for unexpected errors while sending transactions ([\#4358](#4358))
No description provided.