@-mention rejected by webmention.io

[jamietanna]

When this mention was sent, Bridgy Fed saw a 429 error (which is odd but unrelated - logs) but when resending it manually, the following was returned by webmention.io:

{
  "status": "no_link_found",
  "source": "https://fed.brid.gy/render?source=https://k8s.social/@Marcus/109318700896829533",
  "target": "https://www.jvt.me/",
  "summary": "The source document does not have a link to the target URL"
}

Notice that the &target= is missing.

This may be a webmention.io bug, so I'll raise it there too.

[jamietanna]

Not sure why this is happening though, as webutil (source) is using requests_post (source) which seems to work when I run it locally:

requests.post('https://webmention.io/www.jvt.me/webmention', {'source': 'https://fed.brid.gy/render?source=https%3A%2F%2Fk8s.social%2F%40Marcus%2F109318700896829533&target=https%3A%2F%2Fwww.jvt.me%2F', 'target': 'https://www.jvt.me/'})

[snarfed]

Agreed, this seems odd. There definitely could be a bug on Bridgy Fed's side somewhere, but it's been successfully sending complete webmentions with those https://fed.brid.gy/render?... source URLs for a while now. Here's a recent example on my site; note the complete URL with both source and target params.

[snarfed]

And you're right about the code, the webmention goes through webutil.webmention.send, which calls webutil.util.requests_post, which is just a thin wrapper around requests.post, and passes kwargs through:

https://github.com/snarfed/webutil/blob/365f9466ee2cc8d6d41eed126e6e932412444387/webmention.py#L109-L110

    resp = util.requests_post(endpoint, data={'source': source, 'target': target},
                              allow_redirects=False, **requests_kwargs)

Notably, requests itself serializes the params in the data into a form-encoded POST body. Here's an example of using requests.post to POST param values with ? and & to httpbin.org, looks like they're encoded correctly and make it through ok to httpbin:

>>> import requests
>>> r = requests.post('https://httpbin.org/anything', data={'source': 'http://x?y=z', 'target': 'http://a?b=c'})
>>> print(r.text)
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "source": "http://x?y=z", 
    "target": "http://a?b=c"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Content-Length": "68", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.28.1", 
    "X-Amzn-Trace-Id": "Root=1-636d1bb0-2f79ab9a428ffd3e1dfa6326"
  }, 
  "json": null, 
  "method": "POST", 
  "url": "https://httpbin.org/anything"
}

So, I'm still not sure what happened here. cc @aaronpk

[snarfed]

Confirmed the same thing with webutil.util.requests_post:

>>> from oauth_dropins.webutil import util
>>> r = util.requests_post('https://httpbin.org/anything', data={'source': 'http://x?y=z', 'target': 'http://a?b=c'})
>>> print(r.text)
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "source": "http://x?y=z", 
    "target": "http://a?b=c"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Content-Length": "68", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "oauth-dropins (https://oauth-dropins.appspot.com/)", 
    "X-Amzn-Trace-Id": "Root=1-636d3bfe-1c7e18171e7567b53b3d784a"
  }, 
  "json": null, 
  "method": "POST", 
  "url": "https://httpbin.org/anything"
}

[aaronpk]

The examples in your test use ? But the problem is the &. Can you try with a URL with &

[snarfed]

Ah true, good point! Looks like &s are ok too:

>>> r = util.requests_post('https://httpbin.org/anything', data={'source': 'http://x?y=z&u=v', 'target': 'http://a?b=c&d=e'})
>>> print(r.text)
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "source": "http://x?y=z&u=v", 
    "target": "http://a?b=c&d=e"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Content-Length": "84", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "oauth-dropins (https://oauth-dropins.appspot.com/)", 
    "X-Amzn-Trace-Id": "Root=1-636d3d79-21ea7ba3201feeec0df33b67"
  }, 
  "json": null, 
  "method": "POST", 
  "url": "https://httpbin.org/anything"
}

>>> 

[jamietanna]

Just given it a go again, and seems to have worked from indieweb.social https://webmention.io/www.jvt.me/webmention/VEM04dvTW4B4l7SBA6b0

[snarfed]

We couldn't find a smoking gun in either Bridgy Fed or webmention.io, and it sounds like we can't reproduce it, so I suspect it was transient. Tentatively closing, but feel free to reopen!