Tuple index out of range with 3.7.1 and no IPv6

[danielnelson]

🐞 Describe the bug
After upgrading to aiohttp==3.7.1, dns resolution fails with an exception.

💡 To Reproduce

1. I believe this is due to not have IPV6 enabled in my kernel, which is custom compiled with.
   CONFIG_IPV6=n
2. Run:

   import asyncio
   import aiohttp
   
   
   async def main():
       async with aiohttp.ClientSession() as session:
           async with session.get('http://python.org') as response:
               print(response.status)
   
   
   asyncio.run(main())

3. Actual behavior

   $ python bug.py
   Traceback (most recent call last):
     File "bug.py", line 11, in <module>
       asyncio.run(main())
     File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
       return loop.run_until_complete(main)
     File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
       return future.result()
     File "bug.py", line 7, in main
       async with session.get('http://python.org') as response:
     File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 1124, in __aenter__
       self._resp = await self._coro
     File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 528, in _request
       req, traces=traces, timeout=real_timeout
     File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 541, in connect
       proto = await self._create_connection(req, traces, timeout)
     File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 898, in _create_connection
       _, proto = await self._create_direct_connection(req, traces, timeout)
     File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 1005, in _create_direct_connection
       hosts = await asyncio.shield(host_resolved)
     File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 871, in _resolve_host
       addrs = await self._resolver.resolve(host, port, family=self._family)
     File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/resolver.py", line 37, in resolve
       if family == socket.AF_INET6 and address[3]:  # type: ignore
   IndexError: tuple index out of range

💡 Expected behavior

Should print 200.

📋 Logs/tracebacks

📋 Your version of the Python

$ python --version
Python 3.7.9

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.7.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: [email protected]
License: Apache 2
Location: /home/dbn/usr/py37/lib/python3.7/site-packages
Requires: multidict, async-timeout, typing-extensions, yarl, chardet, attrs
Required-by: replay-proxy

$ python -m pip show multidict
Name: multidict
Version: 4.7.6
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/dbn/usr/py37/lib/python3.7/site-packages
Requires:
Required-by: yarl, aiohttp

$ python -m pip show yarl
Name: yarl
Version: 1.5.1
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/dbn/usr/py37/lib/python3.7/site-packages
Requires: typing-extensions, multidict, idna
Required-by: aiohttp, replay-proxy

📋 Additional context

[asvetlov]

I'm curious what parameters are passed into getaddrinfo() in your case and what list is returned?
Could you print them in a comment?

My initial thought is: on systems with IPv6 not installed the check if family == socket.AF_INET6 and address[3]: should fail on family == socket.AF_INET6 test; adress[3] test is not executed in this case.

[danielnelson]

I added some debug statements to resolver.py:

  async def resolve(
      self, hostname: str, port: int = 0, family: int = socket.AF_INET
  ) -> List[Dict[str, Any]]:
+     print('getaddrinfo(%s, %s, type=%s, family=%s)' % (hostname, port, type, family))
      infos = await self._loop.getaddrinfo(
          hostname, port, type=socket.SOCK_STREAM, family=family
      )
+     print('infos:', infos)

getaddrinfo(python.org, 80, type=<class 'type'>, family=0)
infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('45.55.99.72', 80))]
getaddrinfo(python.org, 443, type=<class 'type'>, family=0)
infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('45.55.99.72', 443))]
getaddrinfo(www.python.org, 443, type=<class 'type'>, family=0)
infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.188.223', 443)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00'))]
Traceback (most recent call last):
  File "bug-5156.py", line 11, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "bug-5156.py", line 7, in main
    async with session.get('http://python.org') as response:
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 1124, in __aenter__
    self._resp = await self._coro
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 528, in _request
    req, traces=traces, timeout=real_timeout
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 541, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 898, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 1005, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 871, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/resolver.py", line 39, in resolve
    if family == socket.AF_INET6 and address[3]:  # type: ignore
IndexError: tuple index out of range

For reference, here is my current DNS resolution for www.python.org

$ host www.python.org
www.python.org is an alias for dualstack.python.map.fastly.net.
dualstack.python.map.fastly.net has address 151.101.188.223
dualstack.python.map.fastly.net has IPv6 address 2a04:4e42:2d::223

[asvetlov]

Interesting...
In (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00') the (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00') doesn't look like IPv6 at all!
Is it correct?
I mean if I do host, port = (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00') -- both host and port are broken anyway.

[danielnelson]

I found this which explains the cause: https://bugs.python.org/issue16208

FWIW, I was able to process the results of getaddrinfo in C. Here is some embarrassingly bad C code:

#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#include <arpa/inet.h>
#include <stdlib.h>


#define BUF_SIZE 500

void
print_addr(struct addrinfo *result) {
	char *addr = NULL;
	switch(result->ai_family) {
		case AF_INET: {
			struct sockaddr_in *addr_in = (struct sockaddr_in *)result->ai_addr;
			addr = malloc(INET_ADDRSTRLEN);
			inet_ntop(AF_INET, &(addr_in->sin_addr), addr, INET_ADDRSTRLEN);
			break;
		}
		case AF_INET6: {
			struct sockaddr_in6 *addr_in6 = (struct sockaddr_in6 *)result->ai_addr;
			addr = malloc(INET6_ADDRSTRLEN);
			inet_ntop(AF_INET6, &(addr_in6->sin6_addr), addr, INET6_ADDRSTRLEN);
			break;
		}
		default:
			break;
	}
	printf("IP address: %s\n", addr);
	free(addr);
}

int
main(int argc, char *argv[])
{
	struct addrinfo hints;
	struct addrinfo *result, *rp;
	int sfd, s, j;
	size_t len;
	ssize_t nread;
	char buf[BUF_SIZE];

	if (argc < 3) {
		fprintf(stderr, "Usage: %s host port msg...\n", argv[0]);
		exit(EXIT_FAILURE);
	}

	memset(&hints, 0, sizeof(struct addrinfo));
	hints.ai_family = AF_UNSPEC;    /* Allow IPv4 or IPv6 */
	hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */
	hints.ai_flags = 0;
	hints.ai_protocol = 0;          /* Any protocol */

	s = getaddrinfo(argv[1], argv[2], &hints, &result);
	if (s != 0) {
		fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s));
		exit(EXIT_FAILURE);
	}

	while (result != 0) {
		print_addr(result);
		result = result->ai_next;
	}
}

$ gcc bug.c
$ ./a.out www.python.org 443
IP address: 151.101.188.223
IP address: 2a04:4e42:2d::223

I tried to do something similar in Python, I'm not sure if I'm doing it right though:

import socket

for ai in socket.getaddrinfo('www.python.org', 443):
    print(ai)
    if ai[0] == socket.AF_INET6:
        af, socktype, proto, canonname, sa = ai
        print("AF_INET6", socket.inet_ntop(socket.AF_INET6, sa[1]))

$ python getaddrinfo.py 
(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.188.223', 443))
(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('1imagine51.101.188.223', 443))
(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('151.101.188.223', 443))
(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00'))
Traceback (most recent call last):
  File "getaddrinfo.py", line 7, in <module>
    print("AF_INET6", socket.inet_ntop(socket.AF_INET6, sa[1]))
ValueError: unknown address family 10

I'm running a Gentoo system with the -ipv6 use flag set, but I enabled it for the Python build as it seems problematic.

Perhaps a fix in aiohttp would be to check at AF_INET6 addresses are a 4 tuple and ignore if they are not.

Also, I think this is the same issue as #4557

[webknjaz]

I'm running a Gentoo system with the -ipv6 use flag set, but I enabled it for the Python build as it seems problematic.

So it's enabled in CPython but disabled in the kernel? What about sysctl? Could you post your sudo sysctl -a | grep ipv6 output? Does it have something like net.ipv6.conf.lo.disable_ipv6 set? How about other interfaces? Or is this setting just missing?

[webknjaz]

Also, I think this is the same issue as #4557

So it should probably be mitigated with "has ipv6" check like https://github.com/tornadoweb/tornado/pull/670/files#diff-e9701aba4f8165d329018bef200b0d98dd70c24856c159b5c29cce89ddd69e8bR56

[asvetlov]

Perhaps a fix in aiohttp would be to check at AF_INET6 addresses are a 4 tuple and ignore if they are not.

From a practical point of view, I'm ok with this solution if it raises a warning before the ignoring.
From my understanding, we see a perfect example of a gray corner case that we can pass around easily.
A detection for address tuple size smells bad but looks working.

[danielnelson]

So it's enabled in CPython but disabled in the kernel? What about sysctl?

Sorry, my comment was a little confusing, when I reported this issue it was disabled in Python and in the Kernel. My workaround is to enable it in Python and leave it disabled in the Kernel.

Could you post your sudo sysctl -a | grep ipv6 output?

It looks like there are no results containing ipv6.

I like the socket.has_ipv6 solution, maybe we should test for this when creating the TCPConnector and set the family to AF_UNSPEC or AF_INET. Essentially the linked tornado code.

[webknjaz]

So I looked at those bugs on CPython tracker and I'm puzzled why they are "wontfix". Moreover, I think that CPython may be incorrectly processing NUL-terminated strings or something like that.

I've tried decoding that bytesting manually on a Gentoo laptop with IPv6 on and still got an error (a different error but still):

>>> socket.inet_ntop(socket.AF_INET6, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid length of packed IP address string

Something about it looked weird and I tried to do the packing of a known address 2a04:4e42:41::223:

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6)
[(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('2a04:4e42:41::223', 443, 0, 0))]
>>> socket.inet_pton(socket.AF_INET6, '2a04:4e42:2d::223')
b'*\x04NB\x00-\x00\x00\x00\x00\x00\x00\x00\x00\x02#'

Woah! That looks different! Let's try unpacking that correct value, just to be sure:

>>> socket.inet_ntop(socket.AF_INET6, b'*\x04NB\x00-\x00\x00\x00\x00\x00\x00\x00\x00\x02#')
'2a04:4e42:2d::223'

Yaaay! We've got the correct value!

Based on this, I'm pretty much convinced that the bug in CPython originates in socket.getaddrinfo() returning a malformed packed IPv6 address.

P.S. It's interesting that both representations have identical *\x04NB\x00-\x00\x00 substring ("\x01\xbb\x00\x00\x00\x00 *\x04NB\x00-\x00\x00" and " *\x04NB\x00-\x00\x00\x00\x00\x00\x00\x00\x00\x02#") so I'm convinced it has something to do with processing the memory offsets on the C level.

[webknjaz]

So I looked up the getaddrinfo(3) manpage and it seems like the fix could be as easy as adding a socket.AI_ADDRCONFIG as a flag for the getaddrinfo() call:

If hints.ai_flags includes the AI_ADDRCONFIG flag, then IPv4
addresses are returned in the list pointed to by res only if the
local system has at least one IPv4 address configured, and IPv6
addresses are returned only if the local system has at least one IPv6
address configured. The loopback address is not considered for this
case as valid as a configured address. This flag is useful on, for
example, IPv4-only systems, to ensure that getaddrinfo() does not
return IPv6 socket addresses that would always fail in connect(2) or
bind(2).

@danielnelson could you please try this in your Python REPL:

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)

and also (just in case):

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)

[webknjaz]

Other observations:

1. The broken bytesting is 14 bytes long while the correct one is 16 bytes
2. I haven't noticed anything immediately obvious in https://github.com/python/cpython/blob/master/Modules/socketmodule.c so maybe this is how the underlying C-libs behave...

[danielnelson]

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.188.223', 443))]

[asvetlov]

socket.has_ipv6 is compile-time constant. I wonder how can it help.
@danielnelson do you have has_ipv6 == False on your system?

[webknjaz]

@asvetlov has_ipv6 sounds like a band-aid. It'll probably help but flags​=​socket​.​AI_ADDRCONFIG is preferable as it'll make getaddrinfo() return only the address families that the network stack on the machine actually supports.

[asvetlov]

On Linux, flags=0 for getaddrinfo() is considered as (AI_V4MAPPED | AI_ADDRCONFIG) according to docs: https://man7.org/linux/man-pages/man3/getaddrinfo.3.html

I read it as AI_ADDRCONFIG is passed.

Anyway, whatever we decide we need a PR that passes at least both our CI and works well on @danielnelson box.

Does anybody want to create it?

[danielnelson]

I believe AI_ADDRCONFIG is not passed by default because Python always passed a hints struct.

If I call s = getaddrinfo(argv[1], argv[2], NULL, &result);only IPv4 addresses are returned. But if I call with a zeroed hints struct I get all addresses families:

memset(&hints, 0, sizeof(struct addrinfo));
s = getaddrinfo(argv[1], argv[2], &hints, &result);

[webknjaz]

@asvetlov It's what the underlying C API defaults to.

But no, it not the default in the CPython stdlib. Both the source (https://github.com/python/cpython/blob/dff1ad5/Modules/socketmodule.c#L6463) and the docs (https://docs.python.org/3/library/socket.html#socket.getaddrinfo) confirm that flags=0.

So I'm still convinced that setting AI_ADDRCONFIG is the best solution here.

[webknjaz]

A new issue #5901 just came in today reporting this problem on the recent version of the 3.7 stream (3.7.4.post0). So I'm reopening this for now. I think it may be reasonable to come up with a docker image based on Gentoo with the ipv6 USE-flag disabled globally and enabled just for CPython. We could then use that image to verify the fix and also make a regression test using it.

[dilyanpalauzov]

As mentioned at #5901 the 3.8 branch fails on my system. Here is the output of the questions above:

import socket
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00\x03\x00\x00'))]

>>> socket.has_ipv6 == False
True

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.12.223', 443)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00\x03\x00\x00'))]

I have libc 2.33 and kernel 4.14.217

The patch from #5901 is included in the tip of the 3.8 branch, but does not help.

[Hanaasagi]

Something that might be helpful. Use a self-compiled CPython with --enable-ipv6 option. And disable disable ipv6 via sysctl -w net.ipv6.conf.all.disable_ipv6=1. Then call socket.getaddrinfo with family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG, will get a socket.gaierror: [Errno -2] Name or service not known.

Python 3.8.5 (default, Jul 20 2021, 04:42:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.has_ipv6
True
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/Python-3.8.5/Lib/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.108.223', 443))]

---

Update:
This can reproduce with

1. Enable ipv6 stack in Linux: sysctl -w net.ipv6.conf.all.disable_ipv6=0
2. Compile CPython with --disable-ipv6 option

Python 3.8.5 (default, Jul 20 2021, 05:11:52)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.has_ipv6
False
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00\x1a\x00\x00'))]

See the CPython implemention makesockaddr fucntion:

If we don't enable ipv6 when compile CPython, this case is missing.

#ifdef ENABLE_IPV6
    case AF_INET6:
    {
        const struct sockaddr_in6 *a = (const struct sockaddr_in6 *)addr;
        PyObject *addrobj = make_ipv6_addr(a);
        PyObject *ret = NULL;
        if (addrobj) {
            ret = Py_BuildValue("OiII",
                                addrobj,
                                ntohs(a->sin6_port),
                                ntohl(a->sin6_flowinfo),
                                a->sin6_scope_id);
            Py_DECREF(addrobj);
        }
        return ret;
    }

https://github.com/python/cpython/blob/7766b96ab80b04509bbac708ee5ecf3c1c5934fc/Modules/socketmodule.c#L1237-L1252

And it will go to the default case. Just return the sa_family and raw bytes.

    default:
        /* If we don't know the address family, don't raise an
           exception -- return it as an (int, bytes) tuple. */
        return Py_BuildValue("iy#",
                             addr->sa_family,
                             addr->sa_data,
                             sizeof(addr->sa_data));

https://github.com/python/cpython/blob/7766b96ab80b04509bbac708ee5ecf3c1c5934fc/Modules/socketmodule.c#L1436-L1445

aiohttp will get a integer 10, but it doesn't mean a AddressFamily.AF_INET6. Unfortunately, it's a IntEnum, so we will mistake this for a ipv6 address.

>>> socket.AddressFamily.AF_INET6 == 10
True

My suggestion is to use socket.has_ipv6 for pre-check.

[dilyanpalauzov]

I have compiled python with ./configure --enable-loadable-sqlite-extensions --disable-ipv6 --with-system-expat --with-system-libmpdec --enable-shared.

[Hanaasagi]

Link a fix: #5906.

[asvetlov]

Fixed