Unclosed socket warning

[diegorondini]

Sometimes when running linkchecker against a folder with markdown files I get a warning at the end of the execution:
sys:1: ResourceWarning: unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('<ip-addr-of-local-pc>', 60350), raddr=('<ip-addr-of-remote-url>', 443)> ResourceWarning: Enable tracemalloc to get the object allocation traceback /usr/lib64/python3.8/asyncio/selector_events.py:696: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=6> _warn(f"unclosed transport {self!r}", ResourceWarning, source=self) ResourceWarning: Enable tracemalloc to get the object allocation traceback

This happens with linkcheckmd 1.3.0 on Fedora 32.

[scivision]

which version of Python e.g. 3.7.5 or ? I can try with the same version. There were/are some Python versions that this may be a bug in Python itself for.

[scivision]

Oops, this is noted in aiohttp manual. I added a sleep just before loop close (not in loop) in bd50af6

[diegorondini]

Python version was 3.8, it was hidden in the log.
By the way, you can probably revert the change when you'll move to aiohttp 4.0.0:
aio-libs/aiohttp#1925 (comment)

Thanks you for the quick fix by the way!

[scivision]

Yes looks like it could be a while for 4.0, thanks for your research. Asyncio itself is undergoing enhancements in future Python releases that will possibly bring other changes to my asyncio-using packages as well.

[diegorondini]

Hi Michael,

this is probably low priority, as the issue is in the underlying library(ies), but I still can reproduce the issue on 1.3.1 with:

$ git clone https://gitlab.cern.ch/sft/lcgdocs.git
$ cd lcgdocs/
$ linkcheckMarkdown -r docs/

After several broken links you will get something along the lines of:

...
15.6 seconds to check links
sys:1: ResourceWarning: unclosed <socket.socket fd=15, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('<removed-ip-addr>', 38326), raddr=('188.184.9.234', 443)>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib64/python3.8/asyncio/selector_events.py:696: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=15>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed <socket.socket fd=13, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('<removed-ip-addr>', 57598), raddr=('188.184.20.224', 443)>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib64/python3.8/asyncio/selector_events.py:696: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=13>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib64/python3.8/asyncio/selector_events.py:696: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=19>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib64/python3.8/asyncio/selector_events.py:696: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=6>
  _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

If you're not getting these warnings on your system you can probably reproduce them using a Fedora 32 container.

Anyway, thanks a lot for your effort!

[scivision]

I wonder if git cloning this repo, and then increasing asyncio.sleep() to 1 second or something helps

linkchecker-markdown/src/linkcheckmd/coro.py

Line 45 in 259c8c8

await asyncio.sleep(0.250)

git clone https://github.com/scivision/linkchecker-markdown/
pip install -e linkchecker-markdown

the "-e" installs a live copy, where your edits are reflected on the next fresh import

[diegorondini]

Hi @scivision

I tried with 1 and 2 seconds, but I haven't seen any change, the problem persists.

Thank you