Improve concurrency on Windows

[elacuesta]

Closes #282

Alternative approach to #285

This yields full concurrency ('playwright/page_count/max_concurrent': 32 in stats) with the sample spider from #285:

import scrapy

class DelayTest(scrapy.Spider):
    name = "delay"
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            # "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        "PLAYWRIGHT_MAX_PAGES_PER_CONTEXT": 32,
        "CONCURRENT_REQUESTS_PER_IP": 32,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 32,
        "CONCURRENT_REQUESTS": 32,
        "PLAYWRIGHT_LAUNCH_OPTIONS": {"headless": False},
    }

    def start_requests(self):
        for i in range(32):
            yield scrapy.Request(
                url=f"https://httpbin.org/delay/1?i={i}",
                meta={"playwright": True},
            )

    def parse(self, response):
        print(response.url)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (55fd416) to head (9942e6b).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main      #286   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            6         6           
  Lines          517       543   +26     
=========================================
+ Hits           517       543   +26     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Gallaecio]

It may not apply here but, for the record, when implementing session management for scrapy-zyte-api, I found that I needed to keep live references to task objects until finished to prevent them from being garbage-collected.

[elacuesta]

Interesting, thank you very much for the reference. I haven't seen that in my tests but it might be that my runs were not long enough. I won't include it right away just to keep things simple but I'll keep it in mind in case someone reports it as an issue.

[Gallaecio]

Shouldn’t this still have Windows, Proactor or something of the sort in the name? (I’m looking at def start).

[elacuesta]

I renamed it like this because it's technically possible to use it on other platforms as well. The concurrency issue also happens on Linux with this threaded loop approach, it's not an intrinsic Windows issue, so I did most of the development in Linux and only then I tried it on Windows to make sure it works.