execution getting stucked at Enabled pipeline when using scrapy-playwright on Windows with Scrapy==2.13

[amanengineosol]

When implementing scrapy-playwright with scrapy.Request() my spider execution getting stucked after Enable item pipeline and when i do ctrl+c then execution gets continued.
This problem get resolved when i remove DOWNLOAD HANDLER { } in settings.py used for scrapy-playwright
Using these version
Scrapy 2.13.0
scrapy-playwright 0.0.43

2025-06-17 13:11:23 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2025-06-17 13:11:23 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.start.StartSpiderMiddleware',
'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2025-06-17 13:11:23 [scrapy.middleware] INFO: Enabled item pipelines:
[]

[elacuesta]

I'm sorry, I could not reproduce:

# spider file
import scrapy


class ExampleSpider(scrapy.Spider):
    name = "example"
    allowed_domains = ["example.org"]
    start_urls = ["https://example.org"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url=url, meta={"playwright": True})

    def parse(self, response):
        print(response.text)
        return {"url": response.url, "status": response.status}

# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

$ scrapy version -v
Scrapy       : 2.13.0
lxml         : 5.2.2
libxml2      : 2.12.6
cssselect    : 1.2.0
parsel       : 1.9.1
w3lib        : 2.2.1
Twisted      : 24.3.0
Python       : 3.10.12 (main, Feb  4 2025, 14:57:36) [GCC 11.4.0]
pyOpenSSL    : 24.2.1 (OpenSSL 3.3.1 4 Jun 2024)
cryptography : 43.0.0
Platform     : Linux-6.5.0-45-generic-x86_64-with-glibc2.35

$ playwright --version
Version 1.52.0

$ python -c "import scrapy_playwright; print(scrapy_playwright.__version__)"
0.0.43

Full output

$ scrapy crawl example
2025-06-17 10:08:41 [scrapy.utils.log] INFO: Scrapy 2.13.0 started (bot: test_spider)
2025-06-17 10:08:41 [scrapy.utils.log] INFO: Versions:
{'lxml': '5.2.2',
 'libxml2': '2.12.6',
 'cssselect': '1.2.0',
 'parsel': '1.9.1',
 'w3lib': '2.2.1',
 'Twisted': '24.3.0',
 'Python': '3.10.12 (main, Feb  4 2025, 14:57:36) [GCC 11.4.0]',
 'pyOpenSSL': '24.2.1 (OpenSSL 3.3.1 4 Jun 2024)',
 'cryptography': '43.0.0',
 'Platform': 'Linux-6.5.0-45-generic-x86_64-with-glibc2.35'}
2025-06-17 10:08:41 [scrapy.addons] INFO: Enabled addons:
[]
2025-06-17 10:08:41 [asyncio] DEBUG: Using selector: EpollSelector
2025-06-17 10:08:41 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2025-06-17 10:08:41 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.unix_events._UnixSelectorEventLoop
2025-06-17 10:08:41 [scrapy.extensions.telnet] INFO: Telnet Password: 4a2fa8d91ee44c4b
2025-06-17 10:08:41 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2025-06-17 10:08:41 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'test_spider',
 'EDITOR': 'nano',
 'FEED_EXPORT_ENCODING': 'utf-8',
 'NEWSPIDER_MODULE': 'test_spider.spiders',
 'ROBOTSTXT_OBEY': True,
 'SPIDER_MODULES': ['test_spider.spiders']}
2025-06-17 10:08:42 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2025-06-17 10:08:42 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.start.StartSpiderMiddleware',
 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2025-06-17 10:08:42 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2025-06-17 10:08:42 [scrapy.core.engine] INFO: Spider opened
2025-06-17 10:08:42 [py.warnings] WARNING: /home/eugenio/zyte/scrapy-playwright/venv-scrapy-playwright/lib/python3.10/site-packages/scrapy/core/spidermw.py:433: ScrapyDeprecationWarning: test_spider.spiders.example.ExampleTemplateSpider defines the deprecated start_requests() method. start_requests() has been deprecated in favor of a new method, start(), to support asynchronous code execution. start_requests() will stop being called in a future version of Scrapy. If you use Scrapy 2.13 or higher only, replace start_requests() with start(); note that start() is a coroutine (async def). If you need to maintain compatibility with lower Scrapy versions, when overriding start_requests() in a spider class, override start() as well; you can use super() to reuse the inherited start() implementation without copy-pasting. See the release notes of Scrapy 2.13 for details: https://docs.scrapy.org/en/2.13/news.html
  warn(

2025-06-17 10:08:42 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2025-06-17 10:08:42 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2025-06-17 10:08:42 [scrapy-playwright] INFO: Starting download handler
2025-06-17 10:08:42 [scrapy-playwright] INFO: Starting download handler
2025-06-17 10:08:42 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://example.org/robots.txt> (referer: None)
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 12 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 13 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 14 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 18 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 19 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 20 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 22 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 23 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 25 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 27 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 29 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 31 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 32 without any user agent to enforce it on.
2025-06-17 10:08:42 [protego] DEBUG: Rule at line 43 without any user agent to enforce it on.
2025-06-17 10:08:42 [scrapy-playwright] INFO: Launching browser chromium
2025-06-17 10:08:42 [scrapy-playwright] INFO: Browser chromium launched
2025-06-17 10:08:42 [scrapy-playwright] DEBUG: Browser context started: 'default' (persistent=False, remote=False)
2025-06-17 10:08:43 [scrapy-playwright] DEBUG: [Context=default] New page created, page count is 1 (1 for all contexts)
2025-06-17 10:08:43 [scrapy-playwright] DEBUG: [Context=default] Request: <GET https://example.org/> (resource type: document)
2025-06-17 10:08:43 [scrapy-playwright] DEBUG: [Context=default] Response: <200 https://example.org/>
2025-06-17 10:08:43 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://example.org> (referer: None) ['playwright']
<!DOCTYPE html><html><head>
    <title>Example Domain</title>

    <meta charset="utf-8">
    <meta http-equiv="Content-type" content="text/html; charset=utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <style type="text/css">
    body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
        
    }
    div {
        width: 600px;
        margin: 5em auto;
        padding: 2em;
        background-color: #fdfdff;
        border-radius: 0.5em;
        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
    }
    a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
    @media (max-width: 700px) {
        div {
            margin: 0 auto;
            width: auto;
        }
    }
    </style>    
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
    domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>


</body></html>
2025-06-17 10:08:43 [scrapy.core.scraper] DEBUG: Scraped from <200 https://example.org/>
{'url': 'https://example.org/', 'status': 200}
2025-06-17 10:08:43 [scrapy.core.engine] INFO: Closing spider (finished)
2025-06-17 10:08:43 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 434,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 3238,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 1,
 'downloader/response_status_count/404': 1,
 'elapsed_time_seconds': 1.588985,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2025, 6, 17, 13, 8, 43, 671745, tzinfo=datetime.timezone.utc),
 'item_scraped_count': 1,
 'items_per_minute': 60.0,
 'log_count/DEBUG': 24,
 'log_count/INFO': 14,
 'log_count/WARNING': 1,
 'memusage/max': 70889472,
 'memusage/startup': 70889472,
 'playwright/browser_count': 1,
 'playwright/context_count': 1,
 'playwright/context_count/max_concurrent': 1,
 'playwright/context_count/persistent/False': 1,
 'playwright/context_count/remote/False': 1,
 'playwright/page_count': 1,
 'playwright/page_count/closed': 1,
 'playwright/page_count/max_concurrent': 1,
 'playwright/request_count': 1,
 'playwright/request_count/method/GET': 1,
 'playwright/request_count/navigation': 1,
 'playwright/request_count/resource_type/document': 1,
 'playwright/response_count': 1,
 'playwright/response_count/method/GET': 1,
 'playwright/response_count/resource_type/document': 1,
 'response_received_count': 2,
 'responses_per_minute': 120.0,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/404': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2025, 6, 17, 13, 8, 42, 82760, tzinfo=datetime.timezone.utc)}
2025-06-17 10:08:43 [scrapy.core.engine] INFO: Spider closed (finished)
2025-06-17 10:08:43 [scrapy-playwright] INFO: Closing download handler
2025-06-17 10:08:43 [scrapy-playwright] INFO: Closing download handler
2025-06-17 10:08:43 [scrapy-playwright] DEBUG: Browser context closed: 'default' (persistent=False, remote=False)
2025-06-17 10:08:43 [scrapy-playwright] INFO: Closing browser
2025-06-17 10:08:43 [scrapy-playwright] DEBUG: Browser disconnected

[PhiFever]

I can reproduce this issue. It appears to only occur on Windows. I am using the default settings.py:

from typing import AsyncIterator

import scrapy
from scrapy import Request


class TestSpider(scrapy.Spider):
    name = "Test"

    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        "USER_AGENT": None,
    }

    async def start(self) -> AsyncIterator[Request]:
        for i in range(10):
            self.logger.info(f"Requesting {i}")
            yield scrapy.Request(
                url=f"https://httpbin.org/get?i={i}",
                meta={"playwright": True},
            )

    def parse(self, response, **kwargs):
        self.logger.info(f"Response: {response.status}, body: {response.body}")

versions:

$ scrapy version -v
Scrapy       : 2.13.2
lxml         : 5.4.0
libxml2      : 2.11.9
cssselect    : 1.3.0
parsel       : 1.10.0
w3lib        : 2.3.1
Twisted      : 25.5.0
Python       : 3.12.10 (tags/v3.12.10:0cc8128, Apr  8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]
pyOpenSSL    : 25.1.0 (OpenSSL 3.5.0 8 Apr 2025)
cryptography : 45.0.4
Platform     : Windows-11-10.0.26100-SP0
$ playwright --version
Version 1.52.0
$ python -c "import scrapy_playwright; print(scrapy_playwright.__version__)"
0.0.43

After launching on Windows, the logs become stuck at this point and cannot proceed:

2025-06-18 10:50:39 [scrapy.utils.log] INFO: Scrapy 2.13.2 started (bot: scarpy_playwright_test)
2025-06-18 10:50:39 [scrapy.utils.log] INFO: Versions:
{'lxml': '5.4.0',
 'libxml2': '2.11.9',
 'cssselect': '1.3.0',
 'parsel': '1.10.0',
 'w3lib': '2.3.1',
 'Twisted': '25.5.0',
 'Python': '3.12.10 (tags/v3.12.10:0cc8128, Apr  8 2025, 12:21:36) [MSC v.1943 '
           '64 bit (AMD64)]',
 'pyOpenSSL': '25.1.0 (OpenSSL 3.5.0 8 Apr 2025)',
 'cryptography': '45.0.4',
 'Platform': 'Windows-11-10.0.26100-SP0'}
2025-06-18 10:50:39 [scrapy.addons] INFO: Enabled addons:
[]
2025-06-18 10:50:39 [asyncio] DEBUG: Using selector: SelectSelector
2025-06-18 10:50:39 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2025-06-18 10:50:39 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.windows_events._WindowsSelectorEventLoop
2025-06-18 10:50:39 [scrapy.extensions.telnet] INFO: Telnet Password: c7e92aad391cbfa9
2025-06-18 10:50:39 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2025-06-18 10:50:39 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'scarpy_playwright_test',
 'CONCURRENT_REQUESTS_PER_DOMAIN': 4,
 'FEED_EXPORT_ENCODING': 'utf-8',
 'NEWSPIDER_MODULE': 'scarpy_playwright_test.spiders',
 'ROBOTSTXT_OBEY': True,
 'SPIDER_MODULES': ['scarpy_playwright_test.spiders'],
 'USER_AGENT': None}
2025-06-18 10:50:39 [asyncio] DEBUG: Using proactor: IocpProactor
2025-06-18 10:50:39 [scrapy-playwright] INFO: Started loop on separate thread: <ProactorEventLoop running=True closed=False debug=False>
2025-06-18 10:50:39 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2025-06-18 10:50:39 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.start.StartSpiderMiddleware',
 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2025-06-18 10:50:39 [scrapy.middleware] INFO: Enabled item pipelines:
[]

[PhiFever]

And this is the playwright version, works well on windows.

import asyncio
from playwright.async_api import async_playwright


async def main():
    async with async_playwright() as pw:
        browser = await pw.chromium.launch()
        page = await browser.new_page()
        for i in range(10):
            print(f"Requesting {i}")
            response = await page.goto(f"https://httpbin.org/get?i={i}")
            print(f"Response: {response.status}, body: {await response.text()}")
        await browser.close()


if __name__ == '__main__':
    asyncio.run(main())

[amanengineosol]

And this is the playwright version, works well on windows.

import asyncio
from playwright.async_api import async_playwright

async def main():
async with async_playwright() as pw:
browser = await pw.chromium.launch()
page = await browser.new_page()
for i in range(10):
print(f"Requesting {i}")
response = await page.goto(f"https://httpbin.org/get?i={i}")
print(f"Response: {response.status}, body: {await response.text()}")
await browser.close()

if name == 'main':
asyncio.run(main())

Thanks for sharing! if you have any work around for scrapy.Request + scrapy_playwright then please share.

[wuweok]

I'm sorry, I could not reproduce:

spider file

import scrapy

class ExampleSpider(scrapy.Spider):
name = "example"
allowed_domains = ["example.org"]
start_urls = ["https://example.org"]

def start_requests(self):
    for url in self.start_urls:
        yield scrapy.Request(url=url, meta={"playwright": True})

def parse(self, response):
    print(response.text)
    return {"url": response.url, "status": response.status}

settings.py

DOWNLOAD_HANDLERS = {
"http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

$ scrapy version -v
Scrapy       : 2.13.0
lxml         : 5.2.2
libxml2      : 2.12.6
cssselect    : 1.2.0
parsel       : 1.9.1
w3lib        : 2.2.1
Twisted      : 24.3.0
Python       : 3.10.12 (main, Feb  4 2025, 14:57:36) [GCC 11.4.0]
pyOpenSSL    : 24.2.1 (OpenSSL 3.3.1 4 Jun 2024)
cryptography : 43.0.0
Platform     : Linux-6.5.0-45-generic-x86_64-with-glibc2.35

$ playwright --version
Version 1.52.0

$ python -c "import scrapy_playwright; print(scrapy_playwright.__version__)"
0.0.43

Full output

@elacuesta Have you tried this problem on Windows?
If you can't reproduce, could you so kindly share library information?

[Tim-pijnacker]

I have the same issues on windows, for me the issue stays even when removing DOWNLOAD_HANDLERS from settings.py

[wuweok]

Hi all,

This problem can be avoid by rollback scrapy version to 2.12.0.

Best Regards

[elacuesta]

I was now able to reproduce on Windows with Scrapy 2.13. As @wuweok suggested, please pin scrapy<2.13 while I try to understand what is causing this issue.

[Gallaecio]

It would be great if someone could also check whether installing the latest branch from Scrapy (at scrapy/scrapy@e225d0d at the time of writing) has this issue or not.