adding FromSetting for declarative spider param defaults

[emarondan]

This change introduces FromSetting, a marker that allows spider parameters to declare defaults pulled directly from crawler.settings.
Defaults are resolved in _set_crawler following this precedence:

• Values passed via CLI/kwargs (-a)
• Literal defaults defined in the Pydantic model
• Values from Scrapy settings using FromSetting (with the chosen getter)
• Explicit default provided in FromSetting if the setting is missing
• Validation error if none of the above apply

Unit tests were added to ensure correct precedence and proper integration with Scrapy settings.

[wRAR]

Sorry, where is this called (normally)?

[emarondan]

It is called when a spider is created using from_crawler()
https://github.com/scrapy/scrapy/blob/f2531808f3b0041786c68e50a88f6c5500a37dd4/scrapy/spiders/__init__.py#L76

[wRAR]

Oh.

I think this should override from_crawler() instead, though I'm not 100% sure if it's possible.

[wRAR]

Regarding test failures, it's an incompatibility with Scrapy 2.13 which I haven't thought of checking, either adding install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor") to e.g. conftest.py will be enough or the tests need to be changed to actually call and await crawler.crawl() (I haven't looked at the code).

And it should be unrelated to the changes of this PR.

[emarondan]

I added a conftest.py to make it pass the tests locally. The file content is:

import os

os.environ.setdefault(
    "TWISTED_REACTOR",
    "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
)

try:
    from twisted.internet import asyncioreactor
    asyncioreactor.install()
except Exception:
    pass

Should I add this change to the PR?

[wRAR]

Yes please.

[wRAR]

Though I don't think touching the TWISTED_REACTOR envvar makes sense, Scrapy should only operate on settings (I doubt it even reads this envvar, though I won't be surprised that it does via some deprecated mechanism).

[codecov]

Codecov Report

❌ Patch coverage is 76.47059% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.45%. Comparing base (63b3991) to head (784bf7e).

Files with missing lines	Patch %	Lines
scrapy_spider_metadata/_params.py	70.37%	5 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #25      +/-   ##
==========================================
- Coverage   85.71%   83.45%   -2.27%     
==========================================
  Files           4        5       +1     
  Lines         105      139      +34     
  Branches       19       25       +6     
==========================================
+ Hits           90      116      +26     
- Misses         11       16       +5     
- Partials        4        7       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.