feat: add westminster city council scraper#2086
Conversation
Westminster uses a USRN-based street schedule at transact.westminster.gov.uk. Parses rubbish and recycling tables, converts day-of-week schedules to concrete collection dates. Pure HTTP - no Selenium needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
Your plan currently allows 2 reviews/hour. Refill in 21 minutes and 17 seconds. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more review capacity refills, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than trial, open-source, and free plans. In all cases, review capacity refills continuously over time. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughAdds a WestminsterCityCouncil scraper plus test-data entry. It resolves USRNs from postcodes, parses Westminster street report HTML for weekday-based rubbish and recycling schedules, converts weekday sets into upcoming concrete collection dates, and outputs deduplicated, sorted bin collection entries. ChangesWestminster City Council Support
Sequence DiagramsequenceDiagram
participant Caller as parse_data caller
participant PostcodesIO as postcodes.io
participant Nominatim as Nominatim
participant WestminsterSearch as Westminster street search
participant WestminsterReport as transact.westminster.gov.uk
participant HTMLParser as HTML parser
participant DateHelpers as Date helpers
participant Output as Formatted bins
Caller->>PostcodesIO: lookup(postcode)
PostcodesIO-->>Caller: lat,lng
Caller->>Nominatim: reverse_geocode(lat,lng)
Nominatim-->>Caller: street name
Caller->>WestminsterSearch: search(street)
WestminsterSearch-->>Caller: USRN candidates
Caller->>WestminsterReport: fetch(street_report?usrn)
WestminsterReport-->>HTMLParser: HTML page
HTMLParser-->>Caller: rubbish/recycling weekday text
Caller->>DateHelpers: parse days & generate dates
DateHelpers-->>Caller: concrete collection dates
Caller->>Output: dedupe, format, sort -> bins list
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related issues
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2086 +/- ##
=======================================
Coverage 86.67% 86.67%
=======================================
Files 9 9
Lines 1141 1141
=======================================
Hits 989 989
Misses 152 152 ☔ View full report in Codecov by Sentry. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@uk_bin_collection/tests/input.json`:
- Line 882: The "wiki_note" JSON value contains a Unicode replacement character
(�); replace that character with a proper em-dash (U+2014) or a simple hyphen,
e.g. change the string containing "you can use [FindMyAddress]... of your
property�you can use" to use "—" (or "-" if preferred) and save the file with
UTF-8 encoding (or escape the em-dash as "\u2014" if you must keep ASCII-only
escapes).
In `@uk_bin_collection/uk_bin_collection/councils/WestminsterCityCouncil.py`:
- Around line 17-18: The file docstring/comment incorrectly states "next 7 days"
while the constant LOOKAHEAD_DAYS is set to 14; update the comment(s) (including
the header comment and the line around 38) to accurately reflect the actual
behavior (e.g., "outputs the next 14 days of collection dates") or change
LOOKAHEAD_DAYS to 7 if the intended behavior was 7 days; locate references to
LOOKAHEAD_DAYS and the top-of-file comment in WestminsterCityCouncil.py and make
the text consistent with the chosen value.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 8cbe53e6-2f70-4bd9-8f41-1209c2295395
📒 Files selected for processing (2)
uk_bin_collection/tests/input.jsonuk_bin_collection/uk_bin_collection/councils/WestminsterCityCouncil.py
Adds 3-step USRN resolution: postcodes.io for lat/lng, Nominatim for street name, Westminster dropdown for USRN match. No API keys needed. Falls back to direct USRN if uprn kwarg provided. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
uk_bin_collection/uk_bin_collection/councils/WestminsterCityCouncil.py (1)
222-237:⚠️ Potential issue | 🟠 Major | ⚡ Quick winFail on unsupported weekday tokens instead of partially parsing them.
The
part[:3]fallback turns any token that merely starts with a weekday into a valid match. For example,"Mon - Fri except bank holidays"becomes just Monday, and other unknown fragments are silently dropped. That will generate wrong collection dates instead of detecting a format change. Validate each non-empty token against the supported formats and raise on anything else. Based on learnings: inuk_bin_collection/**/*.py, prefer explicit failures on unexpected formats over silent defaults or swallowed errors.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@uk_bin_collection/uk_bin_collection/councils/WestminsterCityCouncil.py` around lines 222 - 237, The parser currently accepts any token that starts with a weekday because of the fallback using part[:3], which silently swallows unexpected text; change the logic in WestminsterCityCouncil.py (the loop that fills days, the inner_range handling and the fallback using DAY_ABBREV_MAP) so that after splitting and stripping each part you only accept two forms: a matched range (regex ^(\w{3})\s*[-–]\s*(\w{3})$ already handled) or an exact weekday token that maps to DAY_ABBREV_MAP (case-insensitive match of the full token or its 3-letter abbreviation), and for any non-empty token that does not match either form raise an exception (e.g. ValueError or a ParseError) instead of silently ignoring it; ensure you reference DAY_ABBREV_MAP and the existing days list/inner_range variables when implementing the validation.
🧹 Nitpick comments (2)
uk_bin_collection/tests/input.json (2)
2886-2886: 💤 Low valueConsider clarifying USRN terminology for user understanding.
The wiki_note correctly explains the functionality, but users unfamiliar with UK addressing standards might not understand that Westminster uses USRN (Unique Street Reference Number) values instead of UPRN (Unique Property Reference Number) values. Consider explicitly noting this distinction, for example: "Pass postcode for auto-resolution, or pass USRN (street reference, not property UPRN) directly as uprn parameter."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@uk_bin_collection/tests/input.json` at line 2886, Update the "wiki_note" text to explicitly clarify that Westminster expects USRN (Unique Street Reference Number) values rather than UPRN (Unique Property Reference Number) so users understand the distinction; modify the wiki_note string (key "wiki_note" in the test JSON) to read something like: "Pass postcode for auto USRN resolution, or pass USRN (Unique Street Reference Number — street-level, not property-level UPRN) directly as uprn. Street-level (not address-level) collection schedules." ensuring the message clearly calls out USRN vs UPRN.
2882-2882: 💤 Low valueConsider using a test postcode that maps to a verified USRN.
The postcode "SW1P 3BU" is valid, but the PR test plan documents testing with specific USRNs: 8401405 (Marylebone High Street), 8401123 (Baker Street), and 8400910 (Wood's Mews). Using a postcode that resolves to one of these tested USRNs would better align the test configuration with the verified functionality.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@uk_bin_collection/tests/input.json` at line 2882, Replace the "postcode" value in the test input (the "postcode" JSON field) with a postcode that maps to one of the verified USRNs used in the PR test plan (8401405, 8401123, or 8400910) so the test aligns with documented verification; update the "postcode" string to a known postcode that resolves to one of those USRNs (e.g., use the postcode that maps to USRN 8401123 or 8401405) and ensure the JSON remains valid.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@uk_bin_collection/uk_bin_collection/councils/WestminsterCityCouncil.py`:
- Around line 136-147: The code collapses distinct streets when building options
because options currently maps a normalized key to a single (name, USRN); change
options to map each _normalise_street(txt) to a list of (txt, val) by appending
from the loop over select.find_all("option"). When checking the exact-match path
using road_norm and options[road_norm], if the list has length 1 return that
USRN, but if it contains multiple entries do not blindly return the last
one—instead try to disambiguate by finding an entry whose original txt equals
the input road (case-insensitive) and return that USRN if found; otherwise
preserve the ambiguity by not returning and allowing the later fuzzy/alternate
matching logic to run.
---
Outside diff comments:
In `@uk_bin_collection/uk_bin_collection/councils/WestminsterCityCouncil.py`:
- Around line 222-237: The parser currently accepts any token that starts with a
weekday because of the fallback using part[:3], which silently swallows
unexpected text; change the logic in WestminsterCityCouncil.py (the loop that
fills days, the inner_range handling and the fallback using DAY_ABBREV_MAP) so
that after splitting and stripping each part you only accept two forms: a
matched range (regex ^(\w{3})\s*[-–]\s*(\w{3})$ already handled) or an exact
weekday token that maps to DAY_ABBREV_MAP (case-insensitive match of the full
token or its 3-letter abbreviation), and for any non-empty token that does not
match either form raise an exception (e.g. ValueError or a ParseError) instead
of silently ignoring it; ensure you reference DAY_ABBREV_MAP and the existing
days list/inner_range variables when implementing the validation.
---
Nitpick comments:
In `@uk_bin_collection/tests/input.json`:
- Line 2886: Update the "wiki_note" text to explicitly clarify that Westminster
expects USRN (Unique Street Reference Number) values rather than UPRN (Unique
Property Reference Number) so users understand the distinction; modify the
wiki_note string (key "wiki_note" in the test JSON) to read something like:
"Pass postcode for auto USRN resolution, or pass USRN (Unique Street Reference
Number — street-level, not property-level UPRN) directly as uprn. Street-level
(not address-level) collection schedules." ensuring the message clearly calls
out USRN vs UPRN.
- Line 2882: Replace the "postcode" value in the test input (the "postcode" JSON
field) with a postcode that maps to one of the verified USRNs used in the PR
test plan (8401405, 8401123, or 8400910) so the test aligns with documented
verification; update the "postcode" string to a known postcode that resolves to
one of those USRNs (e.g., use the postcode that maps to USRN 8401123 or 8401405)
and ensure the JSON remains valid.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b04c4efa-1fb3-469e-aa96-d568c4f713cf
📒 Files selected for processing (2)
uk_bin_collection/tests/input.jsonuk_bin_collection/uk_bin_collection/councils/WestminsterCityCouncil.py
1 participant
Summary
transact.westminster.gov.uk, not the standard UPRN systemNotes
uprnparameterTest plan
Summary by CodeRabbit
New Features
Tests