Skip to content

Add selector to create a config for europe #974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
1Maxnet1 wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add selector to create a config for europe #974

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bd48582
Add selector to create a config for europe
1Maxnet1 Mar 6, 2025
404cc40
Allow to also fetch feeds only for european countries
1Maxnet1 Mar 6, 2025
8ebb0eb
Merge branch 'main' into work/maxnet/create-europe-config
1Maxnet1 Jul 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions src/fetch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
UrlSource, Source, Region
from pathlib import Path
from datetime import datetime, timezone
from utils import eprint
from utils import eprint, european_iso_codes
from zipfile import ZipFile
from typing import Optional, Any, Iterable, IO
from zoneinfo import ZoneInfo
Expand Down Expand Up @@ -492,7 +492,17 @@ def fetch(self, metadata: Path) -> int:

fetcher = Fetcher()

errors = fetcher.fetch(Path(arguments.metadata_file))
selector = arguments.metadata_file
errors = 0
if selector == "europe":
Copy link
Collaborator

@jbruechert jbruechert Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We generally have ci/fetch-feeds.py for fetching multiple regions. Maybe create a new similar script for this. Having too many special cases always bears the risk of breaking some of them with every change

Copy link
Contributor Author

@1Maxnet1 1Maxnet1 Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So instead of adding it here, creating a python script generate-continent-config.py ? Would be fine by me as well.

metadata_files = [f"feeds/{code}.json" for code in european_iso_codes]
else:
metadata_files = [Path(selector)]
for metadata_file in metadata_files:
try:
errors += fetcher.fetch(Path(metadata_file))
except FileNotFoundError:
continue
if errors > 0:
eprint(f"Error: {errors} errors occurred during fetching.")
sys.exit(1)
3 changes: 3 additions & 0 deletions src/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@

import sys

european_iso_codes = ['al', 'ad', 'at', 'be', 'ba', 'bg', 'hr', 'cy', 'cz', 'dk', 'ee', 'fi', 'fr', 'de', 'gr',
'hu', 'is', 'ie', 'xk', 'lv', 'li', 'lt', 'lu', 'mt', 'md', 'mc', 'nl', 'mk', 'pl', 'pt',
'ro', 'sm', 'rs', 'sk', 'si', 'es', 'se', 'ch', 'tr', 'ua', 'uk', 'ax', 'fo', 'sj', 'eu']

def eprint(*args, **kwargs):
print(*args, file=sys.stderr, **kwargs)