Skip to content

Add timeouts to remote downloads #418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
massy-o wants to merge 2 commits into
base: master
Choose a base branch
from
+43 −25
Open

Add timeouts to remote downloads #418

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e387183
Add timeouts to remote downloads
massy-o May 14, 2026
8323c2b
Clean up failed remote downloads
massy-o May 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions Prj-Python/hyperlpr3/command/sample.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# -*- coding: utf-8 -*-
import hyperlpr3 as lpr3
import cv2
import urllib
import urllib.error
import urllib.request
import numpy as np
import re
import click
Expand All @@ -27,10 +28,14 @@ def is_http_url(s):

def url_to_image(url):
try:
resp = urllib.request.urlopen(url)
resp = urllib.request.urlopen(url, timeout=10)
image = np.asarray(bytearray(resp.read()), dtype="uint8")
image = cv2.imdecode(image, cv2.IMREAD_COLOR)
except urllib.error.URLError as err:
logger.warning(f"Failed to fetch image from URL ({err}): {url}")
return None
except Exception as err:

@devloai devloai Bot May 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The broad except Exception silently swallows the new urllib.error.URLError (raised on timeout) and returns None, giving the caller no indication of why it failed. Now that a timeout is a real code path, at least log a warning:

import urllib.error  # add at top of file alongside existing `import urllib`

def url_to_image(url):
    try:
        resp = urllib.request.urlopen(url, timeout=10)
        image = np.asarray(bytearray(resp.read()), dtype="uint8")
        image = cv2.imdecode(image, cv2.IMREAD_COLOR)
    except urllib.error.URLError as err:
        logger.warning(f"Failed to fetch image from URL ({err}): {url}")
        return None
    except Exception as err:
        logger.warning(f"Unexpected error reading image from URL: {err}")
        return None
    return image

Note: import urllib alone does not make urllib.error available — it must be imported explicitly.

actions

Feedback: Rate this comment to help me improve future code reviews:

  • 👍 Good - Helpful and accurate
  • 👎 Poor - Wrong, unclear, or unhelpful
  • Skip if you don't have any strong opinions either way.

@massy-o massy-o May 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, accepted in 8323c2b by importing urllib.request/urllib.error explicitly and logging URL fetch failures before returning None.

logger.warning(f"Unexpected error reading image from URL: {err}")
return None

return image
Expand Down
59 changes: 36 additions & 23 deletions Prj-Python/hyperlpr3/config/configuration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,36 +4,50 @@
import os
from .settings import _DEFAULT_FOLDER_, _MODEL_VERSION_, _ONLINE_URL_, _REMOTE_URL_, onnx_model_maps, onnx_runtime_config

REQUEST_TIMEOUT_SECONDS = 30


def down_model_file(url, save_path):
resp = requests.get(url, stream=True)
resp = requests.get(url, stream=True, timeout=REQUEST_TIMEOUT_SECONDS)

@massy-o massy-o May 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review: The shared timeout constant keeps model file and zip downloads consistent, and raise_for_status prevents streaming an error page into the model cache.

@devloai devloai Bot May 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The timeout parameter in requests.get() sets both the connect timeout and the read timeout between chunks, not a total download time limit. For large model files over slow connections, 30 seconds between chunks is probably fine, but it's worth documenting this distinction — callers might expect a wall-clock cap.

More importantly, if the server accepts the connection and starts sending but stalls mid-transfer (e.g. after the first few chunks), the 30s read timeout will eventually fire and raise requests.exceptions.ReadTimeout, but the partial file at save_path will already be written and left on disk in a corrupt state. Consider wrapping the download in a try/except that cleans up the partial file on failure:

def down_model_file(url, save_path):
    resp = requests.get(url, stream=True, timeout=REQUEST_TIMEOUT_SECONDS)
    resp.raise_for_status()
    total = int(resp.headers.get('content-length', 0))
    try:
        with open(save_path, 'wb') as file, tqdm(...) as bar:
            for data in resp.iter_content(chunk_size=1024):
                size = file.write(data)
                bar.update(size)
    except Exception:
        if os.path.exists(save_path):
            os.remove(save_path)
        raise

Same applies to down_model_zip.

actions

Feedback: Rate this comment to help me improve future code reviews:

  • 👍 Good - Helpful and accurate
  • 👎 Poor - Wrong, unclear, or unhelpful
  • Skip if you don't have any strong opinions either way.

@massy-o massy-o May 14, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, accepted in 8323c2b by removing partially written model files if streaming fails before re-raising the download error.

resp.raise_for_status()
total = int(resp.headers.get('content-length', 0))
with open(save_path, 'wb') as file, tqdm(
desc="Pull",
total=total,
unit='iB',
unit_scale=True,
unit_divisor=1024,
) as bar:
for data in resp.iter_content(chunk_size=1024):
size = file.write(data)
bar.update(size)
try:
with open(save_path, 'wb') as file, tqdm(
desc="Pull",
total=total,
unit='iB',
unit_scale=True,
unit_divisor=1024,
) as bar:
for data in resp.iter_content(chunk_size=1024):
size = file.write(data)
bar.update(size)
except Exception:
if os.path.exists(save_path):
os.remove(save_path)
raise


def down_model_zip(url, save_path, is_unzip=False):
resp = requests.get(url, stream=True)
resp = requests.get(url, stream=True, timeout=REQUEST_TIMEOUT_SECONDS)
resp.raise_for_status()
total = int(resp.headers.get('content-length', 0))
name = os.path.join(save_path, os.path.basename(url))
with open(name, 'wb') as file, tqdm(
desc="Pull",
total=total,
unit='iB',
unit_scale=True,
unit_divisor=1024,
) as bar:
for data in resp.iter_content(chunk_size=1024):
size = file.write(data)
bar.update(size)
try:
with open(name, 'wb') as file, tqdm(
desc="Pull",
total=total,
unit='iB',
unit_scale=True,
unit_divisor=1024,
) as bar:
for data in resp.iter_content(chunk_size=1024):
size = file.write(data)
bar.update(size)
except Exception:
if os.path.exists(name):
os.remove(name)
raise

if is_unzip:
f = zipfile.ZipFile(name, "r")
Expand All @@ -60,4 +74,3 @@ def initialization(re_download=False):
if not os.path.exists(models_dir) or re_download:
target_url = os.path.join(_ONLINE_URL_, _MODEL_VERSION_) + '.zip'
down_model_zip(target_url, _DEFAULT_FOLDER_, True)