Skip to content

Commit 17e36b8

Browse files
committed
fix: Enhance error handling in scraper for connection, timeout, and SSL errors
1 parent a29c0a1 commit 17e36b8

File tree

1 file changed

+6
-0
lines changed

1 file changed

+6
-0
lines changed

base.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -244,6 +244,12 @@ def fetch_page(self, url: str, headers: dict = None) -> requests.Response:
244244
except requests.exceptions.ConnectionError as e:
245245
logger.error(f"Connection error occurred: {url} - {str(e)}")
246246
time.sleep(3.5)
247+
except requests.exceptions.Timeout as e:
248+
logger.error(f"Timeout error occurred: {url} - {str(e)}")
249+
time.sleep(3.5)
250+
except requests.exceptions.SSLError as e:
251+
logger.error(f"SSL error occurred: {url} - {str(e)}")
252+
time.sleep(3.5)
247253
def parse_html(self, content: str):
248254
return bs(content, "lxml")
249255

0 commit comments

Comments
 (0)