`minn` scraper sometimes stuck in captcha challenge

[grossir]

See an example of the saved HTML in S3. We should ask them to whitelist our IP? If they won't answer, and this is an automated challenge, we should have a subset of scrapers that makes requests every 24 hours, instead of every single hour

This is making our coverage spotty. For example, from the 5 most recent opinions we only have 2

I was trying to run ./manage.py cl_back_scrape_citations --courts juriscraper.opinions.united_states.state.minn --backscrape-start=2024/10/01 --backscrape-end=2024/10/20 --verbosity 3 for #858 (comment) and noticed suspicious 0 results

[flooie]

They have been generally unresponsive. Perhaps we need a mixture of less often and slower?

[flooie]

Also - should this be moved to CL - where its likely not the collection of metadata that is an issue its the downloading of the documents right?

[Luis-manzur]

if you fetch minn rate limit rules page its easy to see that the server only allows 10 calls before showing the captcha.

Request-rate: 10 Visit-time: 0000-1200 GMT is 6PM to 6AM here

I asume that the request rate is 10 per minute.

also we could use one of the listed user agents (last resource) not recommended for obvious reasons.

@flooie @grossir

[grossir]

Nice find @Luis-manzur
So this one would be solvable by not making so many requests.
On the server, every scraper is ran once every hour
I think making a special crontab for every site with special rules will be too much maintenance. Maybe we can just handle it in Juriscraper

Given that every scraper calls super().__init__; we could handle it on AbstractSite, with a default behavior of doing nothing

class AbstractSite:
    """Contains generic methods for scraping data. Should be extended by all
    scrapers.

    Should not contain lists that can't be sorted by the _date_sort function.
    """
    
    def rate_limit(self):
        pass

    def __init__(self, cnt=None, **kwargs):
        super().__init__()
        self.rate_limit()

Then, on the scraper

class Site(OpinionSiteLinear):
    court_query = "supct"
    days_interval = 7
    first_opinion_date = date(1998, 1, 1)

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def rate_limit(self):
        # logic to follow robots.txt: 
        # 1. scrape only in allowed hours
        # 2. maybe allow a single scrape in the day; since the request limit is 10
        #    if we allow the regular behaviour, it will try to re-download 
        #    every opinion even if already scraped that day
        #    ..

This way we can control everything through juriscraper. Can you check the robots.txt for the other sites that block us?

[Luis-manzur]

This way we can control everything through juriscraper. Can you check the robots.txt for the other sites that block us?

sure

[flooie]

Could you expand to all of our current sites, I think a survey of all of them would be informative and interesting.