Two ways to scrape IMDb, and when to use each. IMDb publishes structured data inside every title page, so for a handful of movies you can pull it for free with no proxy at all. Past a few requests, IMDb starts blocking you, and you need an IMDb scraper that rotates residential proxies. This repo shows both, so you know how to scrape IMDb data at any scale.
You need a ScrapingBee API key for the production method. The free tier gives you 1,000 credits with no card required: scrapingbee.com.
Everything on IMDb hangs off a title id, the tt code in the URL. Inception is tt1375666, so its page is:
https://www.imdb.com/title/tt1375666/
Grab the id from any IMDb URL and you can address the movie directly. That is the anchor for both methods below.
IMDb embeds a JSON-LD block (<script type="application/ld+json">) in every title page with the title, rating, genres, cast, and director already structured. For a small job you can read it directly, no API needed.
import requests
from bs4 import BeautifulSoup
import json
def scrape_imdb_title(title_id: str) -> dict:
url = f"https://www.imdb.com/title/{title_id}/"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9"
}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, "html.parser")
json_ld = soup.find("script", {"type": "application/ld+json"})
if not json_ld:
return {}
data = json.loads(json_ld.string)
return {
"title": data.get("name"),
"year": data.get("datePublished", "")[:4],
"rating": data.get("aggregateRating", {}).get("ratingValue"),
"genres": data.get("genre", []),
"description": data.get("description"),
"directors": [d.get("name") for d in data.get("director", [])],
"actors": [a.get("name") for a in data.get("actor", [])],
}
print(scrape_imdb_title("tt0111161"))This works until it does not. IMDb rate-limits and blocks datacenter and repeated traffic, so the moment you loop over more than a few titles, the requests start coming back empty or challenged. That is the point where you move to Method 2.
Send the same IMDb URL through ScrapingBee with premium_proxy on and let extract_rules return the fields as JSON. The proxy rotation is what keeps you from getting blocked, so this is the method that scales.
import requests
import json
API_KEY = 'YOUR_SCRAPINGBEE_API_KEY'
url = 'https://www.imdb.com/title/tt1375666/'
params = {
'api_key': API_KEY,
'url': url,
'premium_proxy': 'true',
'extract_rules': json.dumps({
"title": {"selector": "h1", "type": "text"},
"rating": {"selector": "span[role='img']", "type": "text"},
"genres": {"selector": "div[data-testid='genres'] a", "type": "list"},
"summary": {"selector": "span[data-testid='plot-xl']", "type": "text"},
"director": {"selector": "li[data-testid='title-pc-principal-credit']:first-child a", "type": "text"}
})
}
response = requests.get('https://app.scrapingbee.com/api/v1', params=params)
data = response.json()
print(json.dumps(data, indent=2))Example response:
{
"title": "Inception",
"rating": "8.8/10",
"genres": ["Action", "Adventure", "Sci-Fi"],
"summary": "A thief who steals corporate secrets...",
"director": "Christopher Nolan"
}The selectors above come from ScrapingBee's IMDb guide and target the current title-page markup. Update them if IMDb changes its layout, or switch to ai_extract_rules for natural-language extraction.
curl "https://app.scrapingbee.com/api/v1?api_key=YOUR_API_KEY&url=https%3A%2F%2Fwww.imdb.com%2Ftitle%2Ftt1375666%2F&premium_proxy=true"const axios = require('axios');
axios.get('https://app.scrapingbee.com/api/v1', {
params: {
api_key: 'YOUR_API_KEY',
url: 'https://www.imdb.com/title/tt1375666/',
premium_proxy: 'true',
extract_rules: JSON.stringify({
title: { selector: 'h1', type: 'text' },
rating: { selector: "span[role='img']", type: 'text' },
genres: { selector: "div[data-testid='genres'] a", type: 'list' },
}),
},
}).then((response) => console.log(response.data));- JSON-LD (Method 1): free, no dependencies beyond
requestsand BeautifulSoup, perfect for a few titles or a one-off script. Breaks under volume. - ScrapingBee API (Method 2): handles the blocking so you can loop over hundreds of titles, and returns exactly the fields you define. This is the answer to how to scrape IMDb data at scale.
A practical pattern: prototype with Method 1 to confirm the fields you want, then move the same title ids to Method 2 for the real run.
ScrapingBee bills successful requests. An IMDb title request with premium_proxy and the default JavaScript rendering costs 25 credits. CSS extract_rules adds nothing; ai_extract_rules adds 5. Current rate card: ScrapingBee pricing.
- Film datasets for analysis or a side project, built from a list of title ids.
- Ratings and genre research across a catalog.
- Recommendation or ML training data with structured title, rating, cast, and genre.
- Monitoring a title's rating or metadata over time.
Route the request through rotating residential proxies. The free JSON-LD method works for a few titles, but at volume you need the API method with premium_proxy so IMDb does not block you.
In a JSON-LD block inside each title page. Method 1 reads it directly. Method 2 uses CSS selectors through ScrapingBee, which is more robust when you need many pages.
The tt code in the URL, for example tt1375666 for Inception. Both methods take that id or the full /title/tt.../ URL.
Public data is generally collectible for research and analysis, but IMDb's terms and local rules apply. Scrape public pages only, and do not scrape behind a login.
MIT