BPOM Data Scraper

A Python web scraper to collect product registration data from the Indonesian Food and Drug Authority (BPOM) website and store it in a PostgreSQL database.

Features

• Scrapes 639,024+ product records from BPOM website
• Handles pagination automatically
• Stores data in PostgreSQL database
• Progress tracking with progress bar
• Error handling and retry mechanism
• Batch processing for efficient database operations

Installation

1. Clone the repository
2. Install dependencies:

pip install -r requirements.txt

3. Create .env file from .env.example:

cp .env.example .env

4. Update .env with your database credentials if different

Usage

Run the scraper:

python main.py

The scraper will:

1. Connect to the PostgreSQL database
2. Create the necessary table if it doesn't exist
3. Fetch all product data from BPOM API
4. Store the data in the database with batch processing
5. Show progress with a progress bar

Database Schema

The scraper creates a table bpom_products with all fields from the BPOM API response.

Configuration

Edit .env file to configure:

• DATABASE_URL: PostgreSQL connection string
• BATCH_SIZE: Number of records to insert per batch (default: 100)
• REQUEST_TIMEOUT: HTTP request timeout in seconds (default: 30)
• MAX_RETRIES: Maximum number of retry attempts (default: 3)

Data Source

Data is scraped from: https://cekbpom.pom.go.id/produk-dt/all