PDDikti Mahasiswa Scraper

Streamlit application for collecting public PDDikti student search results from a NIM pattern, filtering them by university and study program, deduplicating by NIM, and exporting clean CSV files for research or data validation.

Features

• Search student records from public PDDikti API endpoints.
• Generate NIM candidates from a prefix and numeric range.
• Filter results by university name and study program.
• Deduplicate records using NIM as the primary key.
• Infer student cohort year from the NIM pattern when possible.
• Show cohort distribution and scraping statistics.
• Export results to CSV with Excel-friendly UTF-8 encoding.
• Use configurable concurrency, delay, timeout, and API endpoint settings.

Tech Stack

• Python 3.10+
• Streamlit
• aiohttp
• requests
• pandas

Project Structure

.
├── api_client.py        # PDDikti API client and connection test
├── app.py               # Streamlit user interface
├── export.py            # CSV export utilities
├── scraper.py           # Async scraping, filtering, deduplication, stats
├── requirements.txt     # Python dependencies
└── tests/               # Unit tests

Installation

Clone the repository and install dependencies:

git clone https://github.com/your-username/pddiktiscrap.git
cd pddiktiscrap
python -m venv .venv

Activate the virtual environment.

Windows PowerShell:

.venv\Scripts\Activate.ps1

macOS or Linux:

source .venv/bin/activate

Install dependencies:

pip install -r requirements.txt

Usage

Start the Streamlit app:

streamlit run app.py

Open the local Streamlit URL shown in the terminal, then:

1. Select an API endpoint.
2. Set concurrency, delay, and timeout.
3. Enter the university name, study program, education level, NIM prefix, and numeric range.
4. Click Mulai scraping.
5. Review the result table and download the CSV.

Example

If the prefix is 13525 and the range is 1 to 200, the application checks:

135250001
135250002
...
135250200

CSV Output

The exported CSV includes columns when available:

• NIM
• Nama Mahasiswa
• Angkatan
• Program Studi
• Jenjang
• Perguruan Tinggi
• Singkatan PT
• ID PDDikti
• Waktu Scraping

Rows are sorted by Angkatan and NIM.

Testing

Run the unit tests:

python -m unittest discover -s tests

Run a syntax check:

python -m py_compile api_client.py scraper.py export.py app.py

Responsible Use

This project uses public API endpoints and should be used responsibly:

• Use reasonable concurrency and delay values.
• Do not overload public services.
• Do not use the tool for harassment, doxxing, credential discovery, or unlawful profiling.
• Treat exported data according to applicable privacy, academic, and institutional policies.
• Verify important findings against official sources before publication or decision-making.

Notes

PDDikti data and third-party API mirrors may be incomplete, delayed, cached differently, or temporarily unavailable. Duplicate records can occur because multiple API responses may return the same NIM with different internal identifiers. This application preserves only the first record for each NIM in the primary output and reports how many duplicates were removed.

License

Add your preferred license before publishing this repository. If unsure, MIT is a common choice for small open-source utility projects.