perf: optimize BNLP — 2-7.4x speedup with Rust-accelerated tokenizer

[Fatin-Ishraq]

🚀 Performance Optimization PR

This PR delivers systematic performance improvements across the BNLP library, achieving 2–7.4× speedup depending on the operation, with an optional Rust extension for maximum performance.

📊 Benchmark Results

Operation	Baseline	Optimized	Speedup
Tokenizer (medium)	110.5 µs	15.0 µs	7.4×
Tokenizer (long)	5.70 ms	0.78 ms	7.3×
Batch Tokenize 1K	20.2 ms	1.3 ms	15.1×
Stopword Lookup (set)	113.6 µs	16.1 µs	7.1×
Remove Punctuations (long)	205.3 µs	89.9 µs	2.3×
Pipeline (long)	29.6 ms	15.0 ms	2.0×
CleanText (medium)	497.7 µs	298.7 µs	1.7×

🔧 What Changed

Rust Extension (Optional — bnlp_rust/)

• New PyO3-based Rust module for hot paths: tokenizer, emoji detection, punctuation, Unicode normalization
• Rust tokenizer eliminates Python-level per-character iteration and unicodedata.category() bridge overhead
• Single-pass clean_text() function that does punctuation + emoji + digit removal in one Rust pass
• Fully optional — if bnlp_rust is not installed, the optimized pure Python fallback is used automatically

Pure Python Optimizations

• BasicTokenizer: Precomputed frozenset of Unicode punctuation codepoints (eliminated 5.2M unicodedata.category() calls), fast-path skip for tokens without punctuation
• remove_punctuations(): Replaced per-character Python loop with precompiled regex — 2.3× faster
• remove_emoji(): Fast-path detection using Rust has_emoji() — skips expensive emoji library for text without emoji
• BengaliCorpus: Added stopwords_set for O(1) membership testing instead of O(n) list scan — 7.1× faster lookups
• BatchProcessor: Fixed to use sequential processing for CPU-bound work — ThreadPool was 30% slower due to GIL
• Lazy imports: BengaliSpellChecker and LanguageDetector now load on-demand via __getattr__ — avoids requiring symspellpy/fasttext at package install

CI / Publishing

• Updated build_and_test.yml: Builds Rust extension before testing, adds pure Python fallback test job, uses pytest
• Updated bnlp-publish-auto.yml: Uses maturin-action to build Rust wheels for Linux/macOS/Windows across Python 3.9–3.13, publishes both pure Python and Rust wheels to PyPI on release

🔒 Backward Compatibility

• All 142 existing tests pass
• Public API is unchanged — same class names, same method signatures, same return types
• Rust is optional — library works perfectly without it (just with pure Python speedups)
• For PyPI release: GitHub Actions builds pre-compiled wheels — users just pip install bnlp, no Rust toolchain needed

📁 Files Changed (22 files, +1,382 / -210 lines)

File	Change
bnlp_rust/ (new)	Rust extension — Cargo.toml, lib.rs (355 lines of PyO3)
bnlp/_rust.py (new)	Centralized Rust availability check with graceful fallback
bnlp/tokenizer/basic.py	Precomputed frozenset, fast-path, Rust integration
bnlp/cleantext/clean.py	Regex punctuation, emoji fast-path, Rust has_emoji
bnlp/corpus/corpus.py	Added stopwords_set for O(1) lookup
bnlp/core/batch.py	Sequential processing for CPU-bound work
bnlp/core/pipeline.py	Integration fixes
bnlp/embedding/	Optimization + error handling
bnlp/spellcheck/spellchecker.py	Refactored with proper OOP
bnlp/utils/registry.py (new)	New registry module
bnlp/__init__.py	Version bump, lazy imports
setup.py	Version bump, extras for rust/all
.github/workflows/	Cross-platform Rust wheel builds + pure Python tests

🧪 How to Test

# Pure Python (no Rust needed)
pip install -r requirements.txt
python -m pytest tests/ -q
With Rust acceleration
cd bnlp_rust && maturin develop --release && cd ..

python -m pytest tests/ -q