Update Rust extension documentation to indicate experimental status

- Add prominent warning that this is NOT PRODUCTION READY
- Document test status: 86 core tests passing, ~85 tests skipped
- List unimplemented features:
  - Custom type encoders (TypeEncoder, TypeRegistry, FallbackEncoder)
  - RawBSONDocument codec options
  - Some DBRef edge cases
  - Complete type checking support
- Document skip mechanism using @skip_if_rust_bson marker
- Update module structure documentation (6 modules)
- Change status from 'production-ready' to 'experimental'
- Clarify performance limitations (~5x slower than C)
- Add clear recommendation to use C extension for production

Author: aclark4life

bson/_rbson/README.md

@@ -1,43 +1,87 @@
 # Rust BSON Extension Module
 
+⚠️ **NOT PRODUCTION READY** - This is an experimental implementation with incomplete feature support and performance limitations. See [Test Status](#test-status) and [Performance Analysis](#performance-analysis) sections below.
+
 This directory contains a Rust-based implementation of BSON encoding/decoding for PyMongo, developed as part of [PYTHON-5683](https://jira.mongodb.org/browse/PYTHON-5683).
 
 ## Overview
 
-The Rust extension (`_rbson`) provides the same interface as the C extension (`_cbson`) but is implemented in Rust using:
+The Rust extension (`_rbson`) provides a **partial implementation** of the C extension (`_cbson`) interface, implemented in Rust using:
 - **PyO3**: Python bindings for Rust
 - **bson crate**: MongoDB's official Rust BSON library
 - **Maturin**: Build tool for Rust Python extensions
 
+## Test Status
+
+### ✅ Core BSON Tests: 86 passed, 2 skipped
+The basic BSON encoding/decoding functionality works correctly (`test/test_bson.py`).
+
+### ⏭️ Skipped Tests: ~85 tests across multiple test files
+The following features are **not implemented** and tests are skipped when using the Rust extension:
+
+#### Custom Type Encoders (test/test_custom_types.py)
+- **`TypeEncoder` and `TypeRegistry`** - Custom type encoding/decoding
+- **`FallbackEncoder`** - Fallback encoding for unknown types
+- **Tests skipped**: All tests in `TestBSONFallbackEncoder`, `TestCustomPythonBSONTypeToBSONMonolithicCodec`, `TestCustomPythonBSONTypeToBSONMultiplexedCodec`
+- **Reason**: Rust extension doesn't support custom type encoders or fallback encoders
+
+#### RawBSONDocument (test/test_raw_bson.py)
+- **`RawBSONDocument` codec options** - Raw BSON document handling
+- **Tests skipped**: All tests in `TestRawBSONDocument`
+- **Reason**: Rust extension doesn't implement RawBSONDocument codec options
+
+#### DBRef Edge Cases (test/test_dbref.py)
+- **DBRef validation and edge cases**
+- **Tests skipped**: Some DBRef tests
+- **Reason**: Incomplete DBRef handling in Rust extension
+
+#### Type Checking (test/test_typing.py)
+- **Type hints and mypy validation**
+- **Tests skipped**: Some typing tests
+- **Reason**: Type checking issues with Rust extension
+
+### Skip Mechanism
+Tests are skipped using the `@skip_if_rust_bson` pytest marker defined in `test/__init__.py`:
+```python
+skip_if_rust_bson = pytest.mark.skipif(
+    _use_rust_bson(), reason="Rust BSON extension does not support this feature"
+)
+```
+
+This marker is applied to test classes and methods that use unimplemented features.
+
 ## Implementation History
 
 This implementation was developed through [PR #2695](https://github.com/mongodb/mongo-python-driver/pull/2695) to investigate using Rust as an alternative to C for Python extension modules.
 
 ### Key Milestones
 
-1. **Initial Implementation** - Complete BSON type support with 100% test compatibility (88/88 tests passing)
+1. **Initial Implementation** - Basic BSON type support with core functionality
 2. **Performance Optimizations** - Type caching, fast paths for common types, direct byte operations
-3. **Architectural Analysis** - Identified fundamental performance differences between Rust and C approaches
+3. **Modular Refactoring** - Split monolithic lib.rs into 6 well-organized modules
+4. **Test Integration** - Added skip markers for unimplemented features (~85 tests skipped)
 
 ## Features
 
 ### Supported BSON Types
 
-The Rust extension supports all BSON types:
+The Rust extension supports basic BSON types:
 - **Primitives**: Double, String, Int32, Int64, Boolean, Null
 - **Complex Types**: Document, Array, Binary, ObjectId, DateTime
 - **Special Types**: Regex, Code, Timestamp, Decimal128, MinKey, MaxKey
 - **Deprecated Types**: DBPointer (decodes to DBRef)
 
 ### CodecOptions Support
 
-Full support for PyMongo's `CodecOptions`:
-- `document_class` - Custom document classes
-- `tz_aware` - Timezone-aware datetime handling
-- `tzinfo` - Timezone conversion
-- `uuid_representation` - UUID encoding/decoding modes
-- `datetime_conversion` - DateTime handling modes (AUTO, CLAMP, MS)
-- `unicode_decode_error_handler` - UTF-8 error handling
+**Partial** support for PyMongo's `CodecOptions`:
+- ✅ `document_class` - Custom document classes (basic support)
+- ✅ `tz_aware` - Timezone-aware datetime handling
+- ✅ `tzinfo` - Timezone conversion
+- ✅ `uuid_representation` - UUID encoding/decoding modes
+- ✅ `datetime_conversion` - DateTime handling modes (AUTO, CLAMP, MS)
+- ✅ `unicode_decode_error_handler` - UTF-8 error handling
+- ❌ `type_registry` - Custom type encoders/decoders (NOT IMPLEMENTED)
+- ❌ RawBSONDocument support (NOT IMPLEMENTED)
 
 ### Runtime Selection
 
@@ -144,8 +188,8 @@ The Copilot POC's claimed 2.89x speedup was likely due to:
 
 When these missing features were added to achieve 100% compatibility, the true performance cost of the Rust `Bson` enum architecture became apparent.
 
-### Current Implementation (PR #2695) - Production-Ready
-**Status**: 88/88 tests passing (100%)
+### Current Implementation (PR #2695) - Experimental
+**Status**: 86/88 core BSON tests passing, ~85 feature tests skipped
 
 **Build System**: `maturin build --release` (proper wheel generation)
 - Uses Maturin for proper Python packaging
@@ -154,9 +198,9 @@ When these missing features were added to achieve 100% compatibility, the true p
 - Located in `bson/_rbson/` directory (proper module structure)
 
 **Improvements over Copilot POC:**
-- ✅ **100% test compatibility** (88/88 vs 53/88)
-- ✅ **Complete CodecOptions support**:
-  - `document_class` - Custom document classes
+- ✅ **Core BSON functionality** (86/88 tests passing in test_bson.py)
+- ✅ **Basic CodecOptions support**:
+  - `document_class` - Custom document classes (basic support)
   - `tzinfo` - Timezone conversion with astimezone()
   - `datetime_conversion` - All modes (AUTO, CLAMP, MS)
   - `unicode_decode_error_handler` - Fallback to Python for non-strict handlers
@@ -166,22 +210,24 @@ When these missing features were added to achieve 100% compatibility, the true p
   - Fast paths for common types (int, str, bool, None)
   - Direct byte operations where possible
   - PyDict fast path with pre-allocation
-- ✅ **Production-ready error handling** (matches C extension error messages exactly)
+- ✅ **Modular code structure** (6 well-organized Rust modules)
 - ✅ **Proper module structure** (`bson/_rbson/` with build.sh and maturin)
 - ✅ **Runtime selection** via PYMONGO_USE_RUST environment variable
-- ✅ **Comprehensive testing** (cross-compatibility tests, performance benchmarks)
+- ✅ **Test skip markers** for unimplemented features
 - ✅ **Same Rust architecture**: PyO3 0.23 + bson 2.13 crate (Python → Bson enum → bytes)
 
+**Missing Features** (see [Test Status](#test-status)):
+- ❌ **Custom type encoders** (`TypeEncoder`, `TypeRegistry`, `FallbackEncoder`)
+- ❌ **RawBSONDocument** codec options
+- ❌ **Some DBRef edge cases**
+- ❌ **Complete type checking support**
+
 **Performance Reality**: ~0.21x (5x slower than C) - see Performance Analysis section
 
 **Key Insights**:
 1. **Same Architecture, Different Results**: Both implementations use the same Rust architecture (PyO3 + bson crate with intermediate `Bson` enum), so the build system (cargo vs maturin) is not the cause of the performance difference.
-2. **Incomplete vs Complete**: The POC's speed claims were based on incomplete functionality (60% test pass rate). Achieving 100% compatibility revealed the true performance cost of:
-   - Complete CodecOptions handling (timezone conversions, datetime modes, etc.)
-   - BSON validation (size checks, null terminators, extra bytes)
-   - Production-ready error handling
-   - Edge case handling for all 88 tests
-3. **The Fundamental Issue**: Both implementations suffer from the same architectural limitation (Python → Bson enum → bytes), but it only becomes a significant bottleneck when you implement all the features required for production use.
+2. **Incomplete Implementation**: The current implementation has ~85 tests skipped due to unimplemented features (custom type encoders, RawBSONDocument, etc.). This is an experimental implementation, not production-ready.
+3. **The Fundamental Issue**: The Rust architecture (Python → Bson enum → bytes) has inherent performance limitations compared to the C extension's direct byte-writing approach.
 
 ## Direct Byte-Writing Performance Results
 
@@ -317,34 +363,70 @@ maturin develop --release
 
 ## Testing
 
-Run the test suite with the Rust extension:
+Run the core BSON test suite with the Rust extension:
+```bash
+PYMONGO_USE_RUST=1 python -m pytest test/test_bson.py -v
+# Expected: 86 passed, 2 skipped
+```
+
+Run all tests (including skipped tests):
 ```bash
-PYMONGO_USE_RUST=1 python -m pytest test/
+PYMONGO_USE_RUST=1 python -m pytest test/ -v
+# Expected: Many tests passed, ~85 tests skipped due to unimplemented features
 ```
 
 Run performance benchmarks:
 ```bash
 python test/performance/perf_test.py
 ```
 
+## Module Structure
+
+The Rust codebase is organized into 6 well-structured modules (refactored from a single 3,117-line file):
+
+- **`lib.rs`** (76 lines) - Module exports and public API
+- **`types.rs`** (266 lines) - Type cache and BSON type markers
+- **`errors.rs`** (56 lines) - Error handling utilities
+- **`utils.rs`** (154 lines) - Utility functions (datetime, regex, validation)
+- **`encode.rs`** (1,545 lines) - BSON encoding functions
+- **`decode.rs`** (1,141 lines) - BSON decoding functions
+
+This modular structure improves:
+- Code organization and maintainability
+- Compilation times (parallel module compilation)
+- Code navigation and testing
+- Clear separation of concerns
+
 ## Conclusion
 
 The Rust extension demonstrates that:
-1. ✅ **Rust can provide a complete, production-ready BSON implementation**
-2. ✅ **100% compatibility with existing tests and APIs is achievable**
+1. ✅ **Rust can provide basic BSON encoding/decoding functionality**
+2. ❌ **Complete feature parity with C extension is not achieved** (~85 tests skipped)
 3. ❌ **Performance parity with C requires bypassing the `bson` crate**
 4. ❌ **The engineering effort may not justify the benefits**
 
 ### Recommendation
 
-The Rust extension is **production-ready** from a correctness standpoint but **not recommended** for performance-critical applications. The C extension remains the better choice for performance.
+⚠️ **NOT PRODUCTION READY** - The Rust extension is **experimental** and has significant limitations:
+
+**Missing Features:**
+- Custom type encoders (`TypeEncoder`, `TypeRegistry`, `FallbackEncoder`)
+- RawBSONDocument codec options
+- Some DBRef edge cases
+- Complete type checking support
+
+**Performance Issues:**
+- ~5x slower than C extension (0.21x performance)
+- Even with direct byte-writing optimizations, still ~2.3x slower (0.43x performance)
 
 **Use Cases for Rust Extension:**
-- Platforms where C compilation is difficult (e.g., WebAssembly)
-- Development environments without C toolchain
-- Testing and validation purposes
+- **Experimental/research purposes only**
+- Testing Rust-Python interop with PyO3
+- Platforms where C compilation is difficult (with caveats about missing features)
 - Future exploration if `bson` crate performance improves
 
+**For production use, the C extension (`_cbson`) is strongly recommended.**
+
 For more details, see:
 - [PYTHON-5683 JIRA ticket](https://jira.mongodb.org/browse/PYTHON-5683)
 - [PR #2695](https://github.com/mongodb/mongo-python-driver/pull/2695)

bson/_rbson/src/lib.rs

@@ -14,18 +14,28 @@
 
 //! Rust implementation of BSON encoding/decoding functions
 //!
-//! This module provides the same interface as the C extension (bson._cbson)
-//! but implemented in Rust using PyO3 and the bson library.
+//! ⚠️ **NOT PRODUCTION READY** - Experimental implementation with incomplete features.
+//!
+//! This module provides a **partial implementation** of the C extension (bson._cbson)
+//! interface, implemented in Rust using PyO3 and the bson library.
+//!
+//! # Implementation Status
+//!
+//! - ✅ Core BSON encoding/decoding: 86/88 tests passing
+//! - ❌ Custom type encoders: NOT IMPLEMENTED (~85 tests skipped)
+//! - ❌ RawBSONDocument: NOT IMPLEMENTED
+//! - ❌ Performance: ~5x slower than C extension
 //!
 //! # Implementation History
 //!
 //! This implementation was developed as part of PYTHON-5683 to investigate
 //! using Rust as an alternative to C for Python extension modules.
 //!
 //! See PR #2695 for the complete implementation history, including:
-//! - Initial implementation with 100% test compatibility
+//! - Initial implementation with core BSON functionality
 //! - Performance optimizations (type caching, fast paths, direct conversions)
-//! - Architectural analysis comparing Rust vs C extension approaches
+//! - Modular refactoring (split into 6 modules)
+//! - Test skip markers for unimplemented features
 //!
 //! # Performance
 //!
@@ -59,7 +69,7 @@ fn _test_rust_extension(py: Python) -> PyResult<PyObject> {
     let result = PyDict::new(py);
     result.set_item("implementation", "rust")?;
     result.set_item("version", "0.1.0")?;
-    result.set_item("status", "production-ready")?;
+    result.set_item("status", "experimental")?;
     result.set_item("pyo3_version", env!("CARGO_PKG_VERSION"))?;
     Ok(result.into())
 }