Merge realease/v1.1.0 into develop: translations, Dockerfile, benchmark section, changelog

Made-with: Cursor

Author: ManuelSLemos

CHANGELOG.md

@@ -92,6 +92,8 @@ All notable changes to RabbitLLM are documented here.
 - **`samples/`** directory with sample text for long-context testing.
 - **Test suite**: `tests/test_kvcache.py`, `tests/test_kvikio_loader.py`, `tests/test_profiler.py`,
   `tests/test_base_model.py`, and 6 additional test modules.
+- **Dockerfile** for easier deployment: build from repo with `docker build -t rabbitllm .`, run with `--gpus all` for GPU inference. Installs RabbitLLM with optional `[gds]` extra. README Docker subsection documents build, run, and env vars (`HF_TOKEN`, `HF_HOME`). Makefile targets `docker-build` and `docker-run`.
+- **Benchmark section** in README: table of benchmark scripts (GDS/long-context, CPU vs CUDA, attention comparison) and link to [docs/BENCHMARK_HISTORY.md](docs/BENCHMARK_HISTORY.md) for detailed 72B results.
 
 ### Changed
 - **Pipeline extracted to `engine/pipeline.py`**: three strategies — `_no_prefetch_pipeline`,
@@ -106,6 +108,7 @@ All notable changes to RabbitLLM are documented here.
   summary line (e.g. `[offload_small_layers=True, prefetch_pin_memory=False]`).
 - README documents `use_gds`, `kv_cache_dir`, `offload_small_layers`, `cache_layers`, and
   the optional `[gds]` extra.
+- **Documentation and in-repo text translated to English**: `docs/TRANSFORMERS_UPGRADE_PLAN.md`, `docs/BENCHMARK_HISTORY.md`, `docs/COMPATIBILITY.md`, `docs/TROUBLESHOOTING.md`, and `example.py` (docstring and comments).
 
 ### Fixed
 - **RoPE decode correctness**: prefill `position_embeddings_cache` is no longer reused in decode

Dockerfile

@@ -0,0 +1,20 @@
+# RabbitLLM — layer-streaming inference for 70B+ LLMs on consumer GPUs
+# Build: docker build -t rabbitllm .
+# Run (GPU): docker run --gpus all -it rabbitllm python scripts/inference_example.py --model Qwen/Qwen2.5-0.5B-Instruct
+# Run (help): docker run --rm rabbitllm
+
+FROM python:3.12-slim
+
+WORKDIR /app
+
+# Install RabbitLLM from the build context with optional GDS (GPU Direct Storage) support.
+# For Flash Attention, use a image with CUDA and install rabbitllm[flash] separately.
+COPY pyproject.toml README.md ./
+COPY src/ src/
+COPY scripts/ scripts/
+COPY example.py ./
+
+RUN pip install --no-cache-dir -e ".[gds]"
+
+# Default: show inference script help (override with full command)
+CMD ["python", "scripts/inference_example.py", "--help"]

Makefile

@@ -1,4 +1,4 @@
-.PHONY: install dev lint format test test-cov typecheck clean bash
+.PHONY: install dev lint format test test-cov typecheck clean bash docker-build docker-run
 
 install:
 	uv sync --extra gds
@@ -23,5 +23,11 @@ typecheck:
 clean:
 	rm -rf build/ dist/ *.egg-info .pytest_cache .ruff_cache .mypy_cache htmlcov/ .coverage .coverage.*
 
+docker-build:
+	docker build -t rabbitllm .
+
+docker-run:
+	docker run --gpus all --rm -it rabbitllm python scripts/inference_example.py --help
+
 bash:
 	docker run --gpus all --rm -it -v $(PWD):/app -w /app python:3.12 bash

README.md

@@ -44,6 +44,21 @@ pip install rabbitllm[flash]
 If the prebuilt wheel is unavailable for your setup, install from
 [flashattn.dev](https://flashattn.dev). Without it, SDPA is used automatically.
 
+### Docker
+
+Build and run with GPU support (requires [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install.html) on the host):
+
+```bash
+docker build -t rabbitllm .
+docker run --gpus all -it rabbitllm python scripts/inference_example.py --model Qwen/Qwen2.5-0.5B-Instruct --max-new-tokens 20
+```
+
+Optional env vars: `HF_TOKEN` for gated models, `HF_HOME` for model cache directory. Example:
+
+```bash
+docker run --gpus all -e HF_TOKEN=hf_... -it rabbitllm python scripts/inference_example.py --model Qwen/Qwen2.5-7B-Instruct
+```
+
 ## Quickstart
 
 ```python
@@ -172,9 +187,17 @@ For 50k+ token contexts, pass `kv_cache_dir` to offload KV cache to SSD:
 model = AutoModel.from_pretrained("Qwen/Qwen2.5-72B-Instruct", kv_cache_dir="./kv_cache")
 ```
 
-### Benchmarking improvements
+### Benchmark
+
+Scripts to measure throughput and compare configurations:
 
-To measure GDS and DiskKVCache improvements:
+| Script | What it measures |
+|--------|------------------|
+| `scripts/benchmark_improvements.py` | GDS (GPU Direct Storage) and long-context DiskKVCache improvements |
+| `scripts/benchmark_cpu_vs_cuda.py` | CPU vs CUDA inference with layer-streaming (same model and prompt) |
+| `scripts/check_attention_and_benchmark.py --benchmark` | Throughput comparison: auto vs SDPA vs eager attention |
+
+**GDS and DiskKVCache:**
 
 ```bash
 # Local: make install pulls in kvikio (--extra gds)
@@ -187,6 +210,21 @@ pip install -e ".[gds]"
 python scripts/benchmark_improvements.py --mode gds
 ```
 
+**Quick CPU vs CUDA comparison:**
+
+```bash
+uv run python scripts/benchmark_cpu_vs_cuda.py
+uv run python scripts/benchmark_cpu_vs_cuda.py --model Qwen/Qwen2.5-1.5B-Instruct --runs 3
+```
+
+**Attention implementation (auto vs SDPA vs eager):**
+
+```bash
+uv run python scripts/check_attention_and_benchmark.py --benchmark
+```
+
+Detailed results and per-step breakdown for Qwen2.5-72B (e.g. pin_memory, async, 4-bit) are in [docs/BENCHMARK_HISTORY.md](docs/BENCHMARK_HISTORY.md).
+
 ### Gated models
 
 Pass a HuggingFace token for repos that require access approval: