feat(plugins): introduce middleware token proxy plugin suite (reorder, dedup, lookup, skill index) - #50
Open
SNM-SNM wants to merge 45 commits into
Open
feat(plugins): introduce middleware token proxy plugin suite (reorder, dedup, lookup, skill index)#50SNM-SNM wants to merge 45 commits into
SNM-SNM wants to merge 45 commits into
Conversation
…dbox, and ZMQ prototypes
…jection and update docs
…console encoding crashes
…e sandbox Dockerfile with transformers/torch
… benchmarking - Added --mode and A/B test routing logic (baseline vs with_plugin) to run_bigcodebench_elm_full.py - Implemented global cache hit telemetry in http_server.py - Added percentage savings calculations to dedup and skill_index plugins - Added HTTP polling server to ZMQ publisher - Added HTTP polling mode to ZMQ subscriber - Created benchmark_routing_latency.py micro-benchmark comparing HTTP vs ZMQ
…ict upstream API; add SLURM eval scripts
…ter for Dedup plugin
- Create model-agnostic run_mcpatlas_eval.py for tool-use evaluation - Implement robust health-checking and boot timeouts for proxy server - Add SLURM scripts for ELM GPT-5.5 test and full runs - Add SLURM scripts for DeepSeek V4 Pro test and full runs - Ensure proper propagation of API keys and caching environment variables
added 4 commits
July 3, 2026 18:06
- Fix bash redirection bug in pip install - Adjust vLLM gpu-memory-utilization for 16GB MIG partition - Add proper vLLM shutdown to prevent Phase 2 VRAM OOM - Fix hardcoded API base URL in evaluation script - Add Qwen tool call parser support to vLLM boot command
- Added academic attribution and MSc vs PhD delineation to README - Added BibTeX citation snippet - Standardized python evaluation harnesses to accept dynamic plugins - Generated 6 universal, parameterized bash entry points - Purged legacy fractured bash scripts and slurm clutter - Archived HPC SLURM wrappers for markers - Added sentence-transformers to requirements.txt - Provided .env.example for local orchestration
…URL, plugin descriptions, shadow-mode table), remove stale INTEGRATION_GUIDE
…sing benchmark scripts - Restored the original, un-refactored evaluation scripts (run_mcpatlas_eval.py, etc.) that were deleted during the refactor but are still explicitly referenced by the HPC SLURM wrappers. This guarantees markers can seamlessly execute the cluster runs. - Added a --distractor_ratio flag to run_mcpatlas_eval.py for the ablation sweep. - Created submit_distractor_sweep_mcpatlas.slurm to execute the new ablation.
…ollisions on cluster
…igorously test attention dilution
…ormalization - Implemented real distractor sampling from global tool pool in MCP-Atlas eval to properly ablate attention dilution. - Added --seed parameter to BigCodeBench full evaluation for reproducibility. - Reverted temperature configurations to safely rely on API default (~1.0) to preserve variance analysis matching historic reported metrics. - Added 3-seed full evaluation SLURM launcher for BigCodeBench.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR introduces a suite of Middleware Token Proxy Plugins designed to sit between agent frameworks (e.g., OpenClaw) and LLM engines (e.g., SGLang) to optimize KV cache sharing, minimize context budgets, and perform cache-aware routing.
These modules implement the core optimizations detailed in the MSc cache optimization project (Direction A).
Key Features
1. Static Context Optimization Plugins (WP1)
ContextReorderPlugin: Processes OpenAI-formatted request batches to group similar contexts, maximizing RadixAttention prefix sharing on SGLang.ContextDedupPlugin: Conversation-aware history compressor. It tracks session turns and automatically replaces redundant historical messages with lightweight reference hints (e.g.,[Reference to Turn 1]) utilizing theConversationTracker.2. Dynamic Routing & Tool Filtering Plugins (WP2)
KVCacheLookupPlugin: Subscribes to SGLang worker event streams via ZeroMQ (ZMQ) to build a real-time, in-memory Shadow Radix Tree representing the workers' GPU KV cache. Routes incoming requests to the worker with the longest prefix match.SkillAwareContextPlugin: Dynamically filters and injects tool schemas into the request'stoolsarray based on the_required_skillslist, trimming unused tool definitions to save context budget.3. Core Engine Optimizations & Fixes
compute_distance_cpu.pywhennum_workers == 1to eliminatemultiprocessing.Poolinitialization and IPC serialization overhead.✓with standard+in all logging/printing calls to preventUnicodeEncodeErrorcrashes on non-UTF-8 consoles.pytest-asyncio,pyzmq, andmsgspecto dependencies to ensure successful integration and testing.Verification & Testing
evaluation/core_merge/mock_proxy.pyto demonstrate the end-to-end integration and telemetry collection of all four plugins.tests/test_kv_lookup.pyandtests/test_skill_index.py.