Merge pull request #341 from HKUDS/dev

release: v1.1.2 — CI fix & release notes cleanup

Author: pancacake

assets/releases/ver1-1-2.md

@@ -4,56 +4,45 @@
 
 ## Highlights
 
-### TutorBot — Schema-Driven Channels Tab, Token Reveal & API-Side Channel Reload (#338)
-Replaced the hard-coded Telegram-only Channels tab in `web/app/(workspace)/agents/page.tsx` with a generic, schema-driven editor that works for **every** built-in or plugin channel without any per-channel front-end code. A new `GET /api/v1/tutorbot/channels/schema` endpoint introspects every discovered `XxxChannel` / `XxxConfig` pair (Telegram, Slack, Discord, Matrix, WhatsApp, Email, Feishu, WeCom, QQ, DingTalk, etc.) and ships back a flattened JSON-Schema payload (`$ref`s inlined), the default config, the channel's `display_name`, and a list of dot-paths for any string field whose name hints at a secret (`token`, `secret`, `password`, `api_key`, `encrypt_key`). The new `SchemaField` React component recursively renders the schema — booleans become checkboxes, enums become selects, string-arrays become one-per-line textareas, nested `Optional[Model]` becomes inline fieldsets, and any flagged secret renders as a `password` input with an inline eye-toggle (`Eye` / `EyeOff`) for explicit reveal. The Channels tab now also surfaces a `last_reload_error` banner so users immediately see when on-disk config and live listeners have diverged.
+### Schema-Driven Channels Tab with Token Reveal (#338)
+The Channels tab in the Agents page is no longer hard-coded for Telegram. It now auto-discovers **every** channel (Telegram, Slack, Discord, Matrix, Email, Feishu, …) and renders a form directly from each channel's Pydantic config schema — no per-channel front-end code required. Secret fields (tokens, passwords, API keys) render as masked inputs with an eye-toggle for explicit reveal. A `last_reload_error` banner warns when live listeners failed to restart after a config change.
 
-### Channel Secret Masking End-to-End
-Added `mask_channel_secrets` and `_is_secret_field` helpers to `TutorBotManager` and rewired `TutorBotInstance.to_dict` with three explicit modes: list-of-channel-names (default for the `GET /tutorbot` index), nested dict with `***` masks (default for `GET /tutorbot/{bot_id}`), and full plaintext only when the caller passes `?include_secrets=true` (used exclusively by the admin edit form). `POST /tutorbot` and `PATCH /tutorbot/{bot_id}` responses are likewise masked. A new `tests/services/tutorbot/test_channel_secrets.py` (223 lines) and `tests/api/test_tutorbot_channel_schema.py` (185 lines) pin the masking rules, snake-case property names, secret-field detection (deliberately excluding non-string fields like `user_token_read_only: bool`), and the schema-endpoint shape across every built-in channel.
+### Channel Secret Masking
+API responses no longer expose raw channel secrets. Tokens and passwords are replaced with `***` by default; the admin edit form uses `?include_secrets=true` to fetch plaintext when needed. Create and update responses are likewise masked.
 
-### TutorBot — Concurrent Reload Lock & 422 on Bad Channel Config
-`PATCH /tutorbot/{bot_id}` now validates `payload.channels` against `ChannelsConfig` at the API boundary and returns a structured 422 (`{message, errors}`) instead of letting the bad config slip through to disk and blow up later inside `reload_channels`. `reload_channels` itself was hardened: a per-instance `asyncio.Lock` serialises concurrent invocations to prevent duplicate listeners, channel-manager construction was extracted into a shared `_build_channel_manager` so `start_bot` and `reload_channels` produce identical behaviour, and any teardown/rebuild failure is captured into `instance.last_reload_error` (cleared on the next success) and surfaced via the API.
+### Channel Config Validation & Reload Hardening
+`PATCH /tutorbot/{bot_id}` now validates channel payloads upfront and returns a 422 with structured errors instead of silently persisting bad config. `reload_channels` is serialised with a per-instance lock to prevent duplicate listeners, and any failure is recorded in `last_reload_error` so the UI can surface it.
 
-### RAG Service — One Backend, One Pipeline
-Removed ~2 600 lines of stale RAG scaffolding (`deeptutor/services/rag/components/{base,routing,chunkers,embedders,indexers,parsers,retrievers}/*`, `services/rag/pipeline.py`, `services/rag/types.py`, `services/rag/utils/__init__.py`, `services/rag/parsers/__init__.py`) that had been carrying placeholders for never-shipped pipelines. The factory (`services/rag/factory.py`) now exposes a single `LlamaIndexPipeline` keyed by `kb_base_dir`, and `RAGService` was simplified to a thin facade over it. `normalize_provider_name`, `LEGACY_PROVIDER_ALIASES`, `has_pipeline`, and the `_validate_registered_provider` API handler are all gone — `rag_provider` is now a stub field that silently coerces any legacy value (`lightrag`, `raganything`, …) into `llamaindex` while flagging the KB for re-indexing.
+### RAG Simplified to a Single Pipeline
+Removed ~2,600 lines of unused RAG scaffolding (chunkers, embedders, indexers, parsers, retrievers, pipeline orchestrator, type definitions) that existed as placeholders for never-shipped backends. The RAG service is now a thin wrapper over the single LlamaIndex pipeline. Legacy `rag_provider` values (e.g. `lightrag`) are silently coerced to `llamaindex` and the KB is flagged for re-indexing.
 
 ### Centralized File Type Routing
-Extracted file-type classification into a single `deeptutor/services/rag/file_routing.py` module (`FileTypeRouter`, `DocumentType`, `FileClassification`) with a flat per-extension API: `get_document_type`, `classify_files`, `read_text_file`, `needs_parser`, `is_text_readable`, `get_supported_extensions`, `get_glob_patterns`. The provider-keyed variants (`get_extensions_for_provider`, `get_glob_patterns_for_provider`) have been removed since there is only one provider; every call-site (`knowledge/initializer.py`, `knowledge/manager.py`, `api/routers/knowledge.py`, `services/rag/pipelines/llamaindex.py`, `deeptutor_cli/kb.py`) now uses the flat helpers. Unknown extensions still fall through to a content-sniff (binary check + UTF-8 probe) before being marked `UNKNOWN`.
+Consolidated file-type classification into a single `FileTypeRouter` module with a flat API (`get_document_type`, `classify_files`, `get_supported_extensions`, etc.). The old per-provider extension helpers are gone — there's only one provider. Unknown extensions still fall through to content sniffing before being rejected.
 
-### RAG/KB Consistency — No Phantom Knowledge Bases
-Closed every code path that could quietly invoke RAG against a non-existent or placeholder knowledge base:
+### No More Phantom Knowledge Bases
+Closed every code path that could silently call RAG against a non-existent KB:
 
-- **`deep_solve`** strips `rag` from the effective tool set when no KB is attached and emits a `trace_kind=warning` progress event so the UI can explain why retrieval was disabled.
-- **`deep_research`** drops `kb` from `request_config.sources` when no KB is attached, emits a warning, and surfaces a hard error if downgrading leaves the sources list empty (so the run aborts cleanly instead of producing an empty report).
-- **`SolveToolRuntime.execute("rag", kb_name=None)`** short-circuits with a structured `ToolResult(success=False, metadata={"skipped": True, "reason": "no_kb_selected"})` and never forwards to the registry, keeping the ReAct loop alive with a clear observation.
-- **`ResearchPipeline._call_tool`** for `rag` / `rag_hybrid` / `rag_naive` returns a structured `status: skipped` JSON (with a matching `tool_result` trace event) when `kb_name` is missing, instead of falling back to the old `DE-all` placeholder. Unknown tool types now return a structured `status: failed, reason: unknown_tool` instead of silently retrying RAG.
-- **`DecomposeAgent`** drops the `ai_textbook` default and disables RAG defensively when no `kb_name` is provided, even when the runtime config still says `enable_rag: True`.
+- **deep_solve** — strips the `rag` tool when no KB is attached and warns the user.
+- **deep_research** — drops `kb` from sources, warns, and aborts if no sources remain.
+- **SolveToolRuntime** — returns a graceful "no KB selected" observation instead of crashing, keeping the ReAct loop alive.
+- **ResearchPipeline** — returns a structured "skipped" event instead of falling back to the old `DE-all` placeholder.
+- **DecomposeAgent** — no longer defaults to `ai_textbook`; disables RAG when no KB is provided.
 
-A new `tests/capabilities/test_rag_consistency.py` (233 lines) and `tests/agents/research/test_research_pipeline_rag.py` (146 lines) lock in the contracts end-to-end.
-
-### Agentic Chat Pipeline — Externalized Prompts
-Extracted every hard-coded zh/en string out of `deeptutor/agents/chat/agentic_pipeline.py` (337 → 108 lines, –68%) into two new YAML files: `deeptutor/agents/chat/prompts/{zh,en}/agentic_chat.yaml` (178 lines each). Stage labels, `kb_hint` injection, the thinking / acting / observing / responding system + user templates, and all UI-visible notices (no-tools, ReAct fallback, etc.) are now editable copy without a code change. Loaded via the existing `get_prompt_manager()` plumbing with a graceful fallback to `_text(zh=, en=)` if the YAML is missing.
+### Externalized Chat Prompts
+Moved all hard-coded zh/en strings out of `AgenticChatPipeline` into editable YAML files (`agentic_chat.yaml` for each language). Stage labels, system prompts, user templates, and UI notices are now configurable without code changes. Falls back gracefully if the YAML is missing.
 
 ### Thai README (#337)
-Added a Thai-language README (`README_TH.md`) and wired it into the language switcher in the main README header.
+Added `README_TH.md` with Thai-language documentation.
 
 ### Bug Fixes
-- **Research pipeline crashed when no KB was selected** — `rag_cfg.get("kb_name", "DE-all")` was falling back to a hard-coded knowledge base name that no longer exists in most installs, surfacing as `KnowledgeBaseNotFoundError` partway through a research run. Now short-circuits with a structured skip event.
-- **Decompose agent attempted RAG against a non-existent default KB** — `kb_name="ai_textbook"` was the constructor default; replaced with `None` plus a defensive `enable_rag = False` guard.
-- **Bad channel config silently persisted on disk and blew up at next bot start** — `PATCH /tutorbot/{bot_id}` now rejects with a 422 listing the validation errors before anything reaches `_save_bot_config`.
-- **Concurrent `reload_channels` could spin up duplicate listeners** — serialised via `instance.reload_lock`; failure leaves the bot running channel-less with `last_reload_error` set instead of in an inconsistent half-rebuilt state.
-- **Channel tokens leaked through `GET /api/v1/tutorbot/{bot_id}` and the create/patch responses** — secrets are now masked unless the caller explicitly opts in via `?include_secrets=true`.
-
-### Test Suite Expansion
-Added 6 new test modules covering the v1.1.2 surface area:
-
-- `tests/services/rag/test_file_routing.py` (108 lines) — extension classification (case-insensitive), unknown-extension content sniffing (text vs. binary), `read_text_file` encoding fallback chain, `classify_files` bucketing, and the helper getters.
-- `tests/services/config/test_knowledge_base_config.py` (147 lines) — pins the stub-provider contract: any legacy or unknown `rag_provider` value coerces to `llamaindex` and the on-disk migration flips legacy KBs to `needs_reindex: true`.
-- `tests/api/test_tutorbot_channel_schema.py` (185 lines) — `resolve_config_model`, `inline_refs` (with nested `$defs`), `collect_secret_fields` (string-only secret detection), and the `GET /api/v1/tutorbot/channels/schema` integration shape across telegram / slack / discord.
-- `tests/services/tutorbot/test_channel_secrets.py` (223 lines) — `mask_channel_secrets` deep-walk semantics for nested objects, lists, tuples, multiple email/slack secret keys, and the three `to_dict` modes (default = list, mask, include).
-- `tests/capabilities/test_rag_consistency.py` (233 lines) — `deep_solve` and `deep_research` strip `rag` / `kb` when no KB is attached, emit the right warning events, and (for research) surface a hard error if sources go empty.
-- `tests/agents/research/test_research_pipeline_rag.py` (146 lines) — pins the removal of the `DE-all` fallback in `ResearchPipeline._call_tool` and `DecomposeAgent`'s defensive RAG disablement.
-
-Extended `tests/agents/solve/test_tool_runtime.py` with two new cases for the `rag`-without-`kb_name` short-circuit (both `None` and empty string), `tests/api/test_knowledge_router.py` to flip the legacy "reject unsupported provider" tests into "coerce to llamaindex" tests, `tests/api/test_tutorbot_router.py` with the secret-handling cases for the stopped-bot GET route, and `tests/services/rag/test_rag_pipelines.py` + `tests/tools/test_rag_tool.py` to track the components → `file_routing` move.
+- **Research pipeline crashed without a KB** — the `DE-all` fallback KB no longer exists in most installs; now short-circuits with a structured skip event.
+- **Decompose agent tried RAG against `ai_textbook`** — replaced the hard-coded default with `None` and a defensive guard.
+- **Bad channel config persisted silently** — now rejected at the API boundary with a 422 before reaching disk.
+- **Concurrent `reload_channels` created duplicate listeners** — serialised via an asyncio lock; failure leaves the bot channel-less with a clear error instead of half-rebuilt.
+- **Channel tokens leaked in API responses** — now masked by default across all endpoints.
+
+### Test Suite
+Added 6 new test modules (1,042 lines total): file-type routing, KB config migration, channel schema introspection, channel secret masking, RAG/KB consistency at the capability layer, and research pipeline RAG safety. Extended existing tests for the tool runtime, knowledge router, TutorBot router, and RAG pipeline modules.
 
 ## Community Contributions
 

deeptutor/tutorbot/channels/base.py

@@ -6,12 +6,15 @@
 from pathlib import Path
 from typing import Any
 
-from loguru import logger
-
 from deeptutor.tutorbot.bus.events import InboundMessage, OutboundMessage
 from deeptutor.tutorbot.bus.queue import MessageBus
 
 
+def _logger():
+    from loguru import logger as _log
+    return _log
+
+
 class BaseChannel(ABC):
     """
     Abstract base class for chat channel implementations.
@@ -46,7 +49,7 @@ async def transcribe_audio(self, file_path: str | Path) -> str:
             provider = GroqTranscriptionProvider(api_key=self.transcription_api_key)
             return await provider.transcribe(file_path)
         except Exception as e:
-            logger.warning("{}: audio transcription failed: {}", self.name, e)
+            _logger().warning("{}: audio transcription failed: {}", self.name, e)
             return ""
 
     @abstractmethod
@@ -80,7 +83,7 @@ def is_allowed(self, sender_id: str) -> bool:
         """Check if *sender_id* is permitted.  Empty list → deny all; ``"*"`` → allow all."""
         allow_list = getattr(self.config, "allow_from", [])
         if not allow_list:
-            logger.warning("{}: allow_from is empty — all access denied", self.name)
+            _logger().warning("{}: allow_from is empty — all access denied", self.name)
             return False
         if "*" in allow_list:
             return True
@@ -109,7 +112,7 @@ async def _handle_message(
             session_key: Optional session key override (e.g. thread-scoped sessions).
         """
         if not self.is_allowed(sender_id):
-            logger.warning(
+            _logger().warning(
                 "Access denied for sender {} on channel {}. "
                 "Add them to allowFrom list in config to grant access.",
                 sender_id, self.name,