fix(deserialization): replace pickle.loads with JSON across all sinks

[amabito]

Why are these changes needed?

autogen/cache/redis_cache.py:77 is pickle.loads(cache.get(...)). That is the
default production cache path -- anything that can write to your Redis instance
gets RCE on every AG2 process using it. Four sibling sinks had the same shape:
pickle.loads in the Cosmos DB cache, pickle.loads on inbound Redis pub/sub
messages, pickle.load from a caller-supplied path_to_db_dir in teachability
(RCE on the next agent startup if you can drop a file there), and
importlib.import_module on the __type__ field arriving over Redis pub/sub
JSON -- a channel writer can trigger arbitrary import-time side effects with any
importable path. All five are fixed here because splitting the rollout would just
extend the exposure window.

JSON replaces pickle everywhere. Cache stores now write a one-byte version prefix
(\x01 + JSON). Legacy pickle reads still work, but only when
AG2_ALLOW_PICKLE_CACHE_READ=1 is set -- they log a DeprecationWarning on every
hit, noisy by design. Without the flag, the reader raises. The importlib resolver
is gone; a module-level _EVENT_REGISTRY dict takes its place. Custom
BaseEvent subclasses published over Redis pub/sub need @register_event_class;
the rejection test in test_serializer_security.py demonstrates what happens
when an unregistered __type__ arrives.

Breaking changes: existing .pkl cache entries and teachability stores need
migration. A migration helper ships with this PR
(python -m autogen.agentchat.contrib.capabilities.teachability_migrate_pickle_to_json).
Stub docs at docs/security/deserialization.md and docs/cache-migration.md;
website integration is a follow-up.

Related issue number

Addresses: N/A (proactive security audit of deserialization paths)

Checks

• I've included any doc changes needed for https://docs.ag2.ai/.
  (Stub docs added to docs/security/deserialization.md and
  docs/cache-migration.md; website integration is a follow-up.)
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 25 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
autogen/beta/streams/redis/serializer.py	0.00%	25 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (49be53e) and HEAD (faf020c). Click for more details.

HEAD has 984 uploads less than BASE

Flag	BASE (49be53e)	HEAD (faf020c)
3.10	93	0
ubuntu-latest	99	0
optional-deps	108	0
commsagent-slack	6	0
browser-use	3	0
3.14	84	0
retrievechat-pgvector	6	0
windows-latest	94	1
retrievechat-mongodb	6	0
3.13	35	2
core-without-llm	15	0
3.11	46	0
graph-rag-falkor-db	6	0
3.12	32	0
macos-latest	97	1
interop-crewai	6	0
commsagent-discord	6	0
interop	6	0
retrievechat	10	0
twilio	6	0
jupyter-executor	6	0
docs	6	0
commsagent-telegram	6	0
interop-langchain	6	0
crawl4ai	4	0
websockets	6	0
interop-pydantic-ai	6	0
google-api	6	0
wikipedia-api	6	0
agent-eval	1	0
gpt-assistant-agent	3	0
teachable	3	0
llama-index-agent	3	0
mcp	5	0
retrievechat-couchbase	2	0
retrievechat-qdrant	6	0
long-context	3	0
websurfer	14	0
lmm	3	0
bedrock	7	0
swarm	13	0
together	13	0
cerebras	12	0
cohere	15	0
mistral	12	0
groq	13	0
anthropic	11	0
gemini	14	0
ollama	14	0
beta	7	2

Files with missing lines	Coverage Δ
autogen/beta/streams/redis/serializer.py	0.00% <0.00%> (-61.77%)	⬇️

... and 393 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[amabito]

CI is red on test/beta/tools/search/test_resolve.py:19 -- empty class body, import right after. Introduced in #2701, unrelated to this PR. Breaks ast check, ruff, and pytest collection. Rebasing pulls the same broken file, so nothing to do from this side until upstream patches it.

[marklysze]

This is a high-quality, important security fix — the design is well-thought-out across all three attack surfaces (Redis cache, CosmosDB cache, teachability). The version-prefix byte approach for forward/backward compatibility is clean, and the migration script + security docs are a nice touch.

One issue to address before merging: unversioned legacy cache data

Existing Redis/CosmosDB users who upgrade will have cache entries written by the old code as raw pickle.dumps(value) — no prefix byte. These payloads start with \x80\x05... (pickle protocol bytes), not \x00.

The current _deserialize logic handles three cases:

• \x01 + body → JSON ✓
• \x00 + body → pickle (with env var) ✓
• anything else → ValueError: Unknown cache payload format prefix: b'\x80' ✗

This means:

1. Option B in the migration doc doesn't work for existing data. Setting AG2_ALLOW_PICKLE_CACHE_READ=1 and calling cache.get(key) still raises ValueError because the existing data doesn't have the \x00 prefix — it's raw pickle.
2. The only viable migration path is Option A (flush), but the docs present B as an alternative.

Suggested fix — add a fallback case in _deserialize for unversioned data:

# Unversioned payload: legacy pickle written by old code (no prefix byte).
# Only honoured with AG2_ALLOW_PICKLE_CACHE_READ=1.
if _PICKLE_CACHE_READ_ENABLED:
    warnings.warn(
        "Reading an unversioned (pre-migration) pickle cache entry. ...",
        DeprecationWarning, stacklevel=4,
    )
    return pickle.loads(raw)  # noqa: S301
raise ValueError(f"Unknown cache payload format prefix: {prefix!r}")

Or alternatively, update Option B in the migration doc to say it only works for data re-written by the new code, and recommend Option A (flush) for existing deployments.

The same logic likely applies to CosmosDB. The beta Redis serializer (pub/sub) is transient so no migration concern there.

Everything else looks great — happy to approve once this edge case is addressed.

[marklysze]

Strong security fix for a real RCE vector. The version-prefix approach (\x01 = JSON, \x00 = legacy pickle) is clean — it's backward compatible for existing caches while defaulting all new writes to JSON.

The design is correct:

• New writes: always JSON regardless of env var
• Legacy pickle reads: blocked by default, opt-in via AG2_ALLOW_PICKLE_CACHE_READ=1 with a DeprecationWarning
• Migration path: teachability_migrate_pickle_to_json.py + docs/cache-migration.md give operators a clear upgrade story
• All five sinks (Redis cache, Cosmos DB cache, Redis pub/sub serializer, teachability) covered consistently

CI passes on the latest run. ✓

[amabito]

@marklysze Thanks for the review. Pushed the fallback in 9264a43:

• _deserialize in redis_cache.py and cosmos_db_cache.py now falls back to pickle.loads(raw) when prefix is unrecognised AND AG2_ALLOW_PICKLE_CACHE_READ=1, with DeprecationWarning.
• Without the env var, ValueError is still raised for unknown prefixes.
• 2 new tests cover the fallback path (with/without env var) in both test files.
• docs/cache-migration.md Option B updated to describe unversioned-pickle reads.

Ready for merge if no other concerns.

[marklysze]

Thanks @amabito!

@Lancetnik can you please check over?

[Lancetnik]

Hi @amabito — this PR now has merge conflicts with main and has been moved to status:changes-requested.

Please rebase onto the current main and resolve the conflicts. Once you push the updated branch, the PR will return to the review queue automatically.

⏳ If there's no update within 30 days, this PR will be marked stale and closed 7 days after that.