[Feature Request] Is there any plan or architect rethinking for help the camel to be become an Agentic AI Runtime ?

[xunavy]

Required prerequisites

• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• Consider asking first in a Discussion.

Motivation

Motivation
From my understanding,as a powerful multi-agent framework, CAMEL currently leans more towards a consumer-facing (toC) product framework .
However, the runtime-level capabilities required for enterprise (toB) deployment (e.g., concurrency control, state governance, security isolation) will be the Agent in future. Elevating it to an Agentic Runtime aligns with AI-native infrastructure(k8s) trends and better addresses toB needs for production readiness.
REF:https://aaif.io/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation-aaif-anchored-by-new-project-contributions-including-model-context-protocol-mcp-goose-and-agents-md/
Feature Request
Support integration with AI-native infrastructure (e.g., Kubernetes, Ray) for cloud-native scheduling and scalability.
Add a unified state model (session/task/intermediate states) with breakpoint recovery and execution trace persistence.
Introduce tool sandboxing and resource scheduling for isolated execution, permission control, and Token/GPU resource limits.

Solution

No response

Alternatives

No response

Additional context

No response

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• [Feature Request] deep research agent and demo #1639
• [Feature Request] add google Agent2Agent toolkit and example #2162
• [Feature Request] Refactor ThinkingToolkit to test and support more tools #1982
• [Feature Request] Integrate aci for tool connection for agent #2256
• [Feature Request] Support more running mode in workforce #1663
• [Feature Request] Workforce support .add_a2a_agent_worker to import A2A agent #2788

👤 Suggested Assignees

• Wendong-Fan
• zjrwtx
• Qihong-Pan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[lightaime]

Yes, we actually have implemented runtimes for tools last year and recently supoort runtimes for multiple toolkits #3551. Runtime orchestration like Kubernetes or Ray is not supported yet. We do have a plan to support k8s like https://github.com/kubernetes-sigs/agent-sandbox/tree/main or ray for runtime orchestration in RL traning and agent development. Could you elaborate more your scenario? That would be very helpful for our design choice and priority. c.c. @hesamsheikh @Wendong-Fan

[coderabbitai]

Coding Plan

Approach

Ticket Summary

Transform CAMEL from a consumer-facing multi-agent framework into an enterprise-ready Agentic AI Runtime by adding: (1) AI-native infrastructure integration (Kubernetes/Ray) for cloud-native scheduling and scalability, (2) unified state model with breakpoint recovery and execution trace persistence, and (3) enhanced tool sandboxing with resource scheduling and permission control.

Codebase Summary

CAMEL is a multi-agent AI framework with a hierarchical runtime system built around BaseRuntime, supporting Docker, HTTP, Daytona, and LLMGuard execution environments. The framework includes a mature storage/memory system (vector DBs, key-value stores, object storage) with ChatHistoryBlock and VectorDBBlock for agent state persistence. Multi-agent coordination exists via the Workforce module with AgentPool, task channels, and event-driven state tracking. Current limitations include: no cloud-native orchestration, no unified state model across components, limited checkpointing/recovery, and basic resource isolation delegated to Docker.

Observations

CAMEL is a multi-agent AI framework with a hierarchical runtime system built around BaseRuntime, supporting Docker, HTTP, Daytona, and LLMGuard execution environments. The framework includes a mature storage/memory system (vector DBs, key-value stores, object storage) with ChatHistoryBlock and VectorDBBlock for agent state persistence. Multi-agent coordination exists via the Workforce module with AgentPool, task channels, and event-driven state tracking. Current limitations include: no cloud-native orchestration, no unified state model across components, limited checkpointing/recovery, and basic resource isolation delegated to Docker.

Assumptions

Assumption 1: Orchestration Backend Priority

Options Considered:

• Start with Kubernetes only: Focus on long-running agent services and tool execution with K8s operators
• Start with Ray only: Focus on distributed RL training and hyperparameter tuning scenarios
• Build abstraction layer supporting both: Maximum flexibility but higher initial complexity

Chosen Option: Start with Kubernetes integration first, then add Ray support in a follow-up phase

Rationale: The maintainer's comment references agent-sandbox (a K8s project) and indicates K8s is aligned with their roadmap. The orchestrator abstraction pattern allows adding Ray later without major refactoring. This provides enterprise deployment capabilities (the primary ticket motivation) while keeping the initial scope manageable.

Assumption 2: Checkpointing Granularity

Options Considered:

• Runtime-level only: Checkpoint tool execution state and runtime configuration
• Agent-level only: Checkpoint agent memory and conversation history (already partially exists)
• Unified model: Single checkpoint capturing runtime, agent, and workflow state together

Chosen Option: Implement unified state model with hierarchical checkpoints

Rationale: A WorkflowState captures the complete execution context (runtime configuration, agent states, task dependencies) while allowing individual components to be checkpointed independently. This aligns with the ticket's request for "unified state model" and supports breakpoint recovery at any level. Leverages existing memory/storage backends to minimize new infrastructure.

Assumption 3: Resource Isolation Approach

Options Considered:

• Extend existing Docker/Daytona runtimes with resource configurations
• Create new secure runtime types (gVisor, Kata) as separate implementations
• Add policy layer above all runtimes for unified resource governance

Chosen Option: Add a unified resource policy and permission system that works across all runtime types, with concrete implementations for Kubernetes-native sandboxing

Rationale: This avoids fragmenting resource control logic across six runtime implementations. Existing Docker runtimes gain resource limits via policy configuration, while new K8s-managed runtimes get native Pod resource quotas and gVisor/Kata isolation options.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: State Management Foundation

Establish the unified state model with checkpoint/restore capabilities before adding orchestration layers. This provides immediate value (breakpoint recovery) and creates the foundation for distributed execution.

Task 1: Create Unified State Model

Define core state abstractions that can capture runtime, agent, and workflow execution state in a serializable format.

• Create camel/states/ module with RuntimeState, AgentState, and WorkflowState classes
• RuntimeState captures: runtime type, configuration, registered tools, active toolkit instances
• AgentState extends existing agent memory patterns to include: model config, system prompts, termination state, tool execution history
• WorkflowState for multi-agent scenarios: task graph, agent assignments, inter-agent message queue, event log
• All state classes implement to_dict() and from_dict() for serialization compatibility with existing storage backends

Task 2: Implement Checkpoint Manager

Create checkpoint orchestration that uses existing storage backends for persistence.

• Add camel/states/checkpoint_manager.py with CheckpointManager class
• Support hierarchical checkpoints: individual components (runtime/agent) or full workflow snapshots
• Implement versioning and checkpoint retention policies (keep last N, time-based expiration)
• Use existing JsonStorage, RedisStorage, or ObjectStorage backends for checkpoint persistence
• Add checkpoint metadata: timestamp, component types, version info, parent checkpoint references
• Integrate with existing MemoryRecord serialization patterns for consistency

Task 3: Extend Runtime Base with State Hooks

Add lifecycle hooks to BaseRuntime for state capture and restoration without breaking existing implementations.

• Add optional save_state() and restore_state() methods to BaseRuntime in camel/runtimes/base.py
• Implement in DockerRuntime, RemoteHttpRuntime, DaytonaRuntime to capture: container IDs, network configs, mounted volumes, loaded toolkit state
• State hooks called during build() (after initialization) and before reset() (for checkpoint before cleanup)
• Backward compatible: existing runtime implementations continue working without state management

Task 4: Extend Agent Base with Checkpoint Support

Enhance agent checkpoint capabilities beyond current save_memory() to capture full execution state.

• Extend BaseAgent.reset() to optionally accept AgentState for restoration
• Add checkpoint() method to ChatAgent that captures: memory state (existing), response terminators, tool call history, model backend configuration
• Integrate with CheckpointManager for automated periodic checkpointing
• Support checkpoint-based initialization: ChatAgent.from_checkpoint(checkpoint_path)

🤖 Prompt for AI agents

Implement unified state model foundation for checkpoint and recovery
functionality.

Create `camel/states/` module with three core state classes:
- `RuntimeState`: Captures runtime type, configuration, registered tools, and
toolkit instances
- `AgentState`: Extends agent memory to include model config, system prompts,
termination state, and tool execution history
- `WorkflowState`: Captures task graph, agent assignments, message queue, and
event log for multi-agent scenarios
- All classes must implement `to_dict()` and `from_dict()` for serialization

Create `CheckpointManager` in `camel/states/checkpoint_manager.py`:
- Support hierarchical checkpoints (component-level or full workflow)
- Implement versioning and retention policies
- Use existing storage backends (JsonStorage, RedisStorage, ObjectStorage)
- Add checkpoint metadata tracking

Extend `BaseRuntime` with state management hooks:
- Add optional `save_state()` and `restore_state()` methods to
`camel/runtimes/base.py`
- Implement in DockerRuntime, RemoteHttpRuntime, and DaytonaRuntime
- Capture container IDs, network configs, volumes, and toolkit state
- Ensure backward compatibility

Extend agent checkpoint capabilities:
- Update `BaseAgent.reset()` to accept optional `AgentState` parameter
- Add `checkpoint()` method to `ChatAgent`
- Integrate with CheckpointManager for periodic checkpointing
- Add `ChatAgent.from_checkpoint()` class method for restoration

Phase 2: Resource Policy and Isolation Framework

Create the policy layer for resource governance and sandboxing before implementing specific orchestrators. This ensures consistent behavior across local Docker and future cloud-native deployments.

Task 1: Define Resource Policy Configuration Model

Create Pydantic models for declarative resource specifications.

• Add camel/runtimes/policies.py with ResourcePolicy, SecurityPolicy, and IsolationPolicy models
• ResourcePolicy includes: CPU limits (cores, shares), memory limits (GB), GPU allocation (device IDs, memory), timeout constraints, disk quotas
• SecurityPolicy includes: allowed network access (egress rules), filesystem permissions (read-only mounts, allowed paths), capability restrictions (no privileged, syscall filtering)
• IsolationPolicy includes: isolation level (process/container/VM), runtime backend preference (Docker/gVisor/Kata)
• Extend existing TaskConfig pattern in camel/runtimes/configs.py to incorporate these policies

Task 2: Implement Policy Enforcement Layer

Add policy validation and enforcement that works across runtime types.

• Create camel/runtimes/policy_enforcer.py with PolicyEnforcer class
• Validates policies against runtime capabilities (e.g., GPU limits only for GPU-enabled runtimes)
• Translates policies into runtime-specific configurations: Docker resource flags, Kubernetes Pod ResourceRequirements, container security contexts
• Raises PolicyViolationError when tools/toolkits request resources exceeding policy limits
• Integrates with BaseRuntime.add() to check policies before registering tools

Task 3: Enhance Docker Runtime with Resource Controls

Extend existing Docker and UbuntuDocker runtimes to support resource policies.

• Modify DockerRuntime in camel/runtimes/docker_runtime.py to accept ResourcePolicy during initialization
• Configure container creation with: mem_limit, cpu_quota, cpu_shares, device_requests (for GPU)
• Apply SecurityPolicy via container security options: cap_drop, read_only_rootfs, network mode restrictions
• Add resource monitoring: track container stats for CPU/memory usage, expose via get_resource_usage() method

Task 4: Create Permission Control System

Add fine-grained permission checking for tool execution within runtimes.

• Create camel/runtimes/permissions.py with PermissionChecker class
• Define permission types: READ_FILE, WRITE_FILE, NETWORK_ACCESS, EXEC_COMMAND, GPU_ACCESS
• Tools declare required permissions via FunctionTool metadata (extend existing FunctionTool class)
• Runtime checks permissions against SecurityPolicy before executing tool functions
• Denial triggers PermissionDeniedError with policy violation details

🤖 Prompt for AI agents

Build resource policy and isolation framework for consistent governance across
all runtime types.

Create Pydantic policy models in `camel/runtimes/policies.py`:
- `ResourcePolicy`: CPU limits, memory limits, GPU allocation, timeouts, disk
quotas
- `SecurityPolicy`: Network access rules, filesystem permissions, capability
restrictions
- `IsolationPolicy`: Isolation level and runtime backend preferences
- Extend `TaskConfig` in `camel/runtimes/configs.py` to incorporate policies

Implement `PolicyEnforcer` in `camel/runtimes/policy_enforcer.py`:
- Validate policies against runtime capabilities
- Translate policies to runtime-specific configurations (Docker flags, K8s
ResourceRequirements)
- Raise `PolicyViolationError` for violations
- Integrate with `BaseRuntime.add()` for pre-registration validation

Enhance DockerRuntime with resource controls:
- Modify `DockerRuntime` in `camel/runtimes/docker_runtime.py` to accept
`ResourcePolicy`
- Apply resource limits: mem_limit, cpu_quota, cpu_shares, device_requests
- Apply security options: cap_drop, read_only_rootfs, network restrictions
- Add `get_resource_usage()` method for monitoring

Create permission system in `camel/runtimes/permissions.py`:
- Implement `PermissionChecker` class
- Define permission types (READ_FILE, WRITE_FILE, NETWORK_ACCESS, EXEC_COMMAND,
GPU_ACCESS)
- Extend `FunctionTool` with permission metadata
- Check permissions before tool execution and raise `PermissionDeniedError` on
denial

Phase 3: Kubernetes Orchestration Layer

Build the Kubernetes integration for cloud-native scheduling, leveraging agent-sandbox patterns and Custom Resource Definitions.

Task 1: Create Orchestrator Abstraction

Define the interface for orchestrators before implementing K8s specifics.

• Create camel/orchestrators/ module with BaseOrchestrator abstract class
• Define orchestrator contract: schedule_runtime(), scale_workload(), monitor_health(), collect_metrics()
• schedule_runtime() accepts runtime config and resource policy, returns orchestrated runtime handle
• Abstract away orchestration-specific details (K8s Pods, Ray actors) from core CAMEL components
• Orchestrator manages lifecycle events: startup, failure recovery, scaling, shutdown

Task 2: Implement Kubernetes Operator Pattern

Create Custom Resource Definitions and controller for CAMEL runtimes on K8s.

• Add camel/orchestrators/kubernetes/ directory with operator implementation
• Define CRDs: CAMELRuntime (represents a runtime instance with tools/toolkits), CAMELAgent (represents agent with runtime binding), CAMELWorkflow (multi-agent workflow)
• CAMELRuntime spec includes: runtime type (Docker/gVisor/Kata), toolkit registrations, resource policies, storage backend config
• Implement controller using kopf framework or Kubernetes Python client with watch API
• Controller reconciles: creates Pods with appropriate runtime class (gVisor/Kata), mounts storage for checkpoints, applies resource limits and security policies
• Supports agent-sandbox integration: use SandboxTemplate for isolation policies, WarmPools for fast startup

Task 3: Build Kubernetes Runtime Implementation

Create a runtime class that represents K8s-orchestrated execution environments.

• Add camel/runtimes/kubernetes_runtime.py with KubernetesRuntime class inheriting from BaseRuntime
• Initialization creates CAMELRuntime custom resource in K8s cluster via API client
• build() method waits for Pod to reach Ready state, establishes communication via LoadBalancer/NodePort service
• Tool execution proxies through HTTP/gRPC to FastAPI server running in Pod (reuse existing api.py pattern)
• State management: checkpoints stored in PersistentVolume or cloud storage configured in CRD
• Cleanup via finalizers: reset() deletes custom resource, triggering cascading Pod deletion

Task 4: Add Kubernetes-Specific Features

Implement enterprise capabilities unique to K8s deployments.

• Horizontal Pod Autoscaling: Create HPA resources based on CPU/memory metrics or custom metrics (requests per second)
• Multi-namespace support: Deploy agent workflows in isolated namespaces for tenant separation
• Service mesh integration: Support Istio/Linkerd for mTLS between agents, traffic management, observability
• ConfigMap/Secret injection: Runtime configurations and API keys loaded from K8s native resources
• Rolling updates: Update toolkit versions without downtime via Deployment rolling update strategy

🤖 Prompt for AI agents

Build Kubernetes orchestration layer with CRDs and operator pattern for
cloud-native deployments.

Create orchestrator abstraction in `camel/orchestrators/`:
- Define `BaseOrchestrator` abstract class with methods: `schedule_runtime()`,
`scale_workload()`, `monitor_health()`, `collect_metrics()`
- `schedule_runtime()` accepts runtime config and resource policy, returns
runtime handle
- Abstract orchestration-specific details from core framework

Implement Kubernetes operator in `camel/orchestrators/kubernetes/`:
- Define CRDs: `CAMELRuntime`, `CAMELAgent`, `CAMELWorkflow`
- `CAMELRuntime` spec includes runtime type, toolkit registrations, resource
policies, storage config
- Implement controller using kopf or Kubernetes Python client
- Controller creates Pods with runtime classes, mounts storage, applies policies
- Support agent-sandbox patterns (SandboxTemplate, WarmPools)

Create `KubernetesRuntime` in `camel/runtimes/kubernetes_runtime.py`:
- Inherit from `BaseRuntime`
- Create `CAMELRuntime` CRD on initialization
- `build()` waits for Pod Ready state and establishes communication
- Proxy tool execution via HTTP/gRPC to Pod FastAPI server
- Store checkpoints in PersistentVolume or cloud storage
- Implement cleanup with finalizers

Add K8s enterprise features:
- Horizontal Pod Autoscaling with HPA resources
- Multi-namespace support for tenant isolation
- Service mesh integration (Istio/Linkerd)
- ConfigMap/Secret injection for configurations
- Rolling updates for zero-downtime toolkit updates

Phase 4: Workflow Orchestration and Event Integration

Connect the unified state model with orchestration layer and extend Workforce for cloud-native execution.

Task 1: Extend Workforce for Orchestrated Runtimes

Modify existing Workforce module to support K8s-orchestrated workers.

• Update camel/societies/workforce/single_agent_worker.py to accept orchestrator parameter
• AgentPool.get_agent() can provision agents via KubernetesOrchestrator.schedule_runtime() instead of local instantiation
• Worker lifecycle tied to K8s Pod lifecycle: pool scaling triggers Pod creation/deletion
• Integrate checkpoint-based agent restoration: failed workers resume from last checkpoint via WorkflowMemoryManager

Task 2: Implement Distributed Event Streaming

Enhance event system for cross-node communication in distributed deployments.

• Extend camel/societies/workforce/events.py to support distributed pub/sub backends (Redis Streams, Kafka)
• WorkforceEvent serialization for transmission across nodes
• Event router: local events stay in-process, distributed deployments publish to message broker
• Agents subscribe to relevant event channels for task coordination and state synchronization

Task 3: Create Workflow State Persistence

Add automatic checkpoint orchestration for entire multi-agent workflows.

• Integrate CheckpointManager with Workforce class: periodic workflow snapshots during execution
• Checkpoint triggers: configurable intervals, task completion milestones, before critical operations
• Failure recovery: Workforce.resume_from_checkpoint() restores task graph, agent states, and pending work
• Checkpoint pruning: retain only latest N checkpoints per workflow to manage storage costs

Task 4: Add Execution Trace Visualization

Provide observability into distributed workflow execution.

• Extend WorkflowMemoryManager to persist execution traces in structured format (OpenTelemetry spans)
• Trace includes: task assignments, agent decisions, tool executions, state transitions, failures/retries
• Export to observability backends: Jaeger, Prometheus, Grafana
• Query interface: retrieve traces by workflow ID, time range, agent ID for debugging and analysis

🤖 Prompt for AI agents

Integrate state management with orchestration layer and extend Workforce for
distributed execution.

Extend Workforce for orchestrated runtimes:
- Update `camel/societies/workforce/single_agent_worker.py` to accept
orchestrator parameter
- Modify `AgentPool.get_agent()` to provision agents via orchestrator
- Tie worker lifecycle to K8s Pod lifecycle
- Integrate checkpoint-based restoration for failed workers via
`WorkflowMemoryManager`

Implement distributed event streaming:
- Extend `camel/societies/workforce/events.py` with distributed pub/sub support
(Redis Streams, Kafka)
- Add `WorkforceEvent` serialization for cross-node transmission
- Implement event router: local events in-process, distributed events to message
broker
- Support agent subscription to event channels

Create workflow state persistence:
- Integrate `CheckpointManager` with `Workforce` class for periodic snapshots
- Configure checkpoint triggers (intervals, milestones, critical operations)
- Implement `Workforce.resume_from_checkpoint()` for failure recovery
- Add checkpoint pruning to retain latest N checkpoints

Add execution trace visualization:
- Extend `WorkflowMemoryManager` to persist traces in OpenTelemetry format
- Capture task assignments, agent decisions, tool executions, state transitions,
failures/retries
- Export to Jaeger, Prometheus, Grafana
- Implement query interface for traces by workflow ID, time range, agent ID

Phase 5: Ray Integration and Advanced Scheduling

Add Ray orchestrator for distributed training and hyperparameter tuning scenarios.

Task 1: Implement Ray Orchestrator

Create Ray-based orchestrator following the same abstraction pattern.

• Add camel/orchestrators/ray/ directory with RayOrchestrator class implementing BaseOrchestrator
• schedule_runtime() creates Ray actor for runtime, places actor based on resource requirements (GPUs, memory)
• Leverage Ray's built-in fault tolerance: actor restart on failure, checkpoint-based recovery
• Support Ray autoscaling: cluster scales up/down based on workload demand

Task 2: Integrate Ray Train Checkpointing

Connect CAMEL state management with Ray Train's checkpoint system for RL/training workflows.

• Map RuntimeState/AgentState to Ray Train Checkpoint objects
• Distributed checkpointing: each worker saves checkpoint shard to cloud storage in parallel
• Checkpoint manager coordinates: tracks checkpoint locations, orchestrates restoration across workers
• Support for fault tolerance: interrupted training jobs resume from latest distributed checkpoint

Task 3: Build Ray-Specific Runtime Utilities

Add features that leverage Ray's distributed computing capabilities.

• Parallel tool execution: distribute tool calls across Ray cluster for batch processing
• Resource-aware scheduling: runtime placement considers GPU availability, data locality, network topology
• Dynamic scaling: add/remove runtime instances based on queue depth and latency requirements
• Integration with Ray Datasets for large-scale data preprocessing in agent workflows

Task 4: Create Hybrid Orchestration Support

Enable workflows that span both Kubernetes and Ray.

• Unified orchestration API: schedule_runtime() accepts orchestrator preference, falls back intelligently
• Use case: K8s for stateful agent services, Ray for distributed training/inference tasks
• State synchronization: checkpoints accessible from both K8s Pods and Ray actors via shared storage
• Workload routing: task characteristics (CPU-bound, GPU-bound, I/O-bound) determine orchestrator selection

🤖 Prompt for AI agents

Add Ray orchestrator for distributed training and implement hybrid orchestration
support.

Implement Ray orchestrator:
- Create `RayOrchestrator` in `camel/orchestrators/ray/` implementing
`BaseOrchestrator`
- `schedule_runtime()` creates Ray actor with resource-based placement
- Leverage Ray's fault tolerance (actor restart, checkpoint-based recovery)
- Support Ray autoscaling based on workload demand

Integrate Ray Train checkpointing:
- Map `RuntimeState`/`AgentState` to Ray Train `Checkpoint` objects
- Implement distributed checkpointing with parallel shard saves to cloud storage
- Coordinate checkpoint tracking and restoration across workers
- Support fault tolerance for interrupted training jobs

Build Ray-specific utilities:
- Implement parallel tool execution across Ray cluster
- Add resource-aware scheduling (GPU availability, data locality, network
topology)
- Support dynamic scaling based on queue depth and latency
- Integrate with Ray Datasets for data preprocessing

Create hybrid orchestration support:
- Implement unified API with orchestrator preference and intelligent fallback
- Enable K8s for stateful services, Ray for training/inference
- Synchronize state via shared storage accessible to both K8s and Ray
- Route workloads based on task characteristics (CPU/GPU/I/O bound)

Phase 6: Developer Experience and Migration Path

Provide tooling and documentation to help developers adopt orchestration features without disrupting existing workflows.

Task 1: Create Configuration Builders

Add fluent API for configuring orchestrated runtimes.

• Extend existing runtime initialization pattern with builder methods: .with_resources(), .with_security_policy(), .with_orchestrator()
• Example: DockerRuntime("image").with_resources(cpu=2, memory="4Gi").with_orchestrator(k8s_orchestrator)
• Validation: builder checks policy compatibility before creating runtime
• Sensible defaults: developers can opt into orchestration without specifying every parameter

Task 2: Implement Backward Compatibility Layer

Ensure existing examples and user code continue working without modification.

• Orchestration is opt-in: existing DockerRuntime, RemoteHttpRuntime work exactly as before
• State management opt-in: checkpoint features enabled via explicit config or environment variables
• Graceful degradation: if K8s/Ray unavailable, fall back to local execution with warning
• Feature detection: runtime introspects environment, enables orchestration only when infrastructure available

Task 3: Add Examples and Documentation

Provide comprehensive examples for common deployment patterns.

• Create examples/orchestration/ directory with:
  • Simple K8s deployment: single agent with resource limits
  • Multi-agent workflow: Workforce on K8s with checkpoint recovery
  • Ray training: RL agent training with distributed checkpointing
  • Hybrid: K8s agent calling Ray-based batch inference
• Documentation in docs covering: resource policy configuration, checkpoint management, orchestrator selection, migration from local to cloud-native

Task 4: Create Migration Tooling

Build utilities to help users transition existing workflows to orchestrated deployments.

• Configuration analyzer: scans existing runtime usage, suggests resource policies and orchestration strategies
• Checkpoint converter: migrates existing agent memories to unified checkpoint format
• Deployment generator: creates K8s YAML manifests from CAMEL runtime configurations
• Validation suite: tests runtime behavior under resource constraints before production deployment

🤖 Prompt for AI agents

Provide developer experience improvements and migration tooling for seamless
adoption.

Create configuration builders:
- Extend runtime initialization with fluent builder methods:
`.with_resources()`, `.with_security_policy()`, `.with_orchestrator()`
- Add validation to check policy compatibility before runtime creation
- Provide sensible defaults for easy opt-in

Implement backward compatibility:
- Make orchestration opt-in (existing runtimes work unchanged)
- Enable state management via explicit config or environment variables
- Implement graceful degradation with fallback to local execution
- Add feature detection to enable orchestration only when infrastructure
available

Add examples and documentation:
- Create `examples/orchestration/` with examples: simple K8s deployment,
multi-agent workflow, Ray training, hybrid orchestration
- Document resource policy configuration, checkpoint management, orchestrator
selection, migration strategies

Create migration tooling:
- Configuration analyzer: scan usage and suggest policies/strategies
- Checkpoint converter: migrate existing memories to unified format
- Deployment generator: create K8s YAML from CAMEL configs
- Validation suite: test runtime behavior under resource constraints

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement unified state model foundation for checkpoint and recovery
functionality.

Create `camel/states/` module with three core state classes:
- `RuntimeState`: Captures runtime type, configuration, registered tools, and
toolkit instances
- `AgentState`: Extends agent memory to include model config, system prompts,
termination state, and tool execution history
- `WorkflowState`: Captures task graph, agent assignments, message queue, and
event log for multi-agent scenarios
- All classes must implement `to_dict()` and `from_dict()` for serialization

Create `CheckpointManager` in `camel/states/checkpoint_manager.py`:
- Support hierarchical checkpoints (component-level or full workflow)
- Implement versioning and retention policies
- Use existing storage backends (JsonStorage, RedisStorage, ObjectStorage)
- Add checkpoint metadata tracking

Extend `BaseRuntime` with state management hooks:
- Add optional `save_state()` and `restore_state()` methods to
`camel/runtimes/base.py`
- Implement in DockerRuntime, RemoteHttpRuntime, and DaytonaRuntime
- Capture container IDs, network configs, volumes, and toolkit state
- Ensure backward compatibility

Extend agent checkpoint capabilities:
- Update `BaseAgent.reset()` to accept optional `AgentState` parameter
- Add `checkpoint()` method to `ChatAgent`
- Integrate with CheckpointManager for periodic checkpointing
- Add `ChatAgent.from_checkpoint()` class method for restoration
===============================================================================

Task: 2

Build resource policy and isolation framework for consistent governance across
all runtime types.

Create Pydantic policy models in `camel/runtimes/policies.py`:
- `ResourcePolicy`: CPU limits, memory limits, GPU allocation, timeouts, disk
quotas
- `SecurityPolicy`: Network access rules, filesystem permissions, capability
restrictions
- `IsolationPolicy`: Isolation level and runtime backend preferences
- Extend `TaskConfig` in `camel/runtimes/configs.py` to incorporate policies

Implement `PolicyEnforcer` in `camel/runtimes/policy_enforcer.py`:
- Validate policies against runtime capabilities
- Translate policies to runtime-specific configurations (Docker flags, K8s
ResourceRequirements)
- Raise `PolicyViolationError` for violations
- Integrate with `BaseRuntime.add()` for pre-registration validation

Enhance DockerRuntime with resource controls:
- Modify `DockerRuntime` in `camel/runtimes/docker_runtime.py` to accept
`ResourcePolicy`
- Apply resource limits: mem_limit, cpu_quota, cpu_shares, device_requests
- Apply security options: cap_drop, read_only_rootfs, network restrictions
- Add `get_resource_usage()` method for monitoring

Create permission system in `camel/runtimes/permissions.py`:
- Implement `PermissionChecker` class
- Define permission types (READ_FILE, WRITE_FILE, NETWORK_ACCESS, EXEC_COMMAND,
GPU_ACCESS)
- Extend `FunctionTool` with permission metadata
- Check permissions before tool execution and raise `PermissionDeniedError` on
denial
===============================================================================

Task: 3

Build Kubernetes orchestration layer with CRDs and operator pattern for
cloud-native deployments.

Create orchestrator abstraction in `camel/orchestrators/`:
- Define `BaseOrchestrator` abstract class with methods: `schedule_runtime()`,
`scale_workload()`, `monitor_health()`, `collect_metrics()`
- `schedule_runtime()` accepts runtime config and resource policy, returns
runtime handle
- Abstract orchestration-specific details from core framework

Implement Kubernetes operator in `camel/orchestrators/kubernetes/`:
- Define CRDs: `CAMELRuntime`, `CAMELAgent`, `CAMELWorkflow`
- `CAMELRuntime` spec includes runtime type, toolkit registrations, resource
policies, storage config
- Implement controller using kopf or Kubernetes Python client
- Controller creates Pods with runtime classes, mounts storage, applies policies
- Support agent-sandbox patterns (SandboxTemplate, WarmPools)

Create `KubernetesRuntime` in `camel/runtimes/kubernetes_runtime.py`:
- Inherit from `BaseRuntime`
- Create `CAMELRuntime` CRD on initialization
- `build()` waits for Pod Ready state and establishes communication
- Proxy tool execution via HTTP/gRPC to Pod FastAPI server
- Store checkpoints in PersistentVolume or cloud storage
- Implement cleanup with finalizers

Add K8s enterprise features:
- Horizontal Pod Autoscaling with HPA resources
- Multi-namespace support for tenant isolation
- Service mesh integration (Istio/Linkerd)
- ConfigMap/Secret injection for configurations
- Rolling updates for zero-downtime toolkit updates
===============================================================================

Task: 4

Integrate state management with orchestration layer and extend Workforce for
distributed execution.

Extend Workforce for orchestrated runtimes:
- Update `camel/societies/workforce/single_agent_worker.py` to accept
orchestrator parameter
- Modify `AgentPool.get_agent()` to provision agents via orchestrator
- Tie worker lifecycle to K8s Pod lifecycle
- Integrate checkpoint-based restoration for failed workers via
`WorkflowMemoryManager`

Implement distributed event streaming:
- Extend `camel/societies/workforce/events.py` with distributed pub/sub support
(Redis Streams, Kafka)
- Add `WorkforceEvent` serialization for cross-node transmission
- Implement event router: local events in-process, distributed events to message
broker
- Support agent subscription to event channels

Create workflow state persistence:
- Integrate `CheckpointManager` with `Workforce` class for periodic snapshots
- Configure checkpoint triggers (intervals, milestones, critical operations)
- Implement `Workforce.resume_from_checkpoint()` for failure recovery
- Add checkpoint pruning to retain latest N checkpoints

Add execution trace visualization:
- Extend `WorkflowMemoryManager` to persist traces in OpenTelemetry format
- Capture task assignments, agent decisions, tool executions, state transitions,
failures/retries
- Export to Jaeger, Prometheus, Grafana
- Implement query interface for traces by workflow ID, time range, agent ID
===============================================================================

Task: 5

Add Ray orchestrator for distributed training and implement hybrid orchestration
support.

Implement Ray orchestrator:
- Create `RayOrchestrator` in `camel/orchestrators/ray/` implementing
`BaseOrchestrator`
- `schedule_runtime()` creates Ray actor with resource-based placement
- Leverage Ray's fault tolerance (actor restart, checkpoint-based recovery)
- Support Ray autoscaling based on workload demand

Integrate Ray Train checkpointing:
- Map `RuntimeState`/`AgentState` to Ray Train `Checkpoint` objects
- Implement distributed checkpointing with parallel shard saves to cloud storage
- Coordinate checkpoint tracking and restoration across workers
- Support fault tolerance for interrupted training jobs

Build Ray-specific utilities:
- Implement parallel tool execution across Ray cluster
- Add resource-aware scheduling (GPU availability, data locality, network
topology)
- Support dynamic scaling based on queue depth and latency
- Integrate with Ray Datasets for data preprocessing

Create hybrid orchestration support:
- Implement unified API with orchestrator preference and intelligent fallback
- Enable K8s for stateful services, Ray for training/inference
- Synchronize state via shared storage accessible to both K8s and Ray
- Route workloads based on task characteristics (CPU/GPU/I/O bound)
===============================================================================

Task: 6

Provide developer experience improvements and migration tooling for seamless
adoption.

Create configuration builders:
- Extend runtime initialization with fluent builder methods:
`.with_resources()`, `.with_security_policy()`, `.with_orchestrator()`
- Add validation to check policy compatibility before runtime creation
- Provide sensible defaults for easy opt-in

Implement backward compatibility:
- Make orchestration opt-in (existing runtimes work unchanged)
- Enable state management via explicit config or environment variables
- Implement graceful degradation with fallback to local execution
- Add feature detection to enable orchestration only when infrastructure
available

Add examples and documentation:
- Create `examples/orchestration/` with examples: simple K8s deployment,
multi-agent workflow, Ray training, hybrid orchestration
- Document resource policy configuration, checkpoint management, orchestrator
selection, migration strategies

Create migration tooling:
- Configuration analyzer: scan usage and suggest policies/strategies
- Checkpoint converter: migrate existing memories to unified format
- Deployment generator: create K8s YAML from CAMEL configs
- Validation suite: test runtime behavior under resource constraints

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord or schedule a call!

[xunavy]

another reference about the agentfield (kubernetes for AI agents Build and run AI like microservices-scalable,observable,and identify-awere. https://www.agentfield.ai/#core-features
https://www.agentfield.ai/docs/why-agentfield/core-architecture
Maybe that is a approach for camel.ai integrated with agentfield .