feat(platform): add AI copilot chat assistant for agent discovery and creation #11698

Swiftyos · 2026-01-05T09:43:22Z

Not to be merged!

Splitting this up into many PR's

Summary

This PR introduces AutoGPT Copilot ("Otto"), an AI assistant that helps users discover, create, and deploy custom business automation agents through a conversational chat interface.

What it does

The copilot provides an intelligent chat experience that:

Understands user needs through natural conversation
Searches for pre-built agents in the marketplace or user's library
Creates custom agents from natural language descriptions
Executes agents with automatic credential and input handling
Learns user business context for personalized recommendations

Changes 🏗️

Backend (backend/api/features/chat/):

service.py - Core streaming chat service with OpenAI-compatible tool calling
routes.py - REST API endpoints for sessions and streaming
config.py - Configuration for LLM providers (OpenRouter/OpenAI)
model.py - Chat session and message data models
db.py - Prisma database operations for persistence
response_model.py - Streaming response types (SSE)
tools/ - 10 executable tools:
- find_agent.py - Search marketplace agents
- find_library_agent.py - Search user's saved agents
- find_block.py - Search available blocks
- create_agent.py - Generate agents from descriptions
- edit_agent.py - Modify existing agents
- run_agent.py - Execute agents with auto credential detection
- run_block.py - Execute single blocks
- agent_output.py - Retrieve execution results
- search_docs.py - Search platform documentation
- add_understanding.py - Save user business context
prompts/ - System prompts for chat persona and onboarding

Frontend (frontend/src/components/contextual/Chat/):

Chat.tsx - Main chat component with header and sessions drawer
ChatContainer.tsx - Message rendering with streaming support
useChatSession.ts - Session lifecycle management
useChatStream.ts - SSE streaming and message parsing
Message components for different response types (text, tool calls, carousels, etc.)

Database:

Added ChatSession and ChatMessage Prisma models

Key Features

Intelligent Tool Orchestration - OpenAI function calling with streaming, automatic retries on errors
Session Management - Anonymous sessions, login claiming, Redis caching with 12-hour TTL
Agent Generation - Natural language to agent workflow with validation and auto-fixing
User Context Learning - Builds business understanding for personalized recommendations
Real-Time Streaming - Server-Sent Events for responsive chat experience

Checklist 📋

For code changes:

For configuration changes:

.env.default is updated or already compatible with my changes
docker-compose.yml is updated or already compatible with my changes
I have included a list of my configuration changes in the PR description

Configuration Requirements

Environment Variables:

CHAT_API_KEY or OPEN_ROUTER_API_KEY or OPENAI_API_KEY - LLM API key
CHAT_BASE_URL - Optional base URL (defaults to OpenRouter)
CHAT_MODEL - Model selection (default: anthropic/claude-sonnet-4)
CHAT_TITLE_MODEL - Fast model for title generation (default: openai/gpt-4o-mini)
CHAT_SESSION_TTL - Redis session lifetime (default: 43200 seconds = 12 hours)

Feature Flag:

CHAT flag in LaunchDarkly controls visibility of the chat feature

Infrastructure:

Requires Redis for session caching
Uses existing Prisma database for persistence

…n/copilot

…utoGPT into hackathon/copilot

…o be able to run library agents directly

…n/copilot

…utoGPT into hackathon/copilot

…n/copilot

…utoGPT into hackathon/copilot

…n/copilot

AutoGPT-Agent · 2026-01-05T10:08:07Z

Thanks for your PR submission! This appears to be adding a significant copilot/assistant feature to the platform, but there are several critical issues that need to be addressed before this can be considered for merging:

Required Changes

Missing PR Description: Please add a clear description explaining what this copilot feature does and why it's being added. The 'Changes' section is completely empty.
Incomplete Checklist: The PR template checklist is not filled out. Please complete this to confirm you've tested the changes properly.
Non-compliant PR Title: Your title doesn't follow our conventional commit format. It should be something like: feat(platform): add AI copilot assistant or similar, with the appropriate type and scope.
Lack of Context: For a change this substantial (adding chat functionality, onboarding, agent generation), we need much more context about:
- Purpose of the feature
- Architecture decisions
- Testing methodology
- Security considerations
User Data Handling: The PR adds several functions that interact with user data. Please confirm that user_id is properly checked in all these functions.

Once you've updated the PR with this information, we'd be happy to review the technical implementation. The code itself looks interesting, but we need the proper context and documentation to evaluate it effectively.

majdyz · 2026-01-06T20:38:44Z

autogpt_platform/backend/backend/api/features/admin/store_admin_routes.py

+    """
+    try:
+        stats = await store_embeddings.get_embedding_stats()
+        return stats


In general, to fix this kind of issue, you should avoid including raw exception messages or stack traces in HTTP responses. Instead, log the detailed error on the server and return a sanitized, generic error object to the client. This maintains debuggability for developers while preventing information disclosure.

For this specific code:

In backend/api/features/store/embeddings.py:

get_embedding_stats currently returns a dict with an "error": str(e) field when an exception occurs. This dict is what propagates to the admin route and then to the client. We should remove the raw exception text from the returned data and replace it with a generic message (for example, "error": "Failed to get embedding stats"). The detailed error is already logged via logger.error, which is safe to keep.

Similarly, backfill_missing_embeddings returns "error": str(e). Even though the provided alert focuses on the stats endpoint, this function has the same pattern and should be sanitized for consistency and safety.

In backend/api/features/admin/store_admin_routes.py, the route-level exception handling is already safe: it logs the exception and raises an HTTPException with a generic detail message. No changes are needed there.

Concretely:

Edit get_embedding_stats’s except block (lines 311–319) to return a dict without str(e) and with a fixed, generic error string.

Edit backfill_missing_embeddings’s except block (lines 385–392) to do the same.

No new imports or methods are required; we reuse existing logging and types.

@Swiftyos does this api needs to be publicly available? is this even used at all?

Ask❓

@Swiftyos does this api needs to be publicly available? is this even used at all?

Answer:

This endpoint is in admin/store_admin_routes.py under the admin router, so it should not be publicly available. It’s a maintenance/observability endpoint (embedding coverage/health) and doesn’t look like something a normal client needs. If the admin router is already protected (admin auth / internal-only), then it’s fine; if not, it should be gated the same way as other admin endpoints.

On “is this even used?”: nothing in the diff you shared shows a frontend caller, and the endpoint name (/embeddings/stats) suggests it’s intended for operational dashboards or manual checks during the embeddings backfill work. If there’s no consumer, we should either:

remove it until we actually need it, or

keep it but document/confirm the intended consumer (admin UI, scripts, monitoring), and add a small test or a note in the PR description.

Also, the code-scanning comment is relevant here: the handler currently has a try: and then return stats, but no visible except block in the hunk. If an exception is raised, FastAPI will return a 500 and (depending on environment/config) may expose stack trace details. To address that:

add an except Exception as e: that logs internally and returns a generic HTTPException(status_code=500, detail="Failed to fetch embedding stats") (no raw exception text), and

ensure the route is protected (admin auth) so even metadata like “coverage %” isn’t exposed to the public.

autogpt_platform/backend/backend/api/features/admin/store_admin_routes.py

+        result = await store_embeddings.backfill_missing_embeddings(
+            batch_size=batch_size
+        )
+        return result


In general, to fix this problem we should avoid returning raw exception messages to the client. Instead, we should log detailed error information on the server, and return only a generic, non-sensitive message (or structured error code) in the API response. That ensures developers can still debug issues from logs, but callers do not see internal details.

For this specific case, the minimal change is inside backfill_missing_embeddings in autogpt_platform/backend/backend/api/features/store/embeddings.py. In the except Exception as e block, we currently both log the error and return a dict containing "error": str(e). We should keep logging (possibly enhancing it to use logger.exception to capture a stack trace) but change the returned dict to omit str(e) and instead include a generic error message that does not disclose implementation details. The admin route in store_admin_routes.py can then safely return this dict as-is without exposing sensitive information.

Concretely:

In embeddings.py, lines 385–392: replace logger.error(f"Failed to backfill embeddings: {e}") with logger.exception("Failed to backfill embeddings") to log the full stack trace server-side, and replace the returned dict so that it no longer includes the raw exception string. For example, return:
return { "processed": 0, "success": 0, "failed": 0, "error": "Failed to backfill embeddings", }

No changes are needed in store_admin_routes.py because once the returned dict no longer contains sensitive data, returning it from the endpoint is safe.

No new imports or helper methods are required; we only adjust logging and the content of the returned dict.

github-actions · 2026-01-05T10:53:01Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

- Updated all chat tools imports from backend.server.v2 to backend.api.features - Updated store imports (backfill_embeddings, db, hybrid_search) - Fixed CredentialsInput import in setup-wizard page

AutoGPT-Agent · 2026-01-05T11:37:19Z

Thank you for your contribution! Before this PR can be considered for merging, several issues need to be addressed:

Missing Description

Please add a clear description of what this PR is implementing. From the code, it looks like you're adding a copilot/chat feature with various capabilities, but this needs to be explicitly stated and explained.

Incomplete Checklist

The PR template checklist needs to be filled out completely. This includes:

Listing your changes
Creating and executing a test plan
Confirming any necessary configuration changes

Title Format

The PR title needs to follow conventional commit format. Based on the changes, something like feat(platform/frontend): implement AI copilot chat interface or similar would be more appropriate.

Scope

This appears to be a very large PR adding multiple features. Please explain:

The overall goal of the copilot feature
Key components you've added
How these components work together
Any configuration requirements for testing

Test Plan

Please add a test plan detailing how you've tested these changes and how reviewers can verify functionality.

Once these items are addressed, we can proceed with a more detailed technical review of the implementation. The code itself looks interesting, but we need proper context and documentation to effectively review it.

github-actions · 2026-01-05T12:04:30Z

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

AutoGPT-Agent · 2026-01-05T12:04:49Z

Thank you for this comprehensive PR adding an AI copilot chat assistant! The implementation looks solid, but before this can be merged, a few items need to be addressed:

Please complete the PR description by filling out the 'Changes' section with a concise summary of what this PR adds to the platform.
The checklist in the PR description is currently empty. Please check off all relevant items to confirm you've completed the necessary steps (testing, configuration checks, etc.).

The code itself looks well-structured with a robust implementation of the chat assistant, including:

Chat session management with database persistence
Multiple tools for agent discovery and creation
Business context understanding
Onboarding system

Once the description and checklist are completed, this PR should be ready for a more detailed review.

majdyz · 2026-01-06T20:38:44Z

autogpt_platform/backend/backend/api/features/admin/store_admin_routes.py

+    """
+    try:
+        stats = await store_embeddings.get_embedding_stats()
+        return stats


@Swiftyos does this api needs to be publicly available? is this even used at all?

majdyz · 2026-01-06T21:18:35Z