Conversation
- Add missing deps to @elizaos/core (drizzle-orm, markdown-it, undici, yaml, sharp) - Fix import paths in autonomy/service.ts and testing/index.ts - Add node types to plugin-anthropic tsconfig to fix uuid type resolution - Fix dependency declarations in several plugins (dangling refs, version ranges) - Upgrade @pixi/react to v8 to match pixi.js v8 in dnd-vtt example Co-authored-by: Cursor <cursoragent@cursor.com>
Manages @elizaos/* dependency references across the monorepo: bun run fix-deps — set workspace:* for local dev bun run fix-deps:restore — restore versioned refs before committing bun run fix-deps:check — CI check for leaked workspace:* refs Co-authored-by: Cursor <cursoragent@cursor.com>
…er detection Explicit env vars (API keys, local URLs) take priority over auto-detected local servers, so users with cloud keys aren't silently overridden by a running Ollama instance. Co-authored-by: Cursor <cursoragent@cursor.com>
…L JSON translation - Schema builders: support complex SQL expression indexes (e.g. ((metadata->>'type'))) and multi-column expression indexes; use sql.raw() for PG and dialect translation for MySQL - buildTable constraints callback now returns array (Drizzle non-deprecated API) - DialectAdapter: add optional translateExpression(expr) for PG→MySQL JSON operator translation - MySQL adapter: translate ->'', ->, ? to JSON_UNQUOTE(JSON_EXTRACT), JSON_EXTRACT, JSON_CONTAINS_PATH - Apply translation to expression indexes and check constraints when building MySQL tables - Fix DialectAdapter.createTable type to (table) => any[] in core and plugin-sql types - Use Object.values in constraint loops to avoid unused key variables Co-authored-by: Cursor <cursoragent@cursor.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Too many files changed for review. ( |
There was a problem hiding this comment.
Pull request overview
Introduces new benchmarking modules (BFCL + AgentBench) and updates repo CI/configuration to support multi-language testing and releases.
Changes:
- Added BFCL benchmark CLI, evaluators, and metrics computation.
- Added AgentBench Python package with adapters, mock runtime, CLI, and tests.
- Updated CI/workflows and added database API documentation/benchmark artifacts; removed some legacy repo configs.
Reviewed changes
Copilot reviewed 299 out of 956 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| benchmarks/bfcl/metrics.py | Adds BFCL metrics aggregation, error analysis, and baseline comparisons |
| benchmarks/bfcl/evaluators/relevance_evaluator.py | Adds relevance (no-tool) correctness evaluator |
| benchmarks/bfcl/evaluators/exec_evaluator.py | Adds mock execution evaluator for function calls |
| benchmarks/bfcl/evaluators/ast_evaluator.py | Adds structural (AST) matcher for function calls/args |
| benchmarks/bfcl/evaluators/init.py | Exposes BFCL evaluators package API |
| benchmarks/bfcl/main.py | Adds BFCL CLI entrypoint (run/models/info) |
| benchmarks/bfcl/init.py | Exposes BFCL public API exports |
| benchmarks/bench_cli_types.py | Adds shared CLI JSON/benchmark typing helpers |
| benchmarks/agentbench/smart_smoke/agentbench-results.json | Adds sample AgentBench results artifact |
| benchmarks/agentbench/smart_smoke/agentbench-report.md | Adds sample AgentBench report artifact |
| benchmarks/agentbench/smart_smoke/agentbench-detailed.json | Adds sample AgentBench detailed run artifact |
| benchmarks/agentbench/pyproject.toml | Adds AgentBench Python package metadata and tooling config |
| benchmarks/agentbench/elizaos_agentbench/trajectory_integration.py | Adds optional trajectory logger integration helpers |
| benchmarks/agentbench/elizaos_agentbench/tests/test_types.py | Adds unit tests for AgentBench types/baselines |
| benchmarks/agentbench/elizaos_agentbench/tests/test_smart_mock_runtime.py | Adds end-to-end deterministic harness test |
| benchmarks/agentbench/elizaos_agentbench/tests/test_runner.py | Adds runner + memory tracker tests |
| benchmarks/agentbench/elizaos_agentbench/tests/test_adapters.py | Adds adapter behavior tests across environments |
| benchmarks/agentbench/elizaos_agentbench/tests/init.py | Marks AgentBench tests package |
| benchmarks/agentbench/elizaos_agentbench/mock_runtime.py | Adds deterministic mock runtime for harness validation |
| benchmarks/agentbench/elizaos_agentbench/cli.py | Adds AgentBench CLI runner and env selection |
| benchmarks/agentbench/elizaos_agentbench/adapters/base.py | Adds base adapter interface and task loop implementation |
| benchmarks/agentbench/elizaos_agentbench/adapters/init.py | Exposes AgentBench adapters API |
| benchmarks/agentbench/elizaos_agentbench/init.py | Exposes AgentBench package API exports |
| benchmarks/agentbench/README.md | Adds AgentBench documentation and usage |
| benchmarks/init.py | Adds benchmarks package marker |
| benchmark_results/bfcl/bfcl_best_results.json | Adds BFCL best-results artifact |
| README.md | Updates branding and some command examples |
| Dockerfile | Removes docker build definition |
| DATABASE_API_README.md | Adds database adapter API documentation |
| DATABASE_API_CHANGELOG.md | Adds database API refactor changelog |
| BENCHMARK_RESULTS.md | Adds write/read benchmark comparison documentation |
| AGENTS.md | Removes repository guidelines doc |
| .vscode/settings.json | Removes editor settings |
| .vscode/launch.json | Removes debug launch configurations |
| .prettierignore | Removes Prettier ignore list |
| .npmrc | Updates npm registry auth configuration |
| .husky/pre-commit | Removes husky pre-commit hook |
| .gitmodules | Removes submodule configuration |
| .github/workflows/weekly-maintenance.yml | Adjusts weekly maintenance workflow settings |
| .github/workflows/update-news.yml | Removes news update workflow |
| .github/workflows/tee-build-deploy.yml | Refactors TEE deployment workflow env handling/concurrency |
| .github/workflows/tauri-ci.yml | Removes Tauri CI workflow |
| .github/workflows/supply-chain.yaml | Adds SBOM + vulnerability scan workflow |
| .github/workflows/release.yaml | Updates release workflow (WASM build, workspace version rewriting) |
| .github/workflows/release-python.yaml | Adds PyPI publishing workflow |
| .github/workflows/pr.yaml | Updates PR title check to set GH_TOKEN for commenting |
| .github/workflows/plugin-sql-tests.yaml | Removes SQL plugin-specific workflow |
| .github/workflows/multi-lang-tests.yaml | Adds Rust/Python/WASM/interop test workflow |
| .github/workflows/jsdoc-automation.yml | Updates JSDoc automation workflow (autodoc existence check) |
| .github/workflows/image.yaml | Updates Docker build workflow actions to pinned major versions |
| .github/workflows/generate-readme-translations.yml | Removes README translation workflow |
| .github/workflows/docs-ci.yml | Adds docs CI workflow (links + quality fixes via Claude) |
| .github/workflows/core-package-tests.yaml | Removes core-package test workflow |
| .github/workflows/codeql.yml | Updates CodeQL workflow triggers/formatting |
| .github/workflows/client-cypress-tests.yml | Removes client Cypress workflow |
| .github/workflows/cli-tests.yml | Removes CLI tests workflow |
| .github/workflows/claude.yml | Updates Claude workflow (timeout + checkout version) |
| .github/workflows/claude-security-review.yml | Updates Claude security review workflow (timeout + formatting) |
| .github/workflows/claude-code-review.yml | Updates Claude code review workflow (timeout + expectations) |
| .github/workflows/ci.yaml | Expands CI to include Python/Rust tests and adjusts paths/timeouts |
| .github/workflows/README.md | Updates workflows documentation |
| .github/renovate-preset.json | Updates Renovate grouping rules and naming |
| .github/dependabot.yml | Updates Dependabot config formatting (currently invalid value) |
| .github/ISSUE_TEMPLATE/feature_request.md | Normalizes YAML quoting |
| .github/ISSUE_TEMPLATE/bug_report.md | Normalizes YAML quoting |
| .dockerignore | Removes dockerignore file |
| .cursorrules | Removes Cursor rules file |
| .cursorignore | Removes Cursor ignore file |
| .biomeignore | Adds Biome ignore config |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
isinstance does not accept PEP-604 unions (int | float) as the second argument; this will raise TypeError at runtime. Replace it with a tuple type check (e.g., (int, float)) and consider excluding bool explicitly since bool is a subclass of int.
| if isinstance(value, (int, float)) and not isinstance(value, bool): |
There was a problem hiding this comment.
execution_error can never be incremented because the loop continues whenever result.ast_match is true, but the execution error condition requires result.ast_match. Fix by restructuring the control flow so AST matches can still be checked for exec_success (e.g., handle execution errors before the continue, or only continue after all relevant counters are updated).
There was a problem hiding this comment.
execution_error can never be incremented because the loop continues whenever result.ast_match is true, but the execution error condition requires result.ast_match. Fix by restructuring the control flow so AST matches can still be checked for exec_success (e.g., handle execution errors before the continue, or only continue after all relevant counters are updated).
There was a problem hiding this comment.
The percentile indexing is not computed consistently with common quantile definitions (e.g., for even n, p50 is biased upward; more generally, int(n * p) is a coarse bucket). Consider using statistics.quantiles(..., method=...) (or an equivalent interpolation approach) and/or using (n - 1) * p with rounding/interpolation for more accurate p50/p95/p99.
| def percentile(p: float) -> float: | |
| """Compute the p-th percentile using linear interpolation.""" | |
| if n == 1: | |
| return sorted_latencies[0] | |
| idx = (n - 1) * p | |
| lower = int(idx) | |
| upper = min(lower + 1, n - 1) | |
| weight = idx - lower | |
| lower_val = sorted_latencies[lower] | |
| upper_val = sorted_latencies[upper] | |
| return lower_val * (1.0 - weight) + upper_val * weight | |
| return { | |
| "avg": statistics.mean(latencies), | |
| "p50": percentile(0.5), | |
| "p95": percentile(0.95), | |
| "p99": percentile(0.99), |
There was a problem hiding this comment.
Using eval() is still unsafe even with __builtins__ removed; crafted inputs can sometimes access dangerous objects via Python object model traversal. Since this is a benchmark mock, consider replacing eval with a small safe expression evaluator (e.g., parse with ast and whitelist numeric literals/operators) to avoid accidental execution risks when running benchmarks on untrusted inputs.
There was a problem hiding this comment.
package-ecosystem cannot be empty; Dependabot will reject this config. Set it to a valid ecosystem (e.g., npm, github-actions, pip, etc.) and add additional entries if you intend to cover multiple ecosystems.
| - package-ecosystem: "github-actions" # See documentation for possible values |
| ## Installation | ||
|
|
||
| ```bash | ||
| # From the benchmarks/agentbench/python directory |
There was a problem hiding this comment.
The install instructions reference benchmarks/agentbench/python, but this package’s pyproject.toml and sources appear to live directly under benchmarks/agentbench/. Update the path in the README so users run the commands from the correct directory.
| # From the benchmarks/agentbench directory |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
benchmarks/bfcl/reporting.py (1)
383-410:⚠️ Potential issue | 🟡 MinorRank numbering will duplicate after insertion.
Incrementing the
enumerateloop variable doesn’t carry to the next iteration, so ranks after the inserted elizaOS row repeat. Use an explicitrankcounter instead.Proposed fix
- for i, (model_name, baseline) in enumerate(sorted_baselines, 1): + rank = 1 + for model_name, baseline in sorted_baselines: # Insert elizaOS in the right position if not elizaos_added and metrics.overall_score > baseline.overall: lines.append( - f"| **{i}** | **elizaOS** | " + f"| **{rank}** | **elizaOS** | " f"**{metrics.overall_score:.2%}** | " f"**{metrics.ast_accuracy:.2%}** | " f"**{metrics.exec_accuracy:.2%}** |" ) elizaos_added = True - i += 1 + rank += 1 lines.append( - f"| {i} | {baseline.model_name} | " + f"| {rank} | {baseline.model_name} | " f"{baseline.overall:.2%} | " f"{baseline.ast:.2%} | " f"{baseline.exec:.2%} |" ) + rank += 1 @@ - if not elizaos_added: + if not elizaos_added: lines.append( - f"| **{len(sorted_baselines) + 1}** | **elizaOS** | " + f"| **{rank}** | **elizaOS** | " f"**{metrics.overall_score:.2%}** | " f"**{metrics.ast_accuracy:.2%}** | " f"**{metrics.exec_accuracy:.2%}** |" )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@benchmarks/bfcl/reporting.py` around lines 383 - 410, The rank numbers duplicate because you incremented the enumerate loop variable (i) which doesn't persist; replace the enumerate logic with an explicit rank counter: create rank = 1 before looping over sorted_baselines, remove usage of enumerate and the i += 1 line, and inside the loop use rank when appending the elizaOS row and baseline row, incrementing rank by 1 after each appended row; after the loop, if elizaos_added is still False append elizaOS using the current rank (instead of len(sorted_baselines)+1) so subsequent ranks stay consistent; update references to i to use rank and keep elizaos_added, sorted_baselines, metrics, and lines unchanged.examples/atropos/textworld/elizaos_atropos_textworld/agent.py (1)
274-295:⚠️ Potential issue | 🟡 MinorClass name violates Python naming convention (PEP 8).
elizaOSAgentstarts with lowercase, but PEP 8 requires CapWords for class names. The standard would beElizaOSAgentorElizaosAgent.This is a public API (exported in
__init__.py), so renaming requires updating imports across the codebase (atropos_integration.py, etc.). Consider whether brand consistency ("elizaOS" as a unit) outweighs style convention compliance.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/atropos/textworld/elizaos_atropos_textworld/agent.py` around lines 274 - 295, Rename the class elizaOSAgent to follow PEP8 (e.g., ElizaOSAgent): update the class declaration name, replace all usages/imports referencing elizaOSAgent (including exports in __init__.py and imports in modules like atropos_integration.py and any tests/examples) to the new name, and add a backwards-compatible alias assignment (elizaOSAgent = ElizaOSAgent) in the module to preserve the public API during rollout.examples/code/src/__tests__/sub-agents.smoke.test.ts (1)
57-82:⚠️ Potential issue | 🟠 Major*Use
bun:testinstead ofvitestin this .test.ts file.The file imports from vitest; repo guidelines require bun:test for all test files.
Change
-import { describe, expect, test } from "vitest"; +import { describe, expect, test } from "bun:test";🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/code/src/__tests__/sub-agents.smoke.test.ts` around lines 57 - 82, Replace vitest imports with bun:test in this test file: change any imports like "import { describe, test, expect } from 'vitest'" to import the same symbols from "bun:test", and remove or adjust any vitest-only helpers if present; ensure the test functions (describe, test, expect) used in the OpenCodeSubAgent and elizaOSNativeSubAgent smoke tests remain referenced exactly as before (e.g., in the two test blocks for "OpenCodeSubAgent completes on DONE response" and "elizaOSNativeSubAgent completes on DONE response") so Bun recognizes the tests.examples/code/src/lib/sub-agents/elizaos-native-sub-agent.ts (1)
347-364:⚠️ Potential issue | 🟡 MinorRename class and factory function to follow camelCase/PascalCase conventions.
Class
elizaOSNativeSubAgentshould beElizaOSNativeSubAgent(PascalCase for classes), and factorycreateelizaOSNativeSubAgentshould becreateElizaOSNativeSubAgent(proper camelCase). Update all 11 call sites: registry.ts, index.ts, sub-agents.smoke.test.ts, and elizaos-native-sub-agent.ts.Affected locations
- Line 347: JSDoc
- Line 362: Class definition
- Line 928-930: Factory function JSDoc and definition
- Line 935: Instantiation in factory
- registry.ts line 4: Import
- registry.ts line 84: Instantiation
- index.ts line 6: Export
- sub-agents.smoke.test.ts lines 3, 70-71: Import and usage
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/code/src/lib/sub-agents/elizaos-native-sub-agent.ts` around lines 347 - 364, Rename the class elizaOSNativeSubAgent to ElizaOSNativeSubAgent and rename the factory createelizaOSNativeSubAgent to createElizaOSNativeSubAgent, updating the class declaration and factory definition in elizaos-native-sub-agent.ts as well as all references/imports/usages (registry.ts, index.ts, sub-agents.smoke.test.ts and any other call sites) to the new identifiers; ensure exported names, JSDoc entries mentioning elizaOSNativeSubAgent and createelizaOSNativeSubAgent are updated to the PascalCase/camelCase forms and that instantiations (new ElizaOSNativeSubAgent() or call to createElizaOSNativeSubAgent()) and any registry registrations use the new names consistently.
🧹 Nitpick comments (5)
examples/browser-use/python/run.py (1)
60-62: Consider specifying encoding when opening JSON file.For cross-platform consistency (especially on Windows), explicitly pass
encoding="utf-8".Proposed fix
if character_file.exists(): - with open(character_file) as f: + with open(character_file, encoding="utf-8") as f: return json.load(f)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/browser-use/python/run.py` around lines 60 - 62, The JSON file is opened without an explicit encoding which can cause cross-platform inconsistencies; update the open call that checks character_file.exists() (the with open(character_file) as f: block used before json.load(f)) to pass encoding="utf-8" so the file is read deterministically across platforms and then call json.load(f) as before.examples/art/elizaos_art/eliza_integration/trajectory_plugin_integration.py (1)
486-502: Fallback export methods are stubs.
export_openpipe_artimports the exporter but doesn't call it, andexport_grpo_groupsjust returns a path without writing data. If the plugin import fails and fallback is used, these methods silently produce nothing.Consider either implementing the actual export logic or raising
NotImplementedErrorto make the limitation explicit.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/art/elizaos_art/eliza_integration/trajectory_plugin_integration.py` around lines 486 - 502, The fallback export methods currently do not perform any export: export_openpipe_art imports export_trajectories_art_format but never calls it (leaving trajectories unwritten) and export_grpo_groups only returns a path without writing groups; update export_openpipe_art to collect trajectories from self._logger (using self._logger.list_trajectories() and self._logger.load_trajectory(tid)) and call export_trajectories_art_format(...) to write to out_path (or, if you prefer not to implement fallback behavior here, explicitly raise NotImplementedError with a clear message), and similarly implement export_grpo_groups to serialize and write the group data to out_path (using self._logger.data_dir for default path) or raise NotImplementedError so the absence of fallback behavior is explicit.examples/chat/typescript/chat.ts (1)
146-161: Local providers always show ❌ regardless of actual status.Since this runs only after
loadLLMPlugin()fails, this is functionally fine. However, a user with Ollama running but misconfigured might be confused seeing "not detected" when the server is up. Consider clarifying the message or making detection async.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/chat/typescript/chat.ts` around lines 146 - 161, printAvailableProviders currently always marks local providers as "❌ ... (not detected)" because it assumes loadLLMPlugin failed; update it to either perform an async runtime check for local providers or clarify the message: create a helper like hasLocalProvider(provider) that attempts a lightweight connection/check (e.g., ping Ollama) and await it in an async printAvailableProviders to show ✅ when reachable, or change the static text for local entries to "❌ or misconfigured — not detected or misconfigured (start the server or set ...)" so users aren’t misled; reference the function printAvailableProviders, the LLM_PROVIDERS list, and the existing hasValidApiKey helper when implementing the new detection or message.DOCUMENTATION_COMPLETE.md (1)
51-70: Add language specifier to fenced code block.Static analysis flags this code block lacks a language identifier. Use
textorplaintextfor ASCII diagrams.📝 Suggested fix
-``` +```text /root/eliza260106/ │ ├── DATABASE_API_DOCS_INDEX.md ← START HERE (navigation)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@DOCUMENTATION_COMPLETE.md` around lines 51 - 70, The fenced ASCII diagram in DOCUMENTATION_COMPLETE.md is missing a language identifier; update the opening fence for that diagram from ``` to ```text (or ```plaintext) so the block is language-specified for linters and renderers; locate the ASCII tree diagram block shown under the repository root snippet and change its fence accordingly.benchmarks/tau-bench/elizaos_tau_bench/__init__.py (1)
35-66: Consider a backward-compat alias for the renamed export.
RemovingElizaOSTauAgentfrom__all__is a breaking change for downstream imports; an alias keeps migrations smoother.♻️ Backward-compat alias
from elizaos_tau_bench.eliza_agent import ( elizaOSTauAgent, MockTauAgent, create_tau_agent, ELIZAOS_AVAILABLE, ) + +# Backward-compat export (deprecated) +ElizaOSTauAgent = elizaOSTauAgent ... __all__ = [ # Core components "TauBenchRunner", "TauBenchEvaluator", "TauAgent", # elizaOS-integrated agents "elizaOSTauAgent", + "ElizaOSTauAgent", "MockTauAgent",🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@benchmarks/tau-bench/elizaos_tau_bench/__init__.py` around lines 35 - 66, The PR removed the old export name ElizaOSTauAgent which breaks downstream imports; restore a backward-compatible alias by assigning ElizaOSTauAgent = elizaOSTauAgent in this module and include "ElizaOSTauAgent" in the __all__ list alongside "elizaOSTauAgent" (ensure you reference the existing symbol elizaOSTauAgent and the module-level __all__ list). This keeps the new name (elizaOSTauAgent) while preserving the legacy import for consumers.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@BENCHMARK_RESULTS.md`:
- Around line 15-24: Multiple fenced code blocks use bare ``` markers causing
markdownlint MD040; update each triple-backtick fence to include a language
identifier (e.g., change ``` to ```text) for every benchmark table block such as
the blocks starting with "WRITE OPERATIONS (N=10,000)", the other table blocks
referenced in the comment (the ones around lines 28-40, 58-66, 75-87, 95-105,
113-119), and any similar fenced tables so that each opening fence is ```text
(and keep the closing fence as ```); this will satisfy MD040 and improve
rendering.
In `@DATABASE_API_README.md`:
- Line 161: The README's API table incorrectly documents
deleteParticipants(participants) as returning boolean; update the documentation
entry for the deleteParticipants method to show a void return (no value) and
adjust the descriptive text if needed to match the design principle that
deleteParticipants returns void rather than boolean so it aligns with the actual
method signature and behavior.
- Around line 118-119: The README table is inconsistent: change the return types
for updateAgents and deleteAgents to match the design (they should be void and
throw on failure rather than returning boolean). Update the entries for
`updateAgents(updates)` and `deleteAgents(ids)` in the table to show `void` as
the return type and ensure any descriptive text reflects that these methods
throw on failure instead of returning success/failure.
In `@examples/app/capacitor/capacitor.config.ts`:
- Line 5: Change the string literal for the appName property to use single
quotes to comply with the repo Prettier settings; locate the appName entry
(appName: "elizaOS Chat",) in capacitor.config.ts and replace the double-quoted
value with a single-quoted one to match the project's formatting rules.
In `@examples/atropos/diplomacy/pyproject.toml`:
- Around line 7-8: The package version in pyproject.toml ("version = \"2.0.0\"")
is out of sync with the module version constant __version__ = "1.0.0" in
__init__.py; update the __version__ symbol in the package's __init__.py (or
alternatively update pyproject.toml) so both declare the same semantic version
(e.g., set __version__ to "2.0.0") and ensure any packaging or CI that reads
either value is consistent.
In `@examples/code/src/__tests__/sub-agents.smoke.test.ts`:
- Line 3: The import and usages reference a class named elizaOSNativeSubAgent
which should be renamed to PascalCase ElizaOSNativeSubAgent; update the import
statement to import { ElizaOSNativeSubAgent } from
"../lib/sub-agents/elizaos-native-sub-agent.js" and update all references/usages
(including the instantiation or type mentions around lines 70–71) to use
ElizaOSNativeSubAgent instead of elizaOSNativeSubAgent, ensuring any exported
class name in the source module matches the new PascalCase identifier.
---
Outside diff comments:
In `@benchmarks/bfcl/reporting.py`:
- Around line 383-410: The rank numbers duplicate because you incremented the
enumerate loop variable (i) which doesn't persist; replace the enumerate logic
with an explicit rank counter: create rank = 1 before looping over
sorted_baselines, remove usage of enumerate and the i += 1 line, and inside the
loop use rank when appending the elizaOS row and baseline row, incrementing rank
by 1 after each appended row; after the loop, if elizaos_added is still False
append elizaOS using the current rank (instead of len(sorted_baselines)+1) so
subsequent ranks stay consistent; update references to i to use rank and keep
elizaos_added, sorted_baselines, metrics, and lines unchanged.
In `@examples/atropos/textworld/elizaos_atropos_textworld/agent.py`:
- Around line 274-295: Rename the class elizaOSAgent to follow PEP8 (e.g.,
ElizaOSAgent): update the class declaration name, replace all usages/imports
referencing elizaOSAgent (including exports in __init__.py and imports in
modules like atropos_integration.py and any tests/examples) to the new name, and
add a backwards-compatible alias assignment (elizaOSAgent = ElizaOSAgent) in the
module to preserve the public API during rollout.
In `@examples/code/src/__tests__/sub-agents.smoke.test.ts`:
- Around line 57-82: Replace vitest imports with bun:test in this test file:
change any imports like "import { describe, test, expect } from 'vitest'" to
import the same symbols from "bun:test", and remove or adjust any vitest-only
helpers if present; ensure the test functions (describe, test, expect) used in
the OpenCodeSubAgent and elizaOSNativeSubAgent smoke tests remain referenced
exactly as before (e.g., in the two test blocks for "OpenCodeSubAgent completes
on DONE response" and "elizaOSNativeSubAgent completes on DONE response") so Bun
recognizes the tests.
In `@examples/code/src/lib/sub-agents/elizaos-native-sub-agent.ts`:
- Around line 347-364: Rename the class elizaOSNativeSubAgent to
ElizaOSNativeSubAgent and rename the factory createelizaOSNativeSubAgent to
createElizaOSNativeSubAgent, updating the class declaration and factory
definition in elizaos-native-sub-agent.ts as well as all
references/imports/usages (registry.ts, index.ts, sub-agents.smoke.test.ts and
any other call sites) to the new identifiers; ensure exported names, JSDoc
entries mentioning elizaOSNativeSubAgent and createelizaOSNativeSubAgent are
updated to the PascalCase/camelCase forms and that instantiations (new
ElizaOSNativeSubAgent() or call to createElizaOSNativeSubAgent()) and any
registry registrations use the new names consistently.
---
Nitpick comments:
In `@benchmarks/tau-bench/elizaos_tau_bench/__init__.py`:
- Around line 35-66: The PR removed the old export name ElizaOSTauAgent which
breaks downstream imports; restore a backward-compatible alias by assigning
ElizaOSTauAgent = elizaOSTauAgent in this module and include "ElizaOSTauAgent"
in the __all__ list alongside "elizaOSTauAgent" (ensure you reference the
existing symbol elizaOSTauAgent and the module-level __all__ list). This keeps
the new name (elizaOSTauAgent) while preserving the legacy import for consumers.
In `@DOCUMENTATION_COMPLETE.md`:
- Around line 51-70: The fenced ASCII diagram in DOCUMENTATION_COMPLETE.md is
missing a language identifier; update the opening fence for that diagram from
``` to ```text (or ```plaintext) so the block is language-specified for linters
and renderers; locate the ASCII tree diagram block shown under the repository
root snippet and change its fence accordingly.
In `@examples/art/elizaos_art/eliza_integration/trajectory_plugin_integration.py`:
- Around line 486-502: The fallback export methods currently do not perform any
export: export_openpipe_art imports export_trajectories_art_format but never
calls it (leaving trajectories unwritten) and export_grpo_groups only returns a
path without writing groups; update export_openpipe_art to collect trajectories
from self._logger (using self._logger.list_trajectories() and
self._logger.load_trajectory(tid)) and call export_trajectories_art_format(...)
to write to out_path (or, if you prefer not to implement fallback behavior here,
explicitly raise NotImplementedError with a clear message), and similarly
implement export_grpo_groups to serialize and write the group data to out_path
(using self._logger.data_dir for default path) or raise NotImplementedError so
the absence of fallback behavior is explicit.
In `@examples/browser-use/python/run.py`:
- Around line 60-62: The JSON file is opened without an explicit encoding which
can cause cross-platform inconsistencies; update the open call that checks
character_file.exists() (the with open(character_file) as f: block used before
json.load(f)) to pass encoding="utf-8" so the file is read deterministically
across platforms and then call json.load(f) as before.
In `@examples/chat/typescript/chat.ts`:
- Around line 146-161: printAvailableProviders currently always marks local
providers as "❌ ... (not detected)" because it assumes loadLLMPlugin failed;
update it to either perform an async runtime check for local providers or
clarify the message: create a helper like hasLocalProvider(provider) that
attempts a lightweight connection/check (e.g., ping Ollama) and await it in an
async printAvailableProviders to show ✅ when reachable, or change the static
text for local entries to "❌ or misconfigured — not detected or misconfigured
(start the server or set ...)" so users aren’t misled; reference the function
printAvailableProviders, the LLM_PROVIDERS list, and the existing hasValidApiKey
helper when implementing the new detection or message.
| ``` | ||
| WRITE OPERATIONS (N=10,000) | OLD (singular API) | NEW (batch-first API) | ||
| ─────────────────────────────┼───────────────────────┼──────────────────────── | ||
| | loop batch spd | loop batch spd | ||
| createAgents | 6964ms 7021ms 1.0x | 2642ms 490ms 5.4x | ||
| createEntities | 4231ms 710ms 6.0x | 3627ms 217ms 16.7x | ||
| createMemories | 8384ms 8365ms 1.0x | 4912ms 443ms 11.1x | ||
| updateAgents | 3899ms 3956ms 1.0x | 2618ms 220ms 11.9x | ||
| upsertAgents | [NOT AVAILABLE] | 488ms 485ms 1.0x | ||
| ``` |
There was a problem hiding this comment.
Add language identifiers to fenced blocks.
markdownlint MD040 flags these fences; add text (or another appropriate language) to keep lint clean and improve rendering.
🔧 Suggested fix
-```
+```text
WRITE OPERATIONS (N=10,000) | OLD (singular API) | NEW (batch-first API)
@@
-```
+```text
Operation OLD batch NEW batch Speedup Change
@@
-```
+```text
WRITE OPERATIONS (N=100,000) — NEW batch-first API
@@
-```
+```text
Query OLD (10K) NEW (10K)
@@
-```
+```text
Query OLD (10K) NEW (10K) Index
@@
-```
+```text
Query 10K rows 100K rows (NEW)Also applies to: 28-40, 58-66, 75-87, 95-105, 113-119
🧰 Tools
🪛 markdownlint-cli2 (0.20.0)
[warning] 15-15: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@BENCHMARK_RESULTS.md` around lines 15 - 24, Multiple fenced code blocks use
bare ``` markers causing markdownlint MD040; update each triple-backtick fence
to include a language identifier (e.g., change ``` to ```text) for every
benchmark table block such as the blocks starting with "WRITE OPERATIONS
(N=10,000)", the other table blocks referenced in the comment (the ones around
lines 28-40, 58-66, 75-87, 95-105, 113-119), and any similar fenced tables so
that each opening fence is ```text (and keep the closing fence as ```); this
will satisfy MD040 and improve rendering.
| | `updateAgents(updates)` | `boolean` | Modify agents | | ||
| | `deleteAgents(ids)` | `boolean` | Remove agents | |
There was a problem hiding this comment.
Documentation inconsistency: return types don't match design principles.
Lines 26-27 and 52-58 state that update* and delete* methods return void (throw on failure). However, this table shows updateAgents and deleteAgents returning boolean.
📝 Suggested fix
| `updateAgents(updates)` | `void` | Modify agents |
| `deleteAgents(ids)` | `void` | Remove agents |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | `updateAgents(updates)` | `boolean` | Modify agents | | |
| | `deleteAgents(ids)` | `boolean` | Remove agents | | |
| | `updateAgents(updates)` | `void` | Modify agents | | |
| | `deleteAgents(ids)` | `void` | Remove agents | |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@DATABASE_API_README.md` around lines 118 - 119, The README table is
inconsistent: change the return types for updateAgents and deleteAgents to match
the design (they should be void and throw on failure rather than returning
boolean). Update the entries for `updateAgents(updates)` and `deleteAgents(ids)`
in the table to show `void` as the return type and ensure any descriptive text
reflects that these methods throw on failure instead of returning
success/failure.
| | `deleteRooms(ids)` | `void` | Remove rooms | | ||
| | `upsertRooms(rooms)` | `void` | Insert or update | | ||
| | `createRoomParticipants(entityIds, roomId)` | `UUID[]` | Add entities to room | | ||
| | `deleteParticipants(participants)` | `boolean` | Remove from room | |
There was a problem hiding this comment.
Same inconsistency: deleteParticipants shows boolean return.
Per the design principles stated earlier, this should return void.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@DATABASE_API_README.md` at line 161, The README's API table incorrectly
documents deleteParticipants(participants) as returning boolean; update the
documentation entry for the deleteParticipants method to show a void return (no
value) and adjust the descriptive text if needed to match the design principle
that deleteParticipants returns void rather than boolean so it aligns with the
actual method signature and behavior.
| const config: CapacitorConfig = { | ||
| appId: "com.elizaos.example.chat", | ||
| appName: "ElizaOS Chat", | ||
| appName: "elizaOS Chat", |
There was a problem hiding this comment.
Use single quotes to match Prettier config.
Line 5 should follow the repo’s Prettier rule for single quotes.
Suggested fix
- appName: "elizaOS Chat",
+ appName: 'elizaOS Chat',As per coding guidelines: “Use Prettier for formatting with 2 spaces, semicolons, single quotes, trailing comma es5, width 100.”
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| appName: "elizaOS Chat", | |
| appName: 'elizaOS Chat', |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@examples/app/capacitor/capacitor.config.ts` at line 5, Change the string
literal for the appName property to use single quotes to comply with the repo
Prettier settings; locate the appName entry (appName: "elizaOS Chat",) in
capacitor.config.ts and replace the double-quoted value with a single-quoted one
to match the project's formatting rules.
| version = "2.0.0" | ||
| description = "Diplomacy multi-agent environment for ElizaOS RL training with Atropos" | ||
| description = "Diplomacy multi-agent environment for elizaOS RL training with Atropos" |
There was a problem hiding this comment.
Version mismatch with __init__.py.
pyproject.toml declares version 2.0.0, but __init__.py has __version__ = "1.0.0". Consider syncing these values.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@examples/atropos/diplomacy/pyproject.toml` around lines 7 - 8, The package
version in pyproject.toml ("version = \"2.0.0\"") is out of sync with the module
version constant __version__ = "1.0.0" in __init__.py; update the __version__
symbol in the package's __init__.py (or alternatively update pyproject.toml) so
both declare the same semantic version (e.g., set __version__ to "2.0.0") and
ensure any packaging or CI that reads either value is consistent.
| import type { IAgentRuntime } from "@elizaos/core"; | ||
| import { describe, expect, test } from "vitest"; | ||
| import { ElizaOSNativeSubAgent } from "../lib/sub-agents/elizaos-native-sub-agent.js"; | ||
| import { elizaOSNativeSubAgent } from "../lib/sub-agents/elizaos-native-sub-agent.js"; |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
cat -n examples/code/src/__tests__/sub-agents.smoke.test.ts | head -80Repository: elizaOS/eliza
Length of output: 2965
🏁 Script executed:
cat -n examples/code/src/lib/sub-agents/elizaos-native-sub-agent.js | head -50Repository: elizaOS/eliza
Length of output: 149
🏁 Script executed:
fd "elizaos-native-sub-agent" examples/code/srcRepository: elizaOS/eliza
Length of output: 117
🏁 Script executed:
fd "elizaos-native-sub-agent"Repository: elizaOS/eliza
Length of output: 117
🏁 Script executed:
cat -n examples/code/src/lib/sub-agents/elizaos-native-sub-agent.ts | head -50Repository: elizaOS/eliza
Length of output: 1933
🏁 Script executed:
cat -n examples/code/src/lib/sub-agents/elizaos-native-sub-agent.ts | tail -100Repository: elizaOS/eliza
Length of output: 3637
🏁 Script executed:
grep -n "^class elizaOSNativeSubAgent\|^export class elizaOSNativeSubAgent" examples/code/src/lib/sub-agents/elizaos-native-sub-agent.tsRepository: elizaOS/eliza
Length of output: 117
Rename class to PascalCase.
elizaOSNativeSubAgent is a class and should follow PascalCase naming convention; rename to ElizaOSNativeSubAgent.
💡 Local alias option
-import { elizaOSNativeSubAgent } from "../lib/sub-agents/elizaos-native-sub-agent.js";
+import { elizaOSNativeSubAgent as ElizaOSNativeSubAgent } from "../lib/sub-agents/elizaos-native-sub-agent.js";
@@
- const agent = new elizaOSNativeSubAgent({
+ const agent = new ElizaOSNativeSubAgent({Also applies to lines 70–71.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@examples/code/src/__tests__/sub-agents.smoke.test.ts` at line 3, The import
and usages reference a class named elizaOSNativeSubAgent which should be renamed
to PascalCase ElizaOSNativeSubAgent; update the import statement to import {
ElizaOSNativeSubAgent } from "../lib/sub-agents/elizaos-native-sub-agent.js" and
update all references/usages (including the instantiation or type mentions
around lines 70–71) to use ElizaOSNativeSubAgent instead of
elizaOSNativeSubAgent, ensuring any exported class name in the source module
matches the new PascalCase identifier.
odilitime
changed the title
…memory API
Rust host:
- Add get_memories_by_ids (batch get) with default impl on DatabaseAdapter/UnifiedDatabaseAdapter
- Add UpdateMemoryItem as partial update (id + optional content, metadata, etc.); update_memories takes &[UpdateMemoryItem]
- Add create_memory_with_unique; create_memory delegates with unique=None
- Integration test MockDatabaseAdapter: implement update_memories(UpdateMemoryItem), add CreateMemoryItem import
Plugins (TS/Python):
- plugin-memory, plugin-planning: migrate from getMemoryManager/get_memory_manager to runtime createMemory/getMemories/updateMemory/deleteMemory (create_memory, get_memories, update_memory, delete_memory in Python)
Tests/fixes:
- message_service: fix test_should_respond_ignore_short_circuits by setting VALIDATION_LEVEL=trusted so mock IGNORE response is accepted
- platform.rs: fix define_platform_trait doctests (import macro, use trait Name [] { } syntax)
Misc: tsconfig/package.json/build updates across examples and plugins.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
Claude finished @odilitime's task in 2m 34s —— View job PR Review Complete ✅Overall Assessment: This is a well-designed database refactor with significant performance improvements (14x+ for batch operations). The batch-first API design is sound and the documentation is excellent. However, there are a few issues to address. ❌ CRITICAL: Missing Tests for
|
| Method | Documented | Design Principle (lines 52-58) |
|---|---|---|
updateAgents |
boolean |
Should be void |
deleteAgents |
boolean |
Should be void |
deleteParticipants |
boolean |
Should be void |
The actual interface (packages/typescript/src/types/database.ts:379-380) shows boolean returns, but the design principles state void. Either update the interface or the documentation to be consistent.
⚠️ IMPORTANT: Benchmark Bug (Already Flagged)
benchmarks/bfcl/reporting.py:394 - The i += 1 inside enumerate loop doesn't persist. This was already identified by CodeRabbit but worth confirming it gets fixed.
💡 SUGGESTIONS
-
In-memory adapter
patchComponent: Consider implementingpatchComponentinplugin-inmemorydbto maintain interface parity, even if it's a simple JS object manipulation. -
Test runner detection: The component tests still use single-item methods (
createComponent,updateComponent) rather than the new batch methods. Consider adding explicit batch method tests. -
Type safety improvement:
plugins/plugin-sql/typescript/base.ts:518usesas anycast incountMemories. Consider adding a proper type union instead.
✅ What's Good
- Batch-first design is well-documented with clear rationale
- 14x+ performance improvements validated with benchmarks
- Path validation in
patchComponentproperly prevents SQL injection - Comprehensive test coverage (62 test files in plugin-sql, 68 in core)
- Good separation of concerns - stores, adapters, types cleanly separated
- Plugin schema registration enables ORM-agnostic plugin data storage
- Transient error retry logic correctly distinguishes permanent vs retryable errors
Summary
The core database refactor is solid and ready for merge after:
- Adding
patchComponenttests - Fixing doc/interface return type inconsistencies
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
| ## 📚 Documentation Structure | ||
|
|
||
| ``` | ||
| /root/eliza260106/ |
There was a problem hiding this comment.
Internal developer paths leaked into committed documentation
Low Severity
This file contains developer-specific local filesystem paths: /root/eliza260106/ in the directory structure diagram and /root/.cursor/plans/database_api_review_e3a9d084.plan.md in the "Future Work" section. These reference a specific developer's machine directory and a Cursor IDE internal plan file, neither of which are meaningful to other contributors or exist in the repository.
Additional Locations (1)
…upsertComponents, patchComponent, upsertMemories - Add optional entityContext/options.entityContext to the five methods in core (IDatabaseAdapter, DatabaseAdapter, InMemoryAdapter) and runtime pass-through. - Plugin-sql base: when entityContext set, run inside withEntityContext so Postgres RLS applies; shared createProxyWithDb for both transaction branches. - Other adapters (LocalDB, InMemoryDB, MySQL base): accept and ignore entityContext. - MySQL base queryEntities: destructure entityContext out before calling store. - Unit tests (pg/pglite adapter, entity-context paths); integration test for adapter.queryEntities with entityContext under RLS. - README section 'Entity context and RLS' with WHYs; CHANGELOG; code comments (WHY). Co-authored-by: Cursor <cursoragent@cursor.com>
|
Claude finished @odilitime's task in 2m 36s —— View job PR Review: Database RefactorTasks:
SummaryOverall this is a well-designed database refactor with solid test coverage (119 test files in plugin-sql using vitest). The batch-first API design is sound, path validation for Issues❌ CRITICAL: Missing Tests for
|
| Method | Documented | Design Principle |
|---|---|---|
updateAgents |
boolean |
void |
deleteAgents |
boolean |
void |
deleteParticipants |
boolean |
void |
The actual interface (packages/typescript/src/types/database.ts:379-380) matches the docs (boolean). Sync documentation to match actual implementation or vice versa.
💡 SUGGESTIONS
-
Type safety in
base.ts: Twoas anycasts at lines 572 and 1202:- Line 572:
stores.countMemories(this.db, roomIdOrParams as any, unique, tableName) - Line 1202:
proxy.db = dbOrTx as any;
Consider adding proper type definitions to eliminate these casts.
- Line 572:
-
InMemory adapter
patchComponent: Consider implementingpatchComponentinplugin-inmemorydbto maintain interface parity.
✅ What's Good
- Test framework: All tests use vitest correctly (131 files with
@elizaos/coreimports) - Path validation:
patchComponentproperly validates paths withPATH_SEGMENT_REto prevent SQL injection - No hardcoded secrets: Config uses
env()substitution correctly - Proper imports: All imports use
@elizaos/core(notpackages/typescript) - Comprehensive RLS tests: New
rls-entity.test.tscovers entity context - CI passing: PR title checks and security review pass
Verdict
The core database refactor is solid. Address the critical issues before merge:
- Add
patchComponenttests - Fix benchmark enumerate bug
- Sync doc/interface return types
1 participant
built on #6496
db go zoom zoom
not done
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Refactor
Chores
Note
Low Risk
Predominantly documentation and string/comment updates, with only a minor class rename in the Mind2Web benchmark agent that could affect imports/type references but not core runtime behavior.
Overview
Adds new Database API documentation artifacts (
DATABASE_API_README.md,DATABASE_API_CHANGELOG.md) plus a newBENCHMARK_RESULTS.mdcapturing old-vs-new batch-first DB API performance measurements and rationale.Standardizes branding text across the benchmarks code/docs/results (renaming “ElizaOS” to “elizaOS” in READMEs, CLI output, reports, and comments) and includes a small code-level rename in Mind2Web by renaming
ElizaOSMind2WebAgenttoelizaOSMind2WebAgent(and updating factory return types/error strings accordingly).Written by Cursor Bugbot for commit 3bfe9c2. This will update automatically on new commits. Configure here.