feat: add single test execution to QA agent testcase generation workflow #3497
Conversation
Extract single testcase execution logic into a reusable utility function. This enables other tools to execute individual test cases without running the full test suite. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Remove duplicate helper functions (buildCombinedSql, extractFailedOperation) - Use shared executeTestcase utility for consistency - Implement parallel execution with Promise.all for better performance - Maintain backward compatibility 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…hema - Update testcaseAnnotation to use Schema object instead of string - Modify generateTestcaseNode and validateSchemaRequirementsNode to use convertSchemaToText() - Add readonly reducer for schemaData in qaAgentAnnotation to prevent InvalidUpdateError - Update distributeRequirements to pass schemaData directly - Restore and update integration tests with proper Schema mock data - Enable DDL generation and extension extraction for future single test execution 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Use aSchema, aTable, and aColumn builder functions instead of verbose object literals for cleaner test setup. Also remove duplicate config spread operator. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add executeSingleTestNode and invokeSingleTestToolNode to handle test execution after testcase generation - Update routeAfterSave to route to test execution when testcases are available - Add runSingleTestTool for executing individual tests - Integrate test execution nodes into the main testcase generation graph 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
|
WalkthroughMigrates QA agent flows from textual schemaContext to structured schemaData, adds a single-test execution path with new nodes and tool integration, and refactors testcase execution into a reusable executeTestcase utility. Updates annotations (with reducer), nodes, tools, and tests accordingly. Minor payload change in distributeRequirements. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor User
participant QA_Agent as QA Agent Graph
participant ExecuteNode as executeSingleTestNode
participant ToolNode as invokeSingleTestToolNode
participant Tool as runSingleTestTool
participant Exec as executeTestcase
participant DB as Database
User->>QA_Agent: Start/Continue testcase flow
QA_Agent->>QA_Agent: routeAfterSave(state)\n- if testcases.length > 0 → executeSingleTest
QA_Agent->>ExecuteNode: Run node
Note right of ExecuteNode: Emits AIMessage with tool_calls\n(name=runSingleTestTool)
ExecuteNode-->>QA_Agent: messages with tool call
QA_Agent->>ToolNode: Invoke ToolNode with config\n{ testcases, ddlStatements, requiredExtensions }
ToolNode->>Tool: runSingleTestTool(config)
Tool->>Exec: executeTestcase(ddlStatements, testcase, requiredExtensions)
Exec->>DB: executeQuery(combined SQL, extensions)
DB-->>Exec: SqlResults
Exec-->>Tool: TestcaseDmlExecutionResult
Tool-->>ToolNode: Updated state { testcases, messages }
ToolNode-->>QA_Agent: Partial state
QA_Agent-->>User: SSE/messages with single-test result
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✨ Finishing touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
|
Updates to Preview Branch (run-test-after-generating-2) ↗︎
Tasks are run on every commit but only new migration files are pushed.
View logs for this Workflow Run ↗︎. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (5)
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/testcaseAnnotation.ts (1)
19-19: Consider identity reducer for schemaData to avoid merge conflicts.Align with qaAgentAnnotation by adding an identity reducer so parallel subgraphs returning the same schema don’t trigger InvalidUpdateError.
Apply:
- schemaData: Annotation<Schema>, + schemaData: Annotation<Schema>({ reducer: (x) => x }),frontend/internal-packages/agent/src/tools/runTestTool.ts (1)
66-68: Don't overwrite prior execution logs.Replace logs instead of appending; retain history for repeated runs.
Apply:
- const updatedDmlOperation = { - ...dmlOp, - dml_execution_logs: [executionLog], - } + const updatedDmlOperation = { + ...dmlOp, + dml_execution_logs: [ + ...(dmlOp?.dml_execution_logs ?? []), + executionLog, + ], + }frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.ts (1)
36-40: Confirm intended post-save routing behavior.The conditional map includes END but routeAfterSave currently returns only 'generateTestcase' or 'executeSingleTest'. If the intended behavior is to terminate when no testcases exist, consider returning END there; otherwise this may loop back to generation unnecessarily.
frontend/internal-packages/agent/src/utils/executeTestcase.ts (2)
14-31: Small readability nit.Explicitly typing sqlParts as string[] makes intent clear to readers and helps tooling.
Apply this diff:
- const sqlParts = [] + const sqlParts: string[] = []
3-3: Import SqlResult from the package rootsrc/index.ts re-exports './types' — replace the deep import with the public export:
import type { SqlResult } from '@liam-hq/pglite-server'
Location: frontend/internal-packages/agent/src/utils/executeTestcase.ts:3
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (14)
frontend/internal-packages/agent/src/qa-agent/distributeRequirements/index.ts(1 hunks)frontend/internal-packages/agent/src/qa-agent/shared/qaAgentAnnotation.ts(1 hunks)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/executeSingleTestNode.ts(1 hunks)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/generateTestcaseNode.ts(2 hunks)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.integration.test.ts(3 hunks)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.ts(2 hunks)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/invokeSingleTestToolNode.ts(1 hunks)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/routeAfterSave.ts(1 hunks)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/testcaseAnnotation.ts(2 hunks)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.integration.test.ts(5 hunks)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.ts(2 hunks)frontend/internal-packages/agent/src/tools/runSingleTestTool.ts(1 hunks)frontend/internal-packages/agent/src/tools/runTestTool.ts(2 hunks)frontend/internal-packages/agent/src/utils/executeTestcase.ts(1 hunks)
🧰 Additional context used
📓 Path-based instructions (5)
**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
Name utility files in camelCase (e.g., mergeSchema.ts)
Files:
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/generateTestcaseNode.tsfrontend/internal-packages/agent/src/utils/executeTestcase.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/executeSingleTestNode.tsfrontend/internal-packages/agent/src/qa-agent/distributeRequirements/index.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/testcaseAnnotation.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/invokeSingleTestToolNode.tsfrontend/internal-packages/agent/src/tools/runTestTool.tsfrontend/internal-packages/agent/src/qa-agent/shared/qaAgentAnnotation.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.integration.test.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/routeAfterSave.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.integration.test.tsfrontend/internal-packages/agent/src/tools/runSingleTestTool.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Use TypeScript/TSX across the codebase
**/*.{ts,tsx}: Use runtime type validation with valibot for external data validation
Use early returns for readability
Write simple, direct code without backward compatibility concerns—update all call sites together
Use named exports only (no default exports)
Follow existing import patterns and tsconfig paths
Prefer const arrow functions instead of function declarations for simple utilities (e.g., const toggle = () => {})
Files:
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/generateTestcaseNode.tsfrontend/internal-packages/agent/src/utils/executeTestcase.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/executeSingleTestNode.tsfrontend/internal-packages/agent/src/qa-agent/distributeRequirements/index.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/testcaseAnnotation.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/invokeSingleTestToolNode.tsfrontend/internal-packages/agent/src/tools/runTestTool.tsfrontend/internal-packages/agent/src/qa-agent/shared/qaAgentAnnotation.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.integration.test.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/routeAfterSave.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.integration.test.tsfrontend/internal-packages/agent/src/tools/runSingleTestTool.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.ts
frontend/internal-packages/**
📄 CodeRabbit inference engine (AGENTS.md)
Infra and tooling (e2e, configs, storybook, agent) live under frontend/internal-packages
Files:
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/generateTestcaseNode.tsfrontend/internal-packages/agent/src/utils/executeTestcase.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/executeSingleTestNode.tsfrontend/internal-packages/agent/src/qa-agent/distributeRequirements/index.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/testcaseAnnotation.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/invokeSingleTestToolNode.tsfrontend/internal-packages/agent/src/tools/runTestTool.tsfrontend/internal-packages/agent/src/qa-agent/shared/qaAgentAnnotation.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.integration.test.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/routeAfterSave.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.integration.test.tsfrontend/internal-packages/agent/src/tools/runSingleTestTool.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.ts
frontend/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
frontend/**/*.{ts,tsx}: Import UI components from @liam-hq/ui when available
Import icons from @liam-hq/ui
Files:
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/generateTestcaseNode.tsfrontend/internal-packages/agent/src/utils/executeTestcase.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/executeSingleTestNode.tsfrontend/internal-packages/agent/src/qa-agent/distributeRequirements/index.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/testcaseAnnotation.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/invokeSingleTestToolNode.tsfrontend/internal-packages/agent/src/tools/runTestTool.tsfrontend/internal-packages/agent/src/qa-agent/shared/qaAgentAnnotation.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.integration.test.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/routeAfterSave.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.integration.test.tsfrontend/internal-packages/agent/src/tools/runSingleTestTool.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.ts
**/*.test.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Write unit tests with filenames ending in .test.ts or .test.tsx colocated near source
Files:
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.integration.test.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.integration.test.ts
🧠 Learnings (4)
📚 Learning: 2025-07-30T05:52:56.270Z
Learnt from: hoshinotsuyoshi
PR: liam-hq/liam#2771
File: frontend/internal-packages/schema-bench/src/cli/executeLiamDb.ts:22-22
Timestamp: 2025-07-30T05:52:56.270Z
Learning: The schema-bench package (frontend/internal-packages/schema-bench) has been converted from ESM to CommonJS mode by removing "type": "module" from package.json, making __dirname available and correct to use in TypeScript files within this package.
Applied to files:
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/generateTestcaseNode.tsfrontend/internal-packages/agent/src/qa-agent/distributeRequirements/index.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/testcaseAnnotation.tsfrontend/internal-packages/agent/src/tools/runTestTool.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.integration.test.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.integration.test.ts
📚 Learning: 2025-09-12T11:37:37.550Z
Learnt from: NoritakaIkeda
PR: liam-hq/liam#3463
File: frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/saveToolNode.ts:12-22
Timestamp: 2025-09-12T11:37:37.550Z
Learning: In the frontend/internal-packages/agent codebase, state.currentRequirement.requirementId is guaranteed to always be present, so guard checks for missing requirementId are unnecessary in saveToolNode.ts.
Applied to files:
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/generateTestcaseNode.tsfrontend/internal-packages/agent/src/qa-agent/distributeRequirements/index.tsfrontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.ts
📚 Learning: 2025-09-04T09:48:12.223Z
Learnt from: FunamaYukina
PR: liam-hq/liam#3331
File: frontend/internal-packages/agent/src/workflow/nodes/validateInitialSchemaNode.ts:94-103
Timestamp: 2025-09-04T09:48:12.223Z
Learning: The postgresqlSchemaDeparser function from liam-hq/schema always returns a string type, and executeQuery properly handles empty strings. Therefore, using !ddlStatements is sufficient to check for empty/invalid DDL without needing to explicitly check for empty arrays.
Applied to files:
frontend/internal-packages/agent/src/utils/executeTestcase.ts
📚 Learning: 2025-09-04T09:48:12.223Z
Learnt from: FunamaYukina
PR: liam-hq/liam#3331
File: frontend/internal-packages/agent/src/workflow/nodes/validateInitialSchemaNode.ts:94-103
Timestamp: 2025-09-04T09:48:12.223Z
Learning: The postgresqlSchemaDeparser function from liam-hq/schema always returns a string type in the value property (even empty strings for empty schemas), and executeQuery properly handles empty strings. Therefore, using !ddlStatements is sufficient to check for empty/invalid DDL without needing to explicitly check for empty arrays.
Applied to files:
frontend/internal-packages/agent/src/utils/executeTestcase.ts
🧬 Code graph analysis (9)
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/generateTestcaseNode.ts (1)
frontend/internal-packages/agent/src/utils/convertSchemaToText.ts (1)
convertSchemaToText(31-43)
frontend/internal-packages/agent/src/utils/executeTestcase.ts (4)
frontend/internal-packages/agent/src/qa-agent/types.ts (1)
Testcase(15-15)frontend/internal-packages/artifact/src/schemas/artifact.ts (1)
DmlOperation(63-63)frontend/internal-packages/pglite-server/src/types.ts (1)
SqlResult(1-11)frontend/internal-packages/agent/src/qa-agent/validateSchema/types.ts (1)
TestcaseDmlExecutionResult(6-18)
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/executeSingleTestNode.ts (1)
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/testcaseAnnotation.ts (1)
testcaseAnnotation(16-24)
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/invokeSingleTestToolNode.ts (3)
frontend/internal-packages/agent/src/tools/runSingleTestTool.ts (1)
runSingleTestTool(19-130)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/testcaseAnnotation.ts (1)
testcaseAnnotation(16-24)frontend/internal-packages/agent/src/utils/generateDdl.ts (1)
generateDdlFromSchema(7-10)
frontend/internal-packages/agent/src/tools/runTestTool.ts (1)
frontend/internal-packages/agent/src/utils/executeTestcase.ts (1)
executeTestcase(55-87)
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.ts (1)
frontend/internal-packages/agent/src/utils/convertSchemaToText.ts (1)
convertSchemaToText(31-43)
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/routeAfterSave.ts (1)
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/testcaseAnnotation.ts (1)
testcaseAnnotation(16-24)
frontend/internal-packages/agent/src/tools/runSingleTestTool.ts (3)
frontend/internal-packages/agent/src/utils/errorHandling.ts (1)
WorkflowTerminationError(5-15)frontend/internal-packages/agent/src/utils/executeTestcase.ts (1)
executeTestcase(55-87)frontend/internal-packages/agent/src/qa-agent/types.ts (1)
Testcase(15-15)
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.ts (5)
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/executeSingleTestNode.ts (1)
executeSingleTestNode(5-21)frontend/internal-packages/agent/src/utils/errorHandling.ts (1)
RETRY_POLICY(22-32)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/invokeSingleTestToolNode.ts (1)
invokeSingleTestToolNode(9-27)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/routeAfterGenerate.ts (1)
routeAfterGenerate(10-24)frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/routeAfterSave.ts (1)
routeAfterSave(7-17)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
- GitHub Check: frontend-ci
- GitHub Check: frontend-lint
- GitHub Check: Supabase Preview
- GitHub Check: agent-deep-modeling
- GitHub Check: security-review
🔇 Additional comments (17)
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/testcaseAnnotation.ts (1)
2-2: LGTM: switch to typed Schema.Type import is correct and improves safety across the flow.
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/generateTestcaseNode.ts (2)
36-39: LGTM: derive schemaContext from schemaData.Good conversion step to keep prompts stable while carrying typed schema through state.
8-8: Import path verified — no duplicates.
convertSchemaToText is defined at frontend/internal-packages/agent/src/utils/convertSchemaToText.ts; imports like ../../utils/convertSchemaToText resolve to that file and no other copies were found.frontend/internal-packages/agent/src/qa-agent/shared/qaAgentAnnotation.ts (1)
13-18: Good call: identity reducer makes schemaData read-only.Prevents InvalidUpdateError when multiple branches propagate the same schema.
frontend/internal-packages/agent/src/qa-agent/distributeRequirements/index.ts (1)
21-21: LGTM: pass schemaData into subgraph.Matches the new testcaseAnnotation shape and avoids early text conversion.
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.ts (2)
48-51: LGTM: derive textual context from structured schema.Keeps validator logic unchanged while unifying on schemaData.
6-6: Import path verified — no change required.
Referenced file exists at frontend/internal-packages/agent/src/utils/convertSchemaToText.ts, so the '../../utils/convertSchemaToText' import is correct.frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.integration.test.ts (3)
2-2: LGTM: use schema builders from @liam-hq/schema.Consistent with typed schema adoption.
24-44: LGTM: mock schema is minimal yet sufficient.Tables/columns are clear; suitable for the integration scenario.
55-56: LGTM: switch test state to schemaData.Matches runtime changes across nodes.
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/routeAfterSave.ts (2)
1-1: LGTM: type-only import for END.Reduces runtime bundle surface.
9-14: Confirmed graph wiring: executeSingleTest → invokeSingleTestTool → ENDNodes/edges are registered in frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.ts — .addNode('executeSingleTest', ...), .addNode('invokeSingleTestTool', ...), .addEdge('executeSingleTest','invokeSingleTestTool'), .addEdge('invokeSingleTestTool', END) (lines 25–42).
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/index.ts (1)
25-30: Graph wiring for single-test path looks correct.Nodes and edges for executeSingleTest → invokeSingleTestTool → END are well-placed and use RETRY_POLICY consistently.
Also applies to: 41-42
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/executeSingleTestNode.ts (1)
9-18: AI tool call setup is good; ensure toolCallId is propagated.The tool_calls payload is correct. Verify that invokeSingleTestToolNode forwards this toolCallId to the tool via config.configurable so ToolMessage.tool_call_id is populated for SSE correlation.
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/validateSchemaRequirementsNode.integration.test.ts (1)
203-204: Remove debug logging from test output.The console.dir noise can mask regressions in CI logs.
[ suggest_nitpick_refactor ]
Apply this diff:- // biome-ignore lint/suspicious/noConsole: Testing - console.dir({ command }, { depth: null })frontend/internal-packages/agent/src/tools/runSingleTestTool.ts (1)
33-46: ToolMessage.tool_call_id may be undefined without explicit propagation.Relies on invokeSingleTestToolNode to pass toolCallId. After implementing that propagation, re-verify these ToolMessages include a non-empty tool_call_id for proper SSE/UI correlation.
Also applies to: 58-69, 108-114
frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/invokeSingleTestToolNode.ts (1)
13-24: Guard ddlStatements and requiredExtensions; do not forward toolCallId manually.generateDdlFromSchema can return undefined — pass '' to avoid executeTestcase runtime errors; guard schema extensions. ToolCall is injected by the runtime (other invoke* nodes don’t set it), so don’t add manual forwarding.
- const ddlStatements = generateDdlFromSchema(state.schemaData) - const requiredExtensions = Object.keys(state.schemaData.extensions).sort() + const ddlStatements = generateDdlFromSchema(state.schemaData) ?? '' + const requiredExtensions = Object.keys(state.schemaData.extensions ?? {}).sort()File: frontend/internal-packages/agent/src/qa-agent/testcaseGeneration/invokeSingleTestToolNode.ts
Likely an incorrect or invalid review comment.
| // Execute single test case | ||
| const testResult = await executeTestcase( | ||
| ddlStatements, | ||
| targetTestcase, | ||
| requiredExtensions, | ||
| ) |
There was a problem hiding this comment.
Guard against undefined ddlStatements/requiredExtensions before execution.
executeTestcase expects a string and a string[]; passing undefined risks runtime errors.
Apply this diff:
- const testResult = await executeTestcase(
- ddlStatements,
- targetTestcase,
- requiredExtensions,
- )
+ const testResult = await executeTestcase(
+ ddlStatements ?? '',
+ targetTestcase,
+ requiredExtensions ?? [],
+ )📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // Execute single test case | |
| const testResult = await executeTestcase( | |
| ddlStatements, | |
| targetTestcase, | |
| requiredExtensions, | |
| ) | |
| // Execute single test case | |
| const testResult = await executeTestcase( | |
| ddlStatements ?? '', | |
| targetTestcase, | |
| requiredExtensions ?? [], | |
| ) |
🤖 Prompt for AI Agents
In frontend/internal-packages/agent/src/tools/runSingleTestTool.ts around lines
72-77, guard against undefined ddlStatements and requiredExtensions before
calling executeTestcase: ensure ddlStatements is a string (default to empty
string) and requiredExtensions is an array of strings (default to empty array)
or explicitly bail with a clear error; then pass those validated/coerced values
to executeTestcase so the function always receives the expected types and avoids
runtime errors.
| const updatedDmlOperation = { | ||
| ...targetTestcase.dmlOperation, | ||
| dml_execution_logs: [executionLog], | ||
| } | ||
|
|
||
| const updatedTestcase: Testcase = { | ||
| ...targetTestcase, | ||
| dmlOperation: updatedDmlOperation, | ||
| } |
There was a problem hiding this comment.
Preserve prior execution logs instead of overwriting.
Appending maintains history across runs.
Apply this diff:
- const updatedDmlOperation = {
- ...targetTestcase.dmlOperation,
- dml_execution_logs: [executionLog],
- }
+ const prevLogs = targetTestcase.dmlOperation.dml_execution_logs ?? []
+ const updatedDmlOperation = {
+ ...targetTestcase.dmlOperation,
+ dml_execution_logs: [...prevLogs, executionLog],
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const updatedDmlOperation = { | |
| ...targetTestcase.dmlOperation, | |
| dml_execution_logs: [executionLog], | |
| } | |
| const updatedTestcase: Testcase = { | |
| ...targetTestcase, | |
| dmlOperation: updatedDmlOperation, | |
| } | |
| const prevLogs = targetTestcase.dmlOperation.dml_execution_logs ?? [] | |
| const updatedDmlOperation = { | |
| ...targetTestcase.dmlOperation, | |
| dml_execution_logs: [...prevLogs, executionLog], | |
| } | |
| const updatedTestcase: Testcase = { | |
| ...targetTestcase, | |
| dmlOperation: updatedDmlOperation, | |
| } |
🤖 Prompt for AI Agents
In frontend/internal-packages/agent/src/tools/runSingleTestTool.ts around lines
88 to 96, the code replaces dml_execution_logs with only the latest
executionLog; instead, preserve prior logs by appending the new executionLog to
the existing array. Update the creation of updatedDmlOperation to set
dml_execution_logs to the existing dmlExecutionLogs (safely handling undefined:
targetTestcase.dmlOperation?.dml_execution_logs or empty array) plus the new
executionLog, then assign that updatedDmlOperation into updatedTestcase as
before.
| return Promise.all( | ||
| testcases.map((testcase) => | ||
| executeTestcase(ddlStatements, testcase, requiredExtensions), | ||
| ), | ||
| ) |
There was a problem hiding this comment.
Avoid concurrent per-test DDL: race risk and flaky tests.
Executing DDL with every testcase in parallel will collide (e.g., CREATE TABLE/EXTENSION) and cause nondeterministic failures. Run sequentially and include DDL only once.
Apply:
- return Promise.all(
- testcases.map((testcase) =>
- executeTestcase(ddlStatements, testcase, requiredExtensions),
- ),
- )
+ const results: TestcaseDmlExecutionResult[] = []
+ let includeDdl = true
+ for (const tc of testcases) {
+ // Include DDL only for the first testcase to set up the environment once
+ const ddl = includeDdl ? ddlStatements : ''
+ results.push(await executeTestcase(ddl, tc, requiredExtensions))
+ includeDdl = false
+ }
+ return results📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| return Promise.all( | |
| testcases.map((testcase) => | |
| executeTestcase(ddlStatements, testcase, requiredExtensions), | |
| ), | |
| ) | |
| const results: TestcaseDmlExecutionResult[] = [] | |
| let includeDdl = true | |
| for (const tc of testcases) { | |
| // Include DDL only for the first testcase to set up the environment once | |
| const ddl = includeDdl ? ddlStatements : '' | |
| results.push(await executeTestcase(ddl, tc, requiredExtensions)) | |
| includeDdl = false | |
| } | |
| return results |
🤖 Prompt for AI Agents
In frontend/internal-packages/agent/src/tools/runTestTool.ts around lines 29 to
33, the code runs executeTestcase for each testcase in parallel including
per-test DDL which causes races; change this to run the DDL once up front and
then execute testcases sequentially: first apply
ddlStatements/requiredExtensions a single time before iterating, then iterate
testcases with an awaited sequential loop (for..of or reduce with await) calling
executeTestcase without reapplying DDL for each test to avoid CREATE/EXTENSION
collisions and flaky failures.
Summary
Changes Made
routeAfterSaveto route to test execution when testcases are availableTechnical Details
executeTestcaseutility for consistent test executionTest Plan
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Improvements
Tests