test(editor): Add comprehensive instance AI e2e tests#28326
Draft
test(editor): Add comprehensive instance AI e2e tests#28326
Conversation
Remove type icons from tab labels, make tabs fill full header height, and replace loading spinners with larger loader-circle icon (80px). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…g (no-changelog) Adds a Playwright e2e test that captures the bug where the last node in the instance AI workflow preview stays in "running" state (spinning border) after execution completes. The test sends a specific prompt to build and execute a 3-node workflow, then asserts that no canvas nodes remain with the .running CSS class. Includes InstanceAiPage page object, navigation helper, and test fixtures with N8N_ENABLED_MODULES=instance-ai. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ionFinished (no-changelog) The event relay watcher only forwarded the last event in the log, so when Vue coalesced multiple ref updates into one callback, intermediate events (e.g. nodeExecuteAfter for the last node) were silently dropped. This left the iframe's executing-node queue with a stale entry, keeping the last node in spinning/running state after the workflow finished. - Track relayed event count so every new event is forwarded, even when the watcher fires once for multiple log additions. - Keep the eventLog intact when executionFinished arrives (instead of clearing it immediately) so the relay can forward pending events before sending the synthetic executionFinished. - Add clearEventLog() to useExecutionPushEvents, called by the relay after all pending events have been forwarded. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add 16 Playwright e2e tests across 6 spec files covering instance AI workflow preview, artifacts, timeline, sidebar, confirmations, and chat basics. Wire up proxy-aware fetch in the AI SDK model creation so MockServer can intercept Anthropic API calls for recording/replay. - Expand InstanceAiPage page object with 30+ locators - Add InstanceAiSidebar component page object - Add data-test-id to preview close button - Add getProxyFetch() to model-factory.ts and instance-ai.service.ts so @ai-sdk/anthropic respects HTTP_PROXY in e2e containers - Rewrite fixtures with proxy service recording support - Replace single execution-state test with comprehensive suite Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…hangelog) Add two-tier trace replay system that records tool I/O during e2e test recording and replays with bidirectional ID remapping in CI. This enables deterministic replay of complex multi-step agent tests where tool execution produces dynamic IDs. - New trace-replay.ts: IdRemapper (ID-field-aware), TraceIndex (per-role cursors), TraceWriter, JSONL I/O helpers, PURE_REPLAY_TOOLS set - Modified langsmith-tracing.ts: replayWrapTool (Tier 1: real execution + ID remap), pureReplayWrapTool (Tier 2: pure replay for external deps), recordWrapTool, createTraceReplayOnlyContext stub for non-LangSmith envs - New test-only controller endpoints: POST/GET/DELETE /test/tool-trace with slug-scoped storage for parallel test isolation - Updated fixture: records trace.jsonl during recording, loads for replay, slug-scoped activate/retrieve/clear lifecycle - 23 unit tests for IdRemapper and TraceIndex - Recorded trace.jsonl files for all 15 instance AI test expectations Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…log) Add subString body matching on the system prompt to disambiguate LLM call types (title generation vs orchestrator vs sub-agent) during proxy replay. Without this, sequential expectations could be served to the wrong call when the call order differs between recording and replay. Re-record all expectations with the body matcher and remove debug logging from trace replay wrappers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When re-recording with a real API key, always use record mode (never load old trace events into the backend). Previously, existing trace files would cause the backend to enter replay mode during re-recording, resulting in trace.jsonl files with only a header and no tool calls. Re-record all trace.jsonl files with proper tool call events. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Re-record all proxy expectations after fixing the recording mode logic. Expectations now have subString body matchers on the system prompt and trace.jsonl files have proper tool call events. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The proxy's sequential mode sets the last expectation as unlimited (fallback for extra agent turns). Previously this applied to the last file alphabetically which could be a community_nodes GET. Now it finds the last /v1/messages POST expectation specifically. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…elog) Background task completion triggers `startInternalFollowUpRun`, which creates a new trace context. Previously each context got a fresh TraceIndex with cursor at 0, so the follow-up run's first tool call (e.g. list-workflows) would mismatch the first trace event (build-workflow-with-agent) and throw. Fix: store a shared TraceIndex/IdRemapper per test slug on the service. All runs within the same slug reuse the same instances, preserving cursor state across the initial run and any follow-up runs. This fixes the two confirmation e2e tests that rely on suspend/resume. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…(no-changelog) waitForAssistantResponse only waited for the first message element to appear (streaming start), not for the agent to finish. Sidebar operations then raced against the still-running agent. New waitForResponseComplete waits for the send button to reappear, which only renders when isStreaming becomes false. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…s between tests (no-changelog) Two preview tests failed because their recorded proxy expectations contained stale LLM responses from previous tests' background task follow-ups. The fixture now cancels leftover background tasks before each test via a new test-only endpoint, preventing future cross-test contamination. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ET (no-changelog) MockServer proxy connections intermittently reset when 4 parallel workers load expectations simultaneously. Add withRetry helper with exponential backoff (3 retries, 500ms base) and re-throw on failure instead of silently swallowing the error. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ation (no-changelog) Positional selectors (.last()) break when parallel tests create threads in shared containers. Switch to getThreadByTitle() with LLM-generated titles from recordings. Also handle missing expectations directories gracefully. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
|
❌ 5 Tests Failed:
View the full list of 5 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
Bundle ReportChanges will increase total bundle size by 1.03kB (0.0%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: editor-ui-esmAssets Changed:
Files in
Files in
|
…ure (no-changelog) Covers the record/replay architecture, ID remapping problem and solution, two-tier tool wrapping strategy, trace format, and troubleshooting guide. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ant proxy expectations (no-changelog) - Add unit tests for TraceWriter, parseTraceJsonl, model-factory proxy fetch, clearEventLog, and useEventRelay coalesced event handling - Extract test-only trace replay endpoints into InstanceAiTestController, conditionally registered when N8N_INSTANCE_AI_TRACE_REPLAY is set - Extract trace replay state from InstanceAiService into TraceReplayState class - Remove 83 irrelevant api-staging community nodes expectation files - Fix stale test that expected eventLog cleared on executionFinished Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
! PR exceeds size limit (1,332 lines added)This PR adds 1,332 lines, exceeding the 1,000-line limit (test files excluded). Large PRs are harder to review and increase the risk of bugs going unnoticed. Please consider:
If the size is genuinely justified (e.g. generated code, large migrations, test fixtures), a maintainer can override by commenting |
…tance AI e2e tests (no-changelog) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Performance ComparisonComparing current → latest master → 14-day baseline Idle baseline with Instance AI module loaded
docker-stats
Memory consumption baseline with starter plan resources
How to read this table
|
…laims (no-changelog) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
n8n-assistant
bot
added
core
Contributor
Author
|
/size-limit-override |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
🤖 Generated with Claude Code