Skip to content

feat: faster remote browser loading #860

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 12, 2025
Merged

feat: faster remote browser loading #860

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1f99295
feat: rm canvas logic, add in-browser loader
RohitR311 Nov 3, 2025
58aa5f5
feat: speed up snapshot emission
RohitR311 Nov 3, 2025
d7da903
fix: add recording setup route
RohitR311 Nov 4, 2025
cd53994
Merge branch 'develop' into fast-loader
RohitR311 Nov 5, 2025
fcb3a12
Merge branch 'develop' into fast-loader
RohitR311 Nov 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/components/browser/BrowserContent.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import {
export const BrowserContent = () => {
const { socket } = useSocketStore();

const [tabs, setTabs] = useState<string[]>(["current"]);
const [tabs, setTabs] = useState<string[]>(["Loading..."]);
Copy link

@coderabbitai coderabbitai bot Nov 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Prevent phantom “Loading...” tab from breaking tab indices.

By keeping the placeholder when getCurrentTabs() returns an empty list you end up appending the first real tab after "Loading...". UI indices (1, 2, …) then no longer match the backend’s tab indices (0, 1, …), so actions like changeTab(1) or closeTab(1) target the wrong tab or fail outright. Please allow the empty array to overwrite the placeholder (or otherwise drop the placeholder before appending) so the frontend stays in sync with the backend.

Apply this diff to restore the previous behavior:

-        if (response && response.length > 0) {
-          setTabs(response);
-        }
+        if (Array.isArray(response)) {
+          setTabs(response);
+        }

Also applies to: 128-130

🤖 Prompt for AI Agents 
In src/components/browser/BrowserContent.tsx around lines 16 and 128-130, the
initial state uses a "Loading..." placeholder which is kept when
getCurrentTabs() returns an empty array, causing frontend tab indices to shift;
remove the placeholder by initializing tabs to an empty array ([]) and/or change
the logic that applies backend results so that when getCurrentTabs() returns []
it overwrites any placeholder instead of appending to it (i.e., replace the tabs
array wholesale with the backend array or drop the placeholder before
concatenation) so frontend indices stay aligned with backend indices.

const [tabIndex, setTabIndex] = React.useState(0);
const [showOutputData, setShowOutputData] = useState(false);
const { browserWidth } = useBrowserDimensionsStore();
Expand Down Expand Up @@ -125,7 +125,7 @@ export const BrowserContent = () => {
useEffect(() => {
getCurrentTabs()
.then((response) => {
if (response) {
if (response && response.length > 0) {
setTabs(response);
}
})
Expand Down
146 changes: 23 additions & 123 deletions src/components/browser/BrowserWindow.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
import React, { useCallback, useContext, useEffect, useRef, useState } from 'react';
import { useSocketStore } from '../../context/socket';
import { Button } from '@mui/material';
import Canvas from "../recorder/Canvas";
import { Highlighter } from "../recorder/Highlighter";
import { GenericModal } from '../ui/GenericModal';
import { useActionContext } from '../../context/browserActions';
import { useBrowserSteps, TextStep, ListStep } from '../../context/browserSteps';
Expand Down Expand Up @@ -38,12 +36,6 @@ interface AttributeOption {
value: string;
}

interface ScreencastData {
image: string;
userId: string;
viewport?: ViewportInfo | null;
}

interface ViewportInfo {
width: number;
height: number;
Expand Down Expand Up @@ -146,8 +138,6 @@ const getAttributeOptions = (tagName: string, elementInfo: ElementInfo | null):
export const BrowserWindow = () => {
const { t } = useTranslation();
const { browserWidth, browserHeight } = useBrowserDimensionsStore();
const [canvasRef, setCanvasReference] = useState<React.RefObject<HTMLCanvasElement> | undefined>(undefined);
const [screenShot, setScreenShot] = useState<string>("");
const [highlighterData, setHighlighterData] = useState<{
rect: DOMRect;
selector: string;
Expand Down Expand Up @@ -1303,17 +1293,6 @@ export const BrowserWindow = () => {
}, []);

const onMouseMove = (e: MouseEvent) => {
if (canvasRef && canvasRef.current && highlighterData) {
const canvasRect = canvasRef.current.getBoundingClientRect();
if (
e.pageX < canvasRect.left
|| e.pageX > canvasRect.right
|| e.pageY < canvasRect.top
|| e.pageY > canvasRect.bottom
) {
setHighlighterData(null);
}
}
};

const resetListState = useCallback(() => {
Expand All @@ -1331,46 +1310,22 @@ export const BrowserWindow = () => {
}
}, [getList, resetListState]);

const screencastHandler = useCallback((data: string | ScreencastData) => {
if (typeof data === 'string') {
setScreenShot(data);
} else if (data && typeof data === 'object' && 'image' in data) {
if (!data.userId || data.userId === user?.id) {
setScreenShot(data.image);

if (data.viewport) {
setViewportInfo(data.viewport);
}
}
}
}, [user?.id]);

useEffect(() => {
if (socket) {
socket.on("screencast", screencastHandler);
socket.on("domcast", rrwebSnapshotHandler);
socket.on("dom-mode-enabled", domModeHandler);
socket.on("dom-mode-error", domModeErrorHandler);
}

if (canvasRef?.current && !isDOMMode && screenShot) {
drawImage(screenShot, canvasRef.current);
}

return () => {
if (socket) {
socket.off("screencast", screencastHandler);
socket.off("domcast", rrwebSnapshotHandler);
socket.off("dom-mode-enabled", domModeHandler);
socket.off("dom-mode-error", domModeErrorHandler);
}
};
}, [
socket,
screenShot,
canvasRef,
isDOMMode,
screencastHandler,
rrwebSnapshotHandler,
domModeHandler,
domModeErrorHandler,
Expand Down Expand Up @@ -1847,24 +1802,7 @@ export const BrowserWindow = () => {

const handleClick = (e: React.MouseEvent<HTMLDivElement>) => {
if (highlighterData) {
let shouldProcessClick = false;

if (!isDOMMode && canvasRef?.current) {
const canvasRect = canvasRef.current.getBoundingClientRect();
const clickX = e.clientX - canvasRect.left;
const clickY = e.clientY - canvasRect.top;
const highlightRect = highlighterData.rect;
const mappedRect =
coordinateMapper.mapBrowserRectToCanvas(highlightRect);

shouldProcessClick =
clickX >= mappedRect.left &&
clickX <= mappedRect.right &&
clickY >= mappedRect.top &&
clickY <= mappedRect.bottom;
} else {
shouldProcessClick = true;
}
const shouldProcessClick = true;

if (shouldProcessClick) {
const options = getAttributeOptions(
Expand Down Expand Up @@ -2209,17 +2147,7 @@ export const BrowserWindow = () => {
!showAttributeModal &&
highlighterData?.rect != null && (
<>
{!isDOMMode && canvasRef?.current && (
<Highlighter
unmodifiedRect={highlighterData?.rect}
displayedSelector={highlighterData?.selector}
width={dimensions.width}
height={dimensions.height}
canvasRect={canvasRef.current.getBoundingClientRect()}
/>
)}

{isDOMMode && highlighterData && (
{highlighterData && (
<div
id="dom-highlight-overlay"
style={{
Expand Down Expand Up @@ -2355,31 +2283,27 @@ export const BrowserWindow = () => {
borderRadius: "0px 0px 5px 5px",
}}
>
{isDOMMode ? (
{currentSnapshot ? (
<>
{currentSnapshot ? (
<DOMBrowserRenderer
width={dimensions.width}
height={dimensions.height}
snapshot={currentSnapshot}
getList={getList}
getText={getText}
listSelector={listSelector}
cachedChildSelectors={cachedChildSelectors}
paginationMode={paginationMode}
paginationType={paginationType}
limitMode={limitMode}
isCachingChildSelectors={isCachingChildSelectors}
onHighlight={domHighlighterHandler}
onElementSelect={handleDOMElementSelection}
onShowDatePicker={handleShowDatePicker}
onShowDropdown={handleShowDropdown}
onShowTimePicker={handleShowTimePicker}
onShowDateTimePicker={handleShowDateTimePicker}
/>
) : (
<DOMLoadingIndicator />
)}
<DOMBrowserRenderer
width={dimensions.width}
height={dimensions.height}
snapshot={currentSnapshot}
getList={getList}
getText={getText}
listSelector={listSelector}
cachedChildSelectors={cachedChildSelectors}
paginationMode={paginationMode}
paginationType={paginationType}
limitMode={limitMode}
isCachingChildSelectors={isCachingChildSelectors}
onHighlight={domHighlighterHandler}
onElementSelect={handleDOMElementSelection}
onShowDatePicker={handleShowDatePicker}
onShowDropdown={handleShowDropdown}
onShowTimePicker={handleShowTimePicker}
onShowDateTimePicker={handleShowDateTimePicker}
/>

{/* --- Loading overlay --- */}
{isCachingChildSelectors && (
Expand Down Expand Up @@ -2492,11 +2416,7 @@ export const BrowserWindow = () => {
)}
</>
) : (
<Canvas
onCreateRef={setCanvasReference}
width={dimensions.width}
height={dimensions.height}
/>
<DOMLoadingIndicator />
)}
</div>
</div>
Expand Down Expand Up @@ -2591,26 +2511,6 @@ const DOMLoadingIndicator: React.FC = () => {
);
};


const drawImage = (image: string, canvas: HTMLCanvasElement): void => {
const ctx = canvas.getContext('2d');
if (!ctx) return;

const img = new Image();
img.onload = () => {
requestAnimationFrame(() => {
ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
});
if (image.startsWith('blob:')) {
URL.revokeObjectURL(image);
}
};
img.onerror = () => {
console.warn('Failed to load image');
};
img.src = image;
};

const modalStyle = {
top: '50%',
left: '50%',
Expand Down
43 changes: 21 additions & 22 deletions src/pages/RecordingPage.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ export const RecordingPage = ({ recordingName }: RecordingPageProps) => {

const { setId, socket } = useSocketStore();
const { setWidth } = useBrowserDimensionsStore();
const { browserId, setBrowserId, recordingId, recordingUrl, setRecordingUrl, setRecordingName, setRetrainRobotId } = useGlobalInfoStore();
const { browserId, setBrowserId, recordingId, recordingUrl, setRecordingUrl, setRecordingName, setRetrainRobotId, setIsDOMMode } = useGlobalInfoStore();

const handleShowOutputData = useCallback(() => {
setShowOutputData(true);
Expand Down Expand Up @@ -77,6 +77,8 @@ export const RecordingPage = ({ recordingName }: RecordingPageProps) => {
useEffect(() => {
let isCancelled = false;
const handleRecording = async () => {
setIsDOMMode(true);

const storedUrl = window.sessionStorage.getItem('recordingUrl');
if (storedUrl && !recordingUrl) {
setRecordingUrl(storedUrl);
Expand Down Expand Up @@ -137,9 +139,12 @@ export const RecordingPage = ({ recordingName }: RecordingPageProps) => {
if (browserId === 'new-recording') {
socket?.emit('new-recording');
}
if (recordingUrl && socket) {
socket.emit('input:url', recordingUrl);
}
setIsLoaded(true);
}
}, [socket, browserId, recordingName, recordingId, isLoaded]);
}, [socket, browserId, recordingName, recordingId, recordingUrl, isLoaded]);

useEffect(() => {
socket?.on('loaded', handleLoaded);
Expand All @@ -153,26 +158,20 @@ export const RecordingPage = ({ recordingName }: RecordingPageProps) => {
<ActionProvider>
<BrowserStepsProvider>
<div id="browser-recorder">
{isLoaded ? (
<>
<Grid container direction="row" style={{ flexGrow: 1, height: '100%' }}>
<Grid item xs={12} md={9} lg={9} style={{ height: '100%', overflow: 'hidden', position: 'relative' }}>
<div style={{ height: '100%', overflow: 'auto' }}>
<BrowserContent />
<InterpretationLog isOpen={showOutputData} setIsOpen={setShowOutputData} />
</div>
</Grid>
<Grid item xs={12} md={3} lg={3} style={{ height: '100%', overflow: 'hidden' }}>
<div className="right-side-panel" style={{ height: '100%' }}>
<RightSidePanel onFinishCapture={handleShowOutputData} />
<BrowserRecordingSave />
</div>
</Grid>
</Grid>
</>
) : (
<Loader text={t('recording_page.loader.browser_startup', { url: recordingUrl })} />
)}
<Grid container direction="row" style={{ flexGrow: 1, height: '100%' }}>
<Grid item xs={12} md={9} lg={9} style={{ height: '100%', overflow: 'hidden', position: 'relative' }}>
<div style={{ height: '100%', overflow: 'auto' }}>
<BrowserContent />
<InterpretationLog isOpen={showOutputData} setIsOpen={setShowOutputData} />
</div>
</Grid>
<Grid item xs={12} md={3} lg={3} style={{ height: '100%', overflow: 'hidden' }}>
<div className="right-side-panel" style={{ height: '100%' }}>
<RightSidePanel onFinishCapture={handleShowOutputData} />
<BrowserRecordingSave />
</div>
</Grid>
</Grid>
</div>
</BrowserStepsProvider>
</ActionProvider>
Expand Down