Conversation
…gation, and enhanced serialization
…`, update tests for cookie handling
…cookie handling logic, update related tests
…an up unused dependencies
…time.CastArgAt` for clarity and consistency
… with `runtime.CastArgAt` and updating type handling for consistency
…streamline frame filtering logic
…proving frame loading and validation logic
…ole-based capability resolution logic with new tests
…ctions and improve capability resolution logic with new tests
… and streamline type handling logic
- Introduced a backend capability testing matrix in `capabilities_test.go` to verify interface implementation consistency across backends. - Added `read_query_contract_test.go` to validate shared query and attribute semantics for document and element targets. - Updated memory driver implementations to refine error handling and improve style parsing behavior.
…ce navigation readiness checks
…anager close behavior
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR primarily improves integration testing coverage and stability for the web/html and web/article modules, while refactoring the HTML module APIs to accept broader “target” types (page/document/element) and updating driver internals (memory driver rename, CDP session/event handling).
Changes:
- Added extensive new FQL integration tests plus dynamic/static test fixtures used by
lab run. - Refactored
modules/web/htmllib functions to use capability-based targets (query/interaction/wait/content/attribute) and expanded CDP session/event infrastructure. - Updated build/test workflow (
Makefile,scripts/modules.sh) and bumped workspace Go version + module deps.
Reviewed changes
Copilot reviewed 283 out of 441 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/modules/web/html/dynamic/element/inner_html/get.fql | Adds INNER_HTML element integration coverage |
| tests/modules/web/html/dynamic/element/iframes/wait_class.fql | Adds iframe wait-class test (frame index usage) |
| tests/modules/web/html/dynamic/element/iframes/select_single.fql | Adds iframe SELECT integration test |
| tests/modules/web/html/dynamic/element/iframes/input.fql | Adds iframe INPUT integration test |
| tests/modules/web/html/dynamic/element/iframes/hover.fql | Adds iframe HOVER integration test |
| tests/modules/web/html/dynamic/element/hover/hover_by_xpath.fql | Adds HOVER XPath variant test |
| tests/modules/web/html/dynamic/element/hover/hover_by_css.fql | Adds HOVER CSS variant test |
| tests/modules/web/html/dynamic/element/hover/hover.fql | Adds HOVER element-handle variant test |
| tests/modules/web/html/dynamic/element/focus/focus_by_xpath.fql | Adds FOCUS XPath variant test |
| tests/modules/web/html/dynamic/element/focus/focus_by_css.fql | Adds FOCUS CSS variant test |
| tests/modules/web/html/dynamic/element/focus/focus.fql | Adds FOCUS element-handle variant test |
| tests/modules/web/html/dynamic/element/click/click_with_count.fql | Adds CLICK count test |
| tests/modules/web/html/dynamic/element/click/click_by_xpath_with_count.fql | Adds CLICK XPath + count test |
| tests/modules/web/html/dynamic/element/click/click_by_xpath.fql | Adds CLICK via XPath within element |
| tests/modules/web/html/dynamic/element/click/click_by_css_with_count.fql | Adds CLICK CSS + count test |
| tests/modules/web/html/dynamic/element/click/click_by_css.fql | Adds CLICK via CSS within element |
| tests/modules/web/html/dynamic/element/click/click.fql | Adds CLICK element-handle test |
| tests/modules/web/html/dynamic/element/clear/clear_by_xpath.fql | Adds INPUT_CLEAR XPath variant test |
| tests/modules/web/html/dynamic/element/clear/clear_by_css.fql | Adds INPUT_CLEAR CSS variant test |
| tests/modules/web/html/dynamic/element/clear/clear.fql | Adds INPUT_CLEAR element-handle test |
| tests/modules/web/html/dynamic/element/children/get_by_index.fql | Adds children indexing test |
| tests/modules/web/html/dynamic/element/children/get.fql | Adds children collection test |
| tests/modules/web/html/dynamic/element/children/count.fql | Adds children length assertions |
| tests/modules/web/html/dynamic/element/blur/blur_by_xpath.fql | Adds BLUR XPath variant test |
| tests/modules/web/html/dynamic/element/blur/blur_by_css.fql | Adds BLUR CSS variant test |
| tests/modules/web/html/dynamic/element/blur/blur.fql | Adds BLUR element-handle test |
| tests/modules/web/html/dynamic/element/attrs/set_many.fql | Adds multi-attribute set test |
| tests/modules/web/html/dynamic/element/attrs/set.fql | Adds single attribute set test |
| tests/modules/web/html/dynamic/element/attrs/remove_style.fql | Adds style removal via ATTR_REMOVE test |
| tests/modules/web/html/dynamic/element/attrs/remove.fql | Adds non-style attribute removal test |
| tests/modules/web/html/dynamic/element/attrs/get_2.fql | Adds ATTR_GET style object test |
| tests/modules/web/html/dynamic/element/attrs/get.fql | Adds basic attribute access test |
| tests/modules/web/html/dynamic/doc/xpath/query.fql | Adds XPATH query integration test |
| tests/modules/web/html/dynamic/doc/xpath/count.fql | Adds XPATH count integration test |
| tests/modules/web/html/dynamic/doc/xpath/attr.fql | Adds XPATH attribute extraction test |
| tests/modules/web/html/dynamic/doc/waitfor_event/navigation.fql | Adds WAITFOR EVENT navigation test |
| tests/modules/web/html/dynamic/doc/waitfor_event/frame_navigation.fql | Adds WAITFOR EVENT frame navigation test |
| tests/modules/web/html/dynamic/doc/waitfor_event/custom_event.fql | Adds WAITFOR EVENT custom DOM event test |
| tests/modules/web/html/dynamic/doc/wait/style_by_xpath.fql | Adds WAIT_STYLE XPath variant |
| tests/modules/web/html/dynamic/doc/wait/style_by_css.fql | Adds WAIT_STYLE CSS variant |
| tests/modules/web/html/dynamic/doc/wait/style_all_by_xpath.fql | Adds WAIT_STYLE_ALL XPath variant |
| tests/modules/web/html/dynamic/doc/wait/style_all_by_css.fql | Adds WAIT_STYLE_ALL CSS variant |
| tests/modules/web/html/dynamic/doc/wait/no_style_by_xpath.fql | Adds WAIT_NO_STYLE XPath variant |
| tests/modules/web/html/dynamic/doc/wait/no_style_by_css.fql | Adds WAIT_NO_STYLE CSS variant |
| tests/modules/web/html/dynamic/doc/wait/no_style_all_by_xpath.fql | Adds WAIT_NO_STYLE_ALL XPath variant |
| tests/modules/web/html/dynamic/doc/wait/no_style_all_by_css.fql | Adds WAIT_NO_STYLE_ALL CSS variant |
| tests/modules/web/html/dynamic/doc/wait/no_element_by_xpath.fql | Adds WAIT_NO_ELEMENT XPath variant |
| tests/modules/web/html/dynamic/doc/wait/no_element_by_css.fql | Adds WAIT_NO_ELEMENT CSS variant |
| tests/modules/web/html/dynamic/doc/wait/no_class_by_xpath.fql | Adds WAIT_NO_CLASS XPath variant |
| tests/modules/web/html/dynamic/doc/wait/no_class_by_css.fql | Adds WAIT_NO_CLASS CSS variant |
| tests/modules/web/html/dynamic/doc/wait/no_class_all_by_xpath.fql | Adds WAIT_NO_CLASS_ALL XPath variant |
| tests/modules/web/html/dynamic/doc/wait/no_class_all_by_css.fql | Adds WAIT_NO_CLASS_ALL CSS variant |
| tests/modules/web/html/dynamic/doc/wait/no_attr_by_xpath.fql | Adds WAIT_NO_ATTR XPath variant |
| tests/modules/web/html/dynamic/doc/wait/no_attr_by_css.fql | Adds WAIT_NO_ATTR CSS variant |
| tests/modules/web/html/dynamic/doc/wait/frame_navigation.fql | Adds WAIT_NAVIGATION frame option test |
| tests/modules/web/html/dynamic/doc/wait/element_by_xpath.fql | Adds WAIT_ELEMENT XPath variant |
| tests/modules/web/html/dynamic/doc/wait/element_by_css.fql | Adds WAIT_ELEMENT CSS variant |
| tests/modules/web/html/dynamic/doc/wait/class_by_xpath.fql | Adds WAIT_CLASS XPath variant |
| tests/modules/web/html/dynamic/doc/wait/class_by_css.fql | Adds WAIT_CLASS CSS variant |
| tests/modules/web/html/dynamic/doc/wait/class_all_by_xpath.fql | Adds WAIT_CLASS_ALL XPath variant |
| tests/modules/web/html/dynamic/doc/wait/class_all_by_css.fql | Adds WAIT_CLASS_ALL CSS variant |
| tests/modules/web/html/dynamic/doc/wait/attr_by_xpath.fql | Adds WAIT_ATTR XPath variant |
| tests/modules/web/html/dynamic/doc/wait/attr_by_css.fql | Adds WAIT_ATTR CSS variant |
| tests/modules/web/html/dynamic/doc/wait/attr_all_by_xpath.fql | Adds WAIT_ATTR_ALL XPath variant |
| tests/modules/web/html/dynamic/doc/wait/attr_all_by_css.fql | Adds WAIT_ATTR_ALL CSS variant |
| tests/modules/web/html/dynamic/doc/viewport/size.fql | Adds viewport sizing integration coverage |
| tests/modules/web/html/dynamic/doc/select/single_by_xpath.fql | Adds SELECT single XPath variant |
| tests/modules/web/html/dynamic/doc/select/single_by_css.fql | Adds SELECT single CSS variant |
| tests/modules/web/html/dynamic/doc/select/multi_by_xpath.fql | Adds SELECT multi XPath variant |
| tests/modules/web/html/dynamic/doc/select/multi_by_css.fql | Adds SELECT multi CSS variant |
| tests/modules/web/html/dynamic/doc/pagination_by_xpath.fql | Adds PAGINATION XPath variant |
| tests/modules/web/html/dynamic/doc/pagination_by_css.fql | Adds PAGINATION CSS variant |
| tests/modules/web/html/dynamic/doc/input/input_by_xpath.fql | Adds INPUT XPath variant |
| tests/modules/web/html/dynamic/doc/input/input_by_css.fql | Adds INPUT CSS variant |
| tests/modules/web/html/dynamic/doc/inner_text/set.fql | Adds INNER_TEXT_SET integration test |
| tests/modules/web/html/dynamic/doc/inner_text/get_by_xpath_all.fql | Adds INNER_TEXT_ALL XPath variant |
| tests/modules/web/html/dynamic/doc/inner_text/get_by_xpath.fql | Adds INNER_TEXT XPath variant |
| tests/modules/web/html/dynamic/doc/inner_text/get_by_css_all.fql | Adds INNER_TEXT_ALL CSS variant |
| tests/modules/web/html/dynamic/doc/inner_text/get_by_css.fql | Adds INNER_TEXT CSS variant |
| tests/modules/web/html/dynamic/doc/inner_text/get.fql | Adds INNER_TEXT document-level test |
| tests/modules/web/html/dynamic/doc/inner_html/set.fql | Adds INNER_HTML_SET integration test |
| tests/modules/web/html/dynamic/doc/inner_html/get_by_xpath_all.fql | Adds INNER_HTML_ALL XPath variant |
| tests/modules/web/html/dynamic/doc/inner_html/get_by_xpath.fql | Adds INNER_HTML XPath variant |
| tests/modules/web/html/dynamic/doc/inner_html/get_by_css_all.fql | Adds INNER_HTML_ALL CSS variant |
| tests/modules/web/html/dynamic/doc/inner_html/get_by_css.fql | Adds INNER_HTML CSS variant |
| tests/modules/web/html/dynamic/doc/inner_html/get.fql | Adds INNER_HTML document-level test |
| tests/modules/web/html/dynamic/doc/iframes/wait_class.fql | Adds iframe wait-class via FRAMES helper |
| tests/modules/web/html/dynamic/doc/iframes/lookup.fql | Adds FRAMES selection test |
| tests/modules/web/html/dynamic/doc/iframes/live_refresh.fql | Adds frames list stability test |
| tests/modules/web/html/dynamic/doc/iframes/length.fql | Adds frames length test |
| tests/modules/web/html/dynamic/doc/iframes/input.fql | Adds iframe INPUT doc-level variant |
| tests/modules/web/html/dynamic/doc/iframes/hover.fql | Adds iframe HOVER doc-level variant |
| tests/modules/web/html/dynamic/doc/iframes/element_exists.fql | Adds iframe ELEMENT_EXISTS checks |
| tests/modules/web/html/dynamic/doc/focus/focus_by_xpath.fql | Adds doc-level FOCUS XPath test |
| tests/modules/web/html/dynamic/doc/focus/focus_by_css.fql | Adds doc-level FOCUS CSS test |
| tests/modules/web/html/dynamic/doc/element/exists_by_xpath.fql | Adds ELEMENT_EXISTS XPath test |
| tests/modules/web/html/dynamic/doc/element/exists_by_css.fql | Adds ELEMENT_EXISTS CSS test |
| tests/modules/web/html/dynamic/doc/click/click_with_count.fql | Adds doc-level CLICK count test |
| tests/modules/web/html/dynamic/doc/click/click_by_xpath_with_count.fql | Adds doc-level CLICK XPath + count test |
| tests/modules/web/html/dynamic/doc/click/click_by_xpath.fql | Adds doc-level CLICK XPath test |
| tests/modules/web/html/dynamic/doc/click/click_by_css_with_count.fql | Adds doc-level CLICK CSS + count test |
| tests/modules/web/html/dynamic/doc/click/click_by_css.fql | Adds doc-level CLICK CSS selector test |
| tests/modules/web/html/dynamic/doc/click/click.fql | Adds doc-level CLICK test |
| tests/modules/web/article/static/text_markdown.fql | Adds WEB::ARTICLE TEXT/MARKDOWN integration checks |
| tests/modules/web/article/static/non_article.fql | Adds non-article negative coverage |
| tests/modules/web/article/static/extract.fql | Adds WEB::ARTICLE::EXTRACT fixture-based assertions |
| tests/modules/web/article/static/element.fql | Adds element-scoped extraction assertions |
| tests/data/pages/static/simple.html | Adds static non-article fixture |
| tests/data/pages/static/article.html | Adds article fixture with safe/unsafe content |
| tests/data/pages/dynamic/utils/random.js | Adds dynamic fixture helper |
| tests/data/pages/dynamic/utils/qs.js | Adds dynamic fixture querystring parsing |
| tests/data/pages/dynamic/index.js | Adds dynamic SPA bootstrap |
| tests/data/pages/dynamic/index.html | Adds dynamic SPA HTML shell |
| tests/data/pages/dynamic/index.css | Adds SPA stylesheet |
| tests/data/pages/dynamic/components/pages/pagination/index.js | Adds pagination page fixture |
| tests/data/pages/dynamic/components/pages/navigation/index.js | Adds navigation page fixture |
| tests/data/pages/dynamic/components/pages/media/index.js | Adds viewport reporting page fixture |
| tests/data/pages/dynamic/components/pages/lists/index.js | Adds list page fixture for children tests |
| tests/data/pages/dynamic/components/pages/index.js | Adds home page fixture content |
| tests/data/pages/dynamic/components/pages/iframes/index.js | Adds iframe page fixture with redirecting |
| tests/data/pages/dynamic/components/pages/forms/index.js | Adds forms page fixture for input/select tests |
| tests/data/pages/dynamic/components/pages/events/pressable.js | Adds keydown fixture |
| tests/data/pages/dynamic/components/pages/events/index.js | Adds events page fixture composition |
| tests/data/pages/dynamic/components/pages/events/hoverable.js | Adds hover fixture |
| tests/data/pages/dynamic/components/pages/events/focusable.js | Adds focus/blur fixture (with random delay) |
| tests/data/pages/dynamic/components/pages/events/clickable.js | Adds click toggling fixture |
| tests/data/pages/dynamic/components/pages/events/appearable.js | Adds appear/disappear fixture |
| tests/data/pages/dynamic/components/pages/events/ajax.js | Adds request-triggering fixture |
| tests/data/pages/dynamic/components/layout.js | Adds SPA layout+navbar fixture |
| tests/data/pages/dynamic/components/app.js | Adds SPA router with redirect handling |
| scripts/modules.sh | Splits unit vs integration test logic; builds runtime |
| modules/yaml/go.mod | Bumps Ferret dependency |
| modules/xml/go.mod | Bumps Ferret dependency |
| modules/web/sitemap/go.mod | Bumps Ferret dependency |
| modules/web/robots/go.mod | Bumps Ferret dependency |
| modules/web/html/lib/xpath.go | Expands XPATH target support (query target) |
| modules/web/html/lib/wait_style_all.go | Uses root wait target for WAIT_STYLE_ALL |
| modules/web/html/lib/wait_style.go | Uses root wait target for WAIT_STYLE |
| modules/web/html/lib/wait_navigation_test.go | Adds unit test for WAIT_NAVIGATION params |
| modules/web/html/lib/wait_navigation.go | Switches to navigation target + JSON tags |
| modules/web/html/lib/wait_element.go | Uses root wait target for WAIT_ELEMENT |
| modules/web/html/lib/wait_class_all.go | Uses root wait target for WAIT_CLASS_ALL |
| modules/web/html/lib/wait_class.go | Uses root wait target for WAIT_CLASS |
| modules/web/html/lib/wait_attr_all.go | Uses root wait target for WAIT_ATTR_ALL |
| modules/web/html/lib/wait_attr.go | Uses root wait target for WAIT_ATTR |
| modules/web/html/lib/set_inner_text.go | Uses root content target for INNER_TEXT_SET |
| modules/web/html/lib/set_inner_html.go | Uses root content target for INNER_HTML_SET |
| modules/web/html/lib/select.go | Expands SELECT to work via root interaction target |
| modules/web/html/lib/scroll_xy.go | Uses viewport target for scrolling |
| modules/web/html/lib/scroll_top.go | Uses viewport target for scrolling |
| modules/web/html/lib/scroll_element.go | Refactors scroll into view to interaction targets |
| modules/web/html/lib/scroll_bottom.go | Uses viewport target for scrolling |
| modules/web/html/lib/screenshot.go | Uses snapshot target for screenshots |
| modules/web/html/lib/root_element.go | Adds root element/capability helpers |
| modules/web/html/lib/press_selector.go | Uses root interaction target for PRESS selector |
| modules/web/html/lib/press.go | Uses root interaction target for PRESS |
| modules/web/html/lib/pdf.go | Uses snapshot target for PDF |
| modules/web/html/lib/parse.go | Strengthens arg casting/validation for PARSE |
| modules/web/html/lib/pagination.go | Uses interaction target for pagination clicking |
| modules/web/html/lib/navigate_forward.go | Uses navigation target for forward nav |
| modules/web/html/lib/navigate_back.go | Uses navigation target for back nav |
| modules/web/html/lib/navigate.go | Uses navigation target for navigate |
| modules/web/html/lib/mouse_xy.go | Uses viewport target for mouse movement |
| modules/web/html/lib/lib.go | Registers FRAMES as fixed-arity A3 |
| modules/web/html/lib/input.go | Refactors INPUT to query+interaction targets |
| modules/web/html/lib/hover.go | Refactors HOVER to root interaction target |
| modules/web/html/lib/get_inner_text_all.go | Refactors INNER_TEXT_ALL to root content target |
| modules/web/html/lib/get_inner_text.go | Refactors INNER_TEXT to root content target |
| modules/web/html/lib/get_inner_html_all.go | Refactors INNER_HTML_ALL to root content target |
| modules/web/html/lib/get_inner_html.go | Refactors INNER_HTML to root content target |
| modules/web/html/lib/focus.go | Refactors FOCUS to root interaction target |
| modules/web/html/lib/find_frames.go | Refactors FRAMES signature and filtering |
| modules/web/html/lib/element.go | Refactors Element() query args to QueryTarget |
| modules/web/html/lib/document_test.go | Updates page load params tests incl cookies |
| modules/web/html/lib/cookie_set.go | Allows bulk cookie parsing + cookie target |
| modules/web/html/lib/cookie_helpers.go | Adds parseCookiesValue helper |
| modules/web/html/lib/cookie_get.go | Uses cookie reader target and arg validation |
| modules/web/html/lib/cookie_del.go | Supports bulk cookie delete parsing + target |
| modules/web/html/lib/click_all.go | Refactors CLICK_ALL to query+interaction targets |
| modules/web/html/lib/click.go | Refactors CLICK to query+interaction targets |
| modules/web/html/lib/clear.go | Refactors INPUT_CLEAR to interaction targets |
| modules/web/html/lib/blur.go | Refactors BLUR to interaction targets |
| modules/web/html/lib/attr_set.go | Changes ATTR_SET casting and value handling |
| modules/web/html/lib/attr_remove.go | Refactors ATTR_REMOVE to attribute targets |
| modules/web/html/lib/attr_query.go | Refactors ATTR_QUERY to query+attribute targets |
| modules/web/html/lib/attr_get.go | Refactors ATTR_GET to attribute target |
| modules/web/html/go.mod | Bumps Ferret dependency |
| modules/web/html/drivers/type.go | Updates runtime type registration style |
| modules/web/html/drivers/selector.go | Adds runtime type for QuerySelector |
| modules/web/html/drivers/memory/xpath.go | Fixes package name and XPath query error handling |
| modules/web/html/drivers/memory/page.go | Removes unsupported methods from memory page |
| modules/web/html/drivers/memory/options_test.go | Renames http -> memory driver tests |
| modules/web/html/drivers/memory/options.go | Renames http -> memory driver impl |
| modules/web/html/drivers/memory/helpers.go | Renames http -> memory driver impl |
| modules/web/html/drivers/memory/element_test.go | Updates tests to memory driver |
| modules/web/html/drivers/memory/driver_test.go | Updates constructor name to New |
| modules/web/html/drivers/memory/driver.go | Renames http driver to memory + constructor |
| modules/web/html/drivers/memory/document_test.go | Updates tests to memory driver |
| modules/web/html/drivers/memory/document.go | Removes panics/unsupported funcs in memory doc |
| modules/web/html/drivers/memory/doc.go | Adds package doc for memory driver |
| modules/web/html/drivers/memory/cssx_exp_test.go | Updates package name |
| modules/web/html/drivers/memory/cssx_exp.go | Updates package name |
| modules/web/html/drivers/memory/cssx_eval_test.go | Updates package name |
| modules/web/html/drivers/memory/cssx.go | Updates package name |
| modules/web/html/drivers/http/doc.go | Removes doc comment (http driver package doc) |
| modules/web/html/drivers/cookies_test.go | Updates expected JSON shape for cookies |
| modules/web/html/drivers/cookie_test.go | Expands cookie runtime getter tests |
| modules/web/html/drivers/cookie.go | Changes cookie JSON + getter field names |
| modules/web/html/drivers/common/getter.go | Capability-gates page getters (cookies/response) |
| modules/web/html/drivers/cdp/templates/select.go | Fixes select template variable reference |
| modules/web/html/drivers/cdp/session/transport.go | Adds CDP session transport implementation |
| modules/web/html/drivers/cdp/session/helpers.go | Adds session attach/new client helpers |
| modules/web/html/drivers/cdp/session/features.go | Adds attached-session feature enablement |
| modules/web/html/drivers/cdp/session/event.go | Adds session manager event types |
| modules/web/html/drivers/cdp/session/doc.go | Adds session package doc |
| modules/web/html/drivers/cdp/session/close_errors.go | Adds ignorable close error detection |
| modules/web/html/drivers/cdp/session/close_conn.go | Adds rpcc closeConn wrapper |
| modules/web/html/drivers/cdp/session/client.go | Adds session client container |
| modules/web/html/drivers/cdp/prepared_navigation_stream.go | Adds prepared navigation stream wrapper |
| modules/web/html/drivers/cdp/options_test.go | Adds loopback normalization unit test |
| modules/web/html/drivers/cdp/options.go | Normalizes loopback addresses; changes default |
| modules/web/html/drivers/cdp/network/stream_builders.go | Adds session-based stream builders |
| modules/web/html/drivers/cdp/network/session_navigation_stream.go | Adds merged navigation event stream |
| modules/web/html/drivers/cdp/network/manager_test.go | Updates cookie fetching to Network domain |
| modules/web/html/drivers/cdp/network/helpers_test.go | Adds cookie helper tests |
| modules/web/html/drivers/cdp/network/helpers.go | Makes cookie param fields conditional |
| modules/web/html/drivers/cdp/network/event.go | Stores source client in NavigationEvent |
| modules/web/html/drivers/cdp/input/manager.go | Fixes click error propagation |
| modules/web/html/drivers/cdp/helpers.go | Adds frame ID matcher + nav-event matching |
| modules/web/html/drivers/cdp/filtered_navigation_stream.go | Adds navigation-event filtering stream |
| modules/web/html/drivers/cdp/events/stream_test.go | Fixes cancel sequence to avoid premature stop |
| modules/web/html/drivers/cdp/events/stream.go | Avoids blocking sends on canceled context |
| modules/web/html/drivers/cdp/events/loop.go | Fixes lock usage for listener map mutations |
| modules/web/html/drivers/cdp/eval/runtime.go | Adds early builder error checks |
| modules/web/html/drivers/cdp/eval/function_test.go | Adds Err() behavior tests |
| modules/web/html/drivers/cdp/eval/function.go | Replaces panics with builder error tracking |
| modules/web/html/drivers/cdp/dom/subscription_target.go | Adds DOM event subscription for targets |
| modules/web/html/drivers/cdp/dom/subscription_helpers.go | Adds unified cleanup + payload decode |
| modules/web/html/drivers/cdp/dom/subscription_cleanup_test.go | Adds cleanup idempotence tests |
| modules/web/html/drivers/cdp/dom/subscription.go | Adds binding-based subscription stream impl |
| modules/web/html/drivers/cdp/dom/frame_id.go | Updates type registration style |
| modules/web/html/drivers/cdp/dom/frame_clients.go | Adds atomic frame->client map helper |
| modules/web/html/drivers/cdp/dom/element.go | Implements Subscribe; replaces Dispatch panic |
| modules/web/html/drivers/cdp/dom/document.go | Removes panics; implements Subscribe |
| modules/web/article/lib/html_input_test.go | Switches article tests to memory HTML driver |
| modules/web/article/lib/helpers.go | Allows WEB::ARTICLE input normalization from page |
| modules/web/article/go.mod | Bumps Ferret dependency and trims indirect deps |
| modules/toml/go.mod | Bumps Ferret dependency |
| modules/csv/go.mod | Bumps Ferret dependency |
| go.work | Updates workspace Go version and adds tests/runtime |
| Makefile | Splits test into unit + integration targets |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if err != nil { | ||
| ch <- runtime.NewErrorMessage(err) | ||
|
|
||
| return | ||
| } |
There was a problem hiding this comment.
These channel sends aren’t guarded by ctx.Done(). If the downstream consumer stops reading (common when contexts are canceled), this goroutine can block forever on ch <- ..., leaking goroutines/resources. Align with the safer pattern used in drivers/cdp/events/stream.go by selecting on ctx.Done() before sending error/value messages.
| if err != nil { | ||
| ch <- runtime.NewErrorMessage(err) | ||
|
|
||
| return | ||
| } | ||
|
|
||
| if val != nil && val != runtime.None { | ||
| ch <- runtime.NewValueMessage(val) | ||
| } |
There was a problem hiding this comment.
These channel sends aren’t guarded by ctx.Done(). If the downstream consumer stops reading (common when contexts are canceled), this goroutine can block forever on ch <- ..., leaking goroutines/resources. Align with the safer pattern used in drivers/cdp/events/stream.go by selecting on ctx.Done() before sending error/value messages.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 230 out of 501 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 230 out of 501 changed files in this pull request and generated 9 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if err != nil { | ||
| ch <- runtime.NewErrorMessage(err) | ||
|
|
||
| return | ||
| } |
There was a problem hiding this comment.
Sends to ch are unconditional; if the consumer stops reading (e.g., context canceled) these sends can block and leak the goroutine. Mirror the pattern used in drivers/cdp/events/stream.go by selecting on ctx.Done() before sending error/value messages.
| if err != nil { | ||
| ch <- runtime.NewErrorMessage(err) | ||
|
|
||
| return | ||
| } | ||
|
|
||
| if val != nil && val != runtime.None { | ||
| ch <- runtime.NewValueMessage(val) | ||
| } |
There was a problem hiding this comment.
Sends to ch are unconditional; if the consumer stops reading (e.g., context canceled) these sends can block and leak the goroutine. Mirror the pattern used in drivers/cdp/events/stream.go by selecting on ctx.Done() before sending error/value messages.
| if err != nil { | ||
| ch <- runtime.NewErrorMessage(err) | ||
| s.logger.Trace().Err(err).Msg("failed to read data from within document navigation event stream") | ||
| return | ||
| } |
There was a problem hiding this comment.
Similar to the DOM subscription stream, unconditional sends to ch can block indefinitely if the reader stops consuming events. Use a select with ctx.Done() when sending messages to prevent goroutine leaks and improve shutdown behavior.
| if err != nil { | ||
| ch <- runtime.NewErrorMessage(err) | ||
| s.logger.Trace().Err(err).Msg("failed to read data from frame navigation event stream") | ||
| return | ||
| } |
There was a problem hiding this comment.
Similar to the DOM subscription stream, unconditional sends to ch can block indefinitely if the reader stops consuming events. Use a select with ctx.Done() when sending messages to prevent goroutine leaks and improve shutdown behavior.
| return res, nil | ||
| } | ||
|
|
||
| if list, err := runtime.CastList(value); err == nil { |
There was a problem hiding this comment.
parseCookiesValue introduces multiple parsing branches (boxed cookies, list of cookies, single cookie) that are now relied on by CookieSet and CookieDel, but there are no direct tests covering these input permutations. Add focused unit tests to lock in expected behavior (especially around list inputs and boxed/unwrapped cookie collections).
| return cookies, nil | ||
| } | ||
|
|
||
| cookie, err := parseCookie(ctx, value) |
There was a problem hiding this comment.
parseCookiesValue introduces multiple parsing branches (boxed cookies, list of cookies, single cookie) that are now relied on by CookieSet and CookieDel, but there are no direct tests covering these input permutations. Add focused unit tests to lock in expected behavior (especially around list inputs and boxed/unwrapped cookie collections).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 232 out of 514 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| test: | ||
| @./scripts/modules.sh test $(filter-out $@,$(MAKECMDGOALS)) | ||
| test: test-unit test-integration |
There was a problem hiding this comment.
The PR description claims a new canonical AGENTS.md operating guide was added, but no such file appears in the provided diffs. Either include AGENTS.md in this PR or update the PR description to match the actual changes.
| <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous"> | ||
| <link rel="stylesheet" href="index.css"> | ||
| </head> | ||
| <body class="text-center"> | ||
| <div id="root"></div> | ||
| <script src="https://unpkg.com/react@16.8.6/umd/react.production.min.js"></script> | ||
| <script src="https://unpkg.com/react-dom@16.8.6/umd/react-dom.production.min.js"></script> | ||
| <script src="https://unpkg.com/history@4.9.0/umd/history.min.js"></script> | ||
| <script src="https://unpkg.com/react-router@4.3.1/umd/react-router.js"></script> | ||
| <script src="https://unpkg.com/react-router-dom@4.3.1/umd/react-router-dom.js"></script> | ||
| <script src="https://unpkg.com/react-bootstrap@2.10.10/dist/react-bootstrap.min.js" crossorigin> </script> |
There was a problem hiding this comment.
The dynamic integration-test fixture pulls React/Bootstrap/etc from public CDNs. This makes integration tests non-deterministic and can fail in CI environments with restricted outbound network access. Prefer serving these assets locally (vendored into tests/data/...) or bundling them into the test fixture to remove external runtime dependencies.
| <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous"> | |
| <link rel="stylesheet" href="index.css"> | |
| </head> | |
| <body class="text-center"> | |
| <div id="root"></div> | |
| <script src="https://unpkg.com/react@16.8.6/umd/react.production.min.js"></script> | |
| <script src="https://unpkg.com/react-dom@16.8.6/umd/react-dom.production.min.js"></script> | |
| <script src="https://unpkg.com/history@4.9.0/umd/history.min.js"></script> | |
| <script src="https://unpkg.com/react-router@4.3.1/umd/react-router.js"></script> | |
| <script src="https://unpkg.com/react-router-dom@4.3.1/umd/react-router-dom.js"></script> | |
| <script src="https://unpkg.com/react-bootstrap@2.10.10/dist/react-bootstrap.min.js" crossorigin> </script> | |
| <link rel="stylesheet" href="vendor/bootstrap.min.css"> | |
| <link rel="stylesheet" href="index.css"> | |
| </head> | |
| <body class="text-center"> | |
| <div id="root"></div> | |
| <script src="vendor/react.production.min.js"></script> | |
| <script src="vendor/react-dom.production.min.js"></script> | |
| <script src="vendor/history.min.js"></script> | |
| <script src="vendor/react-router.js"></script> | |
| <script src="vendor/react-router-dom.js"></script> | |
| <script src="vendor/react-bootstrap.min.js"></script> |
| <script src="https://unpkg.com/react@16.8.6/umd/react.production.min.js"></script> | ||
| <script src="https://unpkg.com/react-dom@16.8.6/umd/react-dom.production.min.js"></script> | ||
| <script src="https://unpkg.com/history@4.9.0/umd/history.min.js"></script> | ||
| <script src="https://unpkg.com/react-router@4.3.1/umd/react-router.js"></script> | ||
| <script src="https://unpkg.com/react-router-dom@4.3.1/umd/react-router-dom.js"></script> | ||
| <script src="https://unpkg.com/react-bootstrap@2.10.10/dist/react-bootstrap.min.js" crossorigin> </script> |
There was a problem hiding this comment.
The dynamic integration-test fixture pulls React/Bootstrap/etc from public CDNs. This makes integration tests non-deterministic and can fail in CI environments with restricted outbound network access. Prefer serving these assets locally (vendored into tests/data/...) or bundling them into the test fixture to remove external runtime dependencies.
| 'https://www.montferret.dev/try/', | ||
| 'https://www.montferret.dev/docs/', | ||
| 'https://www.montferret.dev/blog/', | ||
| 'https://www.montferret.dev/cookbook/' |
There was a problem hiding this comment.
This test page triggers real outbound HTTP requests to www.montferret.dev. That can introduce flakiness (network/DNS/rate limits) and can break in CI environments without Internet access. Consider pointing these requests at locally served endpoints (e.g., under tests/data/pages/...) or a controllable test server so integration runs are reliable.
| ) (runtime.Iterator, error) { | ||
| func NewIterator(node drivers.HTMLElement) (runtime.Iterator, error) { | ||
| if node == nil { | ||
| return nil, runtime.Error(runtime.ErrMissedArgument, "result") |
There was a problem hiding this comment.
The error reports a missing argument named "result", but the nil check is for the node parameter. Updating the message to "node" (or similar) would make failures easier to diagnose.
| return nil, runtime.Error(runtime.ErrMissedArgument, "result") | |
| return nil, runtime.Error(runtime.ErrMissedArgument, "node") |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 232 out of 514 changed files in this pull request and generated 8 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| let search = ''; | ||
|
|
||
| Object.keys(params).forEach((key) => { | ||
| if (key !== 'redirect') { | ||
| search += `${key}=${params[key]}`; | ||
| } | ||
| }); | ||
|
|
There was a problem hiding this comment.
This builds a query string without & separators and without URL-encoding, so multiple params will produce an invalid search (e.g. a=1b=2) and special characters can break redirects. Prefer building search via URLSearchParams (or at least join with & and encodeURIComponent).
| let search = ''; | |
| Object.keys(params).forEach((key) => { | |
| if (key !== 'redirect') { | |
| search += `${key}=${params[key]}`; | |
| } | |
| }); | |
| const searchParams = new URLSearchParams(); | |
| Object.keys(params).forEach((key) => { | |
| if (key !== 'redirect') { | |
| searchParams.append(key, params[key]); | |
| } | |
| }); | |
| const search = searchParams.toString(); |
| <script src="https://unpkg.com/react@16.8.6/umd/react.production.min.js"></script> | ||
| <script src="https://unpkg.com/react-dom@16.8.6/umd/react-dom.production.min.js"></script> |
There was a problem hiding this comment.
react-bootstrap@2.x expects newer React versions than 16.8.6 (React 16.14+ / 17 / 18), so this fixture may fail at runtime in integration tests. Either upgrade React/ReactDOM to a compatible version, or pin react-bootstrap to a version that supports React 16.8.
| // The iterator starts from the current page i.e. it does not change the page on 1st iteration. | ||
| // That allows you to keep scraping logic inside FOR loop. | ||
| // @param {HTMLPage | HTMLDocument | HTMLElement} node - Target html node. | ||
| // @param {HTMLPage} page - Target html page. |
There was a problem hiding this comment.
The docstring now states Pagination only accepts HTMLPage, which is a breaking API change compared to the previous {HTMLPage | HTMLDocument | HTMLElement} contract. If callers still need to paginate from a document/element context, consider preserving the broader input type and deriving the page/frame internally; otherwise, this should be explicitly called out as a breaking change.
| // @param {HTMLPage} page - Target html page. | |
| // @param {HTMLPage | HTMLDocument | HTMLElement} page - Target html page or a document/element within that page. |
| h.Write([]byte("CDP")) | ||
| h.Write([]byte(p.Type().String())) | ||
| h.Write([]byte(":")) | ||
| h.Write([]byte(p.getCurrentDocument().GetURL())) |
There was a problem hiding this comment.
GetURL() returns runtime.String (a defined string type in most Ferret runtimes). Converting it directly to []byte may not compile; use an explicit conversion to string first (e.g. []byte(string(...))) to ensure this builds across Go.
| h.Write([]byte(p.getCurrentDocument().GetURL())) | |
| h.Write([]byte(string(p.getCurrentDocument().GetURL()))) |
|
|
||
| services: | ||
| chromium: | ||
| image: ghcr.io/montferret/chromium:latest |
There was a problem hiding this comment.
The workflow uses an unpinned container tag (:latest) and downloads an install script from a moving branch (master) without a checksum/signature check, which increases supply-chain risk and can cause non-reproducible CI. Pin the Chromium image to a version/digest and pin the install script to a commit SHA (and ideally verify a checksum).
| curl --fail --silent --show-error --location \ | ||
| https://raw.githubusercontent.com/MontFerret/lab/master/install.sh \ | ||
| -o install.sh |
There was a problem hiding this comment.
The workflow uses an unpinned container tag (:latest) and downloads an install script from a moving branch (master) without a checksum/signature check, which increases supply-chain risk and can cause non-reproducible CI. Pin the Chromium image to a version/digest and pin the install script to a commit SHA (and ideally verify a checksum).
| if (e.key == 'Unidentified') { | ||
| return; | ||
| } | ||
|
|
||
| this.setState({ | ||
| key: this.state.key ? this.state.key + ' + ' + e.key : e.key | ||
| }) |
There was a problem hiding this comment.
setState derives the next value from this.state, which can drop updates under rapid keydown events due to React state batching. Use functional setState(prev => ...) to guarantee correct accumulation, and prefer strict equality (===) for the key check.
| if (e.key == 'Unidentified') { | |
| return; | |
| } | |
| this.setState({ | |
| key: this.state.key ? this.state.key + ' + ' + e.key : e.key | |
| }) | |
| if (e.key === 'Unidentified') { | |
| return; | |
| } | |
| this.setState((prevState) => ({ | |
| key: prevState.key ? prevState.key + ' + ' + e.key : e.key | |
| })) |
| func parseCookiesValue(ctx context.Context, value runtime.Value) (*drivers.HTTPCookies, error) { | ||
| if value == nil || value == runtime.None { | ||
| return nil, fmt.Errorf("cookies are required") | ||
| } |
There was a problem hiding this comment.
This introduces plain fmt.Errorf errors where surrounding code often uses runtime.Error(...) / runtime.ErrMissedArgument for consistent error typing and downstream handling. Consider returning a runtime-typed error (and/or wrapping with runtime.ArgError at call sites) so callers can reliably identify missing-argument vs invalid-type conditions.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 232 out of 514 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| func toRootElement(value runtime.Value) (drivers.HTMLElement, error) { | ||
| switch v := value.(type) { | ||
| case drivers.HTMLPage: | ||
| return drivers.ToElement(v.GetMainFrame().GetElement()) | ||
| case drivers.HTMLDocument: | ||
| return drivers.ToElement(v.GetElement()) | ||
| case drivers.HTMLElement: | ||
| return drivers.ToElement(v) | ||
| default: | ||
| return nil, runtime.TypeErrorOf(value, drivers.HTMLPageType, drivers.HTMLDocumentType, drivers.HTMLElementType) | ||
| } | ||
| } | ||
|
|
||
| func toRootContentTarget(value runtime.Value) (drivers.ContentTarget, error) { | ||
| return toRootElementCapability[drivers.ContentTarget](value, "content") | ||
| } | ||
|
|
There was a problem hiding this comment.
These new root-target resolution helpers are now central to many stdlib HTML operations, but there are no tests included here to validate behavior across (page/document/element) inputs and unsupported capability cases. Adding focused unit tests for: (1) HTMLPage → main frame element; (2) HTMLDocument → element; (3) HTMLElement pass-through; and (4) capability-not-supported errors would help prevent regressions as driver types evolve.
| func toRootInteractionTarget(value runtime.Value) (drivers.InteractionTarget, error) { | ||
| return toRootElementCapability[drivers.InteractionTarget](value, "interaction") | ||
| } | ||
|
|
||
| func toRootWaitTarget(value runtime.Value) (drivers.WaitTarget, error) { | ||
| return toRootElementCapability[drivers.WaitTarget](value, "wait") | ||
| } | ||
|
|
||
| func toRootElementCapability[T any](value runtime.Value, capability string) (T, error) { | ||
| var zero T | ||
|
|
||
| el, err := toRootElement(value) | ||
| if err != nil { | ||
| return zero, err | ||
| } | ||
|
|
||
| target, ok := any(el).(T) | ||
| if !ok { | ||
| return zero, runtime.Errorf(runtime.ErrNotSupported, "root element %s capability", capability) | ||
| } | ||
|
|
||
| return target, nil | ||
| } |
There was a problem hiding this comment.
These new root-target resolution helpers are now central to many stdlib HTML operations, but there are no tests included here to validate behavior across (page/document/element) inputs and unsupported capability cases. Adding focused unit tests for: (1) HTMLPage → main frame element; (2) HTMLDocument → element; (3) HTMLElement pass-through; and (4) capability-not-supported errors would help prevent regressions as driver types evolve.
This pull request introduces several important improvements to the repository’s structure, documentation, and development workflow. The main highlights include the addition of a comprehensive agent operating guide, enhancements to the testing workflow, updates to Go versioning and workspace configuration, and dependency updates.
Key changes:
Documentation and Guidance
AGENTS.mdfile that serves as the canonical operating guide for coding agents in this repository. It details ownership boundaries, module structure, development practices, code structure rules, and workflow expectations. This will help ensure consistency and clarity for all contributors.Build and Test Workflow
Makefileto split thetesttarget intotest-unitandtest-integration, allowing for more granular testing. Thetesttarget now runs both unit and integration tests by default.Go Workspace and Versioning
go.workto require Go version1.26.1(was1.25.6), ensuring the workspace uses the latest compatible Go version.tests/runtimeto the workspace modules ingo.work, making it part of the local development environment.Dependency Management
go.work.sumto include new and updated dependency checksums, reflecting changes in direct and indirect dependencies required by the workspace and its modules. [1] [2] [3]These changes improve repository maintainability, clarify contributor expectations, and enhance the development and testing experience.