What happened?
What happened?
The URL extraction endpoint resolves relative links against the fetched document URL but does not honor the document's HTML element.
For a page fetched from:
https://example.com/de/page.html
with:
Article
OpenSERP currently resolves the link as:
https://example.com/de/de/article.html
The expected URL is:
https://example.com/de/article.html
The same issue affects:
entries in ExtractResult.links
canonical URLs
relative links in generated Markdown
relative image URLs in generated Markdown
The relative URLs appear to be resolved directly against the fetched document URL instead of the effective base URL defined by the first element.
How do we reproduce it?
How do we reproduce it?
Serve the following minimal HTML document at:
https://example.com/de/page.html
<!doctype html>
Article
Then call the extraction endpoint:
curl -sS
-H 'Accept: application/json'
-H 'Content-Type: application/json'
--data-binary @-
'http://127.0.0.1:7000/extract' <<'JSON'
{
"url": "https://example.com/de/page.html",
"mode": "auto",
"clean": true,
"use_llms_txt": false,
"min_runes": 0
}
JSON
The effective resolution base should be:
document URL: https://example.com/de/page.html
base href: /
effective: https://example.com/
Expected resolved URLs:
https://example.com/de/article.html
https://example.com/images/example.png
https://example.com/de/canonical.html
Actual link resolution before the fix:
https://example.com/de/de/article.html
The issue appears to originate in parseMetadata(), where links are resolved using:
resolved := resolveURL(baseURL, href)
The supplied baseURL is the fetched document URL and does not account for the document's element.
Markdown conversion similarly receives the fetched document URL through:
converter.WithDomain(baseURL)
I tested a local fix that determines the effective document base after parsing the HTML and uses it for metadata links, canonical URLs, Markdown links, and Markdown images.
The actual fetched document URL remains in use as Trafilatura's OriginalURL.
Regression tests covering relative links, canonical URLs, images, and duplicated path segments such as /de/de/ pass with the change.
Retrieval mode
POST /extract
mode: auto
mode_used: fast
clean: true
proxy: none
Retrieval mode
not sure
Engine
not engine-specific
OpenSERP version
openserp version 0.8.12, 29c7b0f
Environment
Environment
OS: Debian 13
Go version: go1.24.6
Deployment: Native systemd service
Service endpoint: http://127.0.0.1:7000
Extraction mode requested: auto
Extraction mode used: fast
Proxy: none
Browser profile observed: chrome-linux-amd
Logs or response body
Logs or response body
Relevant response before the fix:
{
"url": "https://example.com/de/page.html",
"links": [
{
"text": "Article",
"url": "https://example.com/de/de/article.html"
}
],
"canonical": "https://example.com/de/de/canonical.html",
"meta": {
"mode_used": "fast"
}
}
Expected response:
{
"url": "https://example.com/de/page.html",
"links": [
{
"text": "Article",
"url": "https://example.com/de/article.html"
}
],
"canonical": "https://example.com/de/canonical.html",
"meta": {
"mode_used": "fast"
}
}
A real-world page containing:
<base href="/">
produced duplicated language paths before the fix:
https://example.com/de/de/article.html
After calculating and using the effective HTML base URL, the same reference resolves correctly:
https://example.com/de/article.html
All tests in the extract package passed after applying the local fix.
What happened?
What happened?
The URL extraction endpoint resolves relative links against the fetched document URL but does not honor the document's HTML element.
For a page fetched from:
https://example.com/de/page.html
with:
ArticleOpenSERP currently resolves the link as:
https://example.com/de/de/article.html
The expected URL is:
https://example.com/de/article.html
The same issue affects:
entries in ExtractResult.links
canonical URLs
relative links in generated Markdown
relative image URLs in generated Markdown
The relative URLs appear to be resolved directly against the fetched document URL instead of the effective base URL defined by the first element.
How do we reproduce it?
How do we reproduce it?
Serve the following minimal HTML document at:
https://example.com/de/page.html
Article<!doctype html>
Then call the extraction endpoint:
curl -sS
-H 'Accept: application/json'
-H 'Content-Type: application/json'
--data-binary @-
'http://127.0.0.1:7000/extract' <<'JSON'
{
"url": "https://example.com/de/page.html",
"mode": "auto",
"clean": true,
"use_llms_txt": false,
"min_runes": 0
}
JSON
The effective resolution base should be:
document URL: https://example.com/de/page.html
base href: /
effective: https://example.com/
Expected resolved URLs:
https://example.com/de/article.html
https://example.com/images/example.png
https://example.com/de/canonical.html
Actual link resolution before the fix:
https://example.com/de/de/article.html
The issue appears to originate in parseMetadata(), where links are resolved using:
resolved := resolveURL(baseURL, href)
The supplied baseURL is the fetched document URL and does not account for the document's element.
Markdown conversion similarly receives the fetched document URL through:
converter.WithDomain(baseURL)
I tested a local fix that determines the effective document base after parsing the HTML and uses it for metadata links, canonical URLs, Markdown links, and Markdown images.
The actual fetched document URL remains in use as Trafilatura's OriginalURL.
Regression tests covering relative links, canonical URLs, images, and duplicated path segments such as /de/de/ pass with the change.
Retrieval mode
POST /extract
mode: auto
mode_used: fast
clean: true
proxy: none
Retrieval mode
not sure
Engine
not engine-specific
OpenSERP version
openserp version 0.8.12, 29c7b0f
Environment
Environment
OS: Debian 13
Go version: go1.24.6
Deployment: Native systemd service
Service endpoint: http://127.0.0.1:7000
Extraction mode requested: auto
Extraction mode used: fast
Proxy: none
Browser profile observed: chrome-linux-amd
Logs or response body
Logs or response body Relevant response before the fix: { "url": "https://example.com/de/page.html", "links": [ { "text": "Article", "url": "https://example.com/de/de/article.html" } ], "canonical": "https://example.com/de/de/canonical.html", "meta": { "mode_used": "fast" } } Expected response: { "url": "https://example.com/de/page.html", "links": [ { "text": "Article", "url": "https://example.com/de/article.html" } ], "canonical": "https://example.com/de/canonical.html", "meta": { "mode_used": "fast" } } A real-world page containing: <base href="/"> produced duplicated language paths before the fix: https://example.com/de/de/article.html After calculating and using the effective HTML base URL, the same reference resolves correctly: https://example.com/de/article.html All tests in the extract package passed after applying the local fix.