Skip to content

fix: don't remap cross-repo GitHub blob URLs to raw - #103

Closed
zeitlinger wants to merge 1 commit into
mainfrom
fix/cross-repo-blob-remap
Closed

zeitlinger wants to merge 1 commit into
mainfrom
fix/cross-repo-blob-remap

Conversation

@zeitlinger

Copy link
Copy Markdown
Member

Summary

  • Remove global remap of all GitHub /blob/ URLs to raw.githubusercontent.com
  • Keep same-repo remaps (build_remap_args) and issue/PR comment anchor stripping

Problem

The global remap breaks when a /blob/ URL points to a directory (e.g. github.com/org/repo/blob/main/some-dir). GitHub handles these via redirect to /tree/, but raw.githubusercontent.com returns 404 for directories.

Found in open-telemetry/opentelemetry-java-contrib#2654 where https://github.com/open-telemetry/opentelemetry-java-examples/blob/main/javaagent (a directory) was being remapped and failing.

Test plan

  • Verify link checking passes on repos with cross-repo /blob/ directory links
  • Verify same-repo remaps still work (rate limiting avoidance)

Copilot AI review requested due to automatic review settings March 11, 2026 12:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts the link-checking remap logic used by lint:links to avoid incorrectly rewriting cross-repo GitHub /blob/ URLs to raw.githubusercontent.com, which can cause false 404s when the /blob/ URL points to a directory (GitHub redirects those to /tree/, but raw does not).

Changes:

  • Removes the global remap of all GitHub /blob/ URLs to raw.githubusercontent.com.
  • Keeps same-repo remaps via build_remap_args() and retains issue/PR comment anchor stripping.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tasks/lint/links.sh
echo "--remap"
# shellcheck disable=SC2016 # single quotes are intentional
# shellcheck disable=SC2016 # single quotes are intentional: these are regex capture groups, not shell vars
echo '^https://github.com/([^/]+/[^/]+)/(issues|pull)/([0-9]+)#issuecomment-.*$ https://github.com/$1/$2/$3'

Copilot AI Mar 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_global_github_args() no longer strips GitHub /blob/ #L… and #:~:text=… fragments for cross-repo URLs. Since lychee runs with include_fragments=true (see .github/config/lychee.toml) and the repo contains external #:~:text links (e.g. tests/test-links.md), this will reintroduce deterministic false failures because Scroll-to-Text fragments are not present in static HTML. Consider keeping fragment-stripping remaps for /blob/ URLs (rewrite to the same https://github.com/<owner>/<repo>/blob/<ref>/<path> without the fragment) while avoiding any remap to raw.githubusercontent.com, which preserves the directory /blob/ → /tree/ redirect behavior that motivated this change.

Suggested change
echo '^https://github.com/([^/]+/[^/]+)/(issues|pull)/([0-9]+)#issuecomment-.*$ https://github.com/$1/$2/$3'
echo '^https://github.com/([^/]+/[^/]+)/(issues|pull)/([0-9]+)#issuecomment-.*$ https://github.com/$1/$2/$3'
# Cross-repo /blob/ URLs with JS-rendered fragments:
# - Line anchors (#L123, #L10-L20, etc.)
# - Scroll-to-Text fragments (#:~:text=...)
# Lychee can't verify these when include_fragments=true, and the
# fragments don't exist in static HTML. Strip the fragment while
# keeping the URL on github.com (no remap to raw.githubusercontent.com).
echo "--remap"
# Strip line-number anchors on blob URLs
# shellcheck disable=SC2016 # single quotes are intentional: these are regex capture groups, not shell vars
echo '^https://github.com/([^/]+/[^/]+)/blob/([^#]+)#L[0-9]+.*$ https://github.com/$1/blob/$2'
echo "--remap"
# Strip Scroll-to-Text fragments on blob URLs
# shellcheck disable=SC2016 # single quotes are intentional: these are regex capture groups, not shell vars
echo '^https://github.com/([^/]+/[^/]+)/blob/([^#]+)#:~:text=.*$ https://github.com/$1/blob/$2'

Copilot uses AI. Check for mistakes.
@zeitlinger
zeitlinger force-pushed the fix/cross-repo-blob-remap branch 2 times, most recently from bb2ae48 to 314edf2 Compare March 11, 2026 15:18
@zeitlinger
zeitlinger marked this pull request as draft March 11, 2026 15:27
@zeitlinger
zeitlinger force-pushed the fix/cross-repo-blob-remap branch from 314edf2 to f7b03b5 Compare March 12, 2026 06:28
GitHub blob URLs are remapped to raw.githubusercontent.com so lychee
can verify #section fragments (GitHub serves blob pages as React apps,
so fragments aren't in the HTML — lycheeverse/lychee#1729). This also
avoids GitHub's web rate limits (429s).

The remap breaks when the URL points to a directory (raw returns 404).
Fix by only remapping paths where the last component contains a dot
(file extension). Directories like /blob/main/some-dir are left for
GitHub to handle normally.
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
@zeitlinger
zeitlinger force-pushed the fix/cross-repo-blob-remap branch from f7b03b5 to ce0d32e Compare March 12, 2026 06:43
@zeitlinger

Copy link
Copy Markdown
Member Author

After investigating all 170 cross-repo /blob/ URLs across the 11 consuming repos, only 4 point to directories. GitHub redirects /blob//tree/ for directories (301), so they work — but raw.githubusercontent.com returns 400/404.

Rather than adding the extension heuristic here, the simpler fix is to correct those 4 URLs upstream to use /tree/. The raw remap being stricter is a feature — it surfaces genuinely wrong URL types.

Fixed the 3 URLs causing CI failures in java-contrib#2654: c309e075. Keeping this PR open until we confirm that resolves the issue, then closing.

@zeitlinger

Copy link
Copy Markdown
Member Author

Closing — the fix is to correct the 4 upstream directory URLs to use /tree/ instead of /blob/, not to add a heuristic in flint. See #103 (comment) for details.

@zeitlinger zeitlinger closed this Mar 12, 2026
@zeitlinger
zeitlinger deleted the fix/cross-repo-blob-remap branch March 19, 2026 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants