chore: update docs by medcl · Pull Request #47 · infinilabs/loadgen

medcl · 2026-03-05T10:59:43Z

What does this PR do

Rationale for this change

Standards checklist

The PR title is descriptive
The commit messages are semantic
Necessary tests are added
Updated the release notes
Necessary documents have been added if this is a new feature
Performance tests checked, no obvious performance degradation

Copilot

Pull request overview

This PR updates the Hugo docs build configuration to generate additional non-HTML outputs (Markdown + llms.txt), tightens Hugo security settings for env var access, and adds a post-build step to maintain a root robots.txt with sitemap/LLMs links.

Changes:

Expand docs/config.yaml to enable Hugo env var access and add custom output formats (markdown, llms_txt) with corresponding outputs.
Update docs-build to use an absolute baseURL on docs.infinilabs.com.
Add docs-update-robots Makefile target to append sitemap and llms.txt references into $(OUTPUT)/robots.txt.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
docs/config.yaml	Adds Hugo security allowlist and new media/output formats for Markdown + `llms.txt`.
docs/Makefile	Changes Hugo `baseURL` and adds a robots.txt update step after building docs.

Comments suppressed due to low confidence (1)

docs/Makefile:60

docs-update-robots writes Allow: /llms.txt, but the generated llms.txt URL you add later is versioned (.../$(PRODUCT)/$(VERSION)/llms.txt) and Hugo will emit it under that path. Since there are no Disallow rules, the Allow line is also redundant; consider removing it or updating it to the versioned path to avoid misleading robots.txt consumers.

		echo "User-agent: *" > $(OUTPUT)/robots.txt; \
		echo "Allow: /" >> $(OUTPUT)/robots.txt; \
		echo "Allow: /llms.txt" >> $(OUTPUT)/robots.txt; \
		echo "" >> $(OUTPUT)/robots.txt; \

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-05T11:04:11Z

+      - 'CURRENT_VERSION'
+      - 'VERSIONS'


In the Hugo security.funcs.getenv allowlist, CURRENT_VERSION and VERSIONS are unanchored regexes. Since Hugo treats these entries as regex patterns, this can unintentionally match other env var names containing those substrings. Consider anchoring them (e.g., ^CURRENT_VERSION$, ^VERSIONS$) to keep the allowlist strict.

Suggested change

- 'CURRENT_VERSION'

- 'VERSIONS'

- '^CURRENT_VERSION$'

- '^VERSIONS$'

Copilot · 2026-03-05T11:04:12Z

+	@mkdir -p $(OUTPUT)
+	@touch $(OUTPUT)/robots.txt
+	@if [ ! -s $(OUTPUT)/robots.txt ]; then \
+		echo "Adding default robots.txt content..."; \
+		echo "User-agent: *" > $(OUTPUT)/robots.txt; \
+		echo "Allow: /" >> $(OUTPUT)/robots.txt; \
+		echo "Allow: /llms.txt" >> $(OUTPUT)/robots.txt; \
+		echo "" >> $(OUTPUT)/robots.txt; \
+	fi
+	@SITEMAP_LINE="Sitemap: https://docs.infinilabs.com/$(PRODUCT)/$(VERSION)/sitemap.xml"; \
+	if ! grep -Fq "$$SITEMAP_LINE" $(OUTPUT)/robots.txt; then \
+		echo "$$SITEMAP_LINE" >> $(OUTPUT)/robots.txt; \
+		echo "Added sitemap entry to robots.txt"; \
+	else \
+		echo "Sitemap entry already exists in robots.txt"; \
+	fi
+	@LLMS_LINE="X-llms-txt: https://docs.infinilabs.com/$(PRODUCT)/$(VERSION)/llms.txt"; \
+	if ! grep -Fq "$$LLMS_LINE" $(OUTPUT)/robots.txt; then \
+		echo "$$LLMS_LINE" >> $(OUTPUT)/robots.txt; \
+		echo "Added llms.txt entry to robots.txt"; \
+	else \
+		echo "LLMs.txt entry already exists in robots.txt"; \
+	fi
+
 docs-place-redirect:
 	echo "<!DOCTYPE html> <html> <head> <meta http-equiv=refresh content=0;url=main /> </head> <body> <p><a href=main />REDIRECT TO THE LATEST_VERSION</a>.</p> </body> </html>" > $(OUTPUT)/$(PRODUCT)/index.html


docs-update-robots uses unquoted $(OUTPUT)/robots.txt in touch, redirections, and grep. Since OUTPUT is user-configurable, paths containing spaces or shell metacharacters can break this target. Quote the path consistently (and consider using set -e / && chaining) to make the rule robust.

Suggested change

@mkdir -p $(OUTPUT)

@touch $(OUTPUT)/robots.txt

@if [ ! -s $(OUTPUT)/robots.txt ]; then \

echo "Adding default robots.txt content..."; \

echo "User-agent: *" > $(OUTPUT)/robots.txt; \

echo "Allow: /" >> $(OUTPUT)/robots.txt; \

echo "Allow: /llms.txt" >> $(OUTPUT)/robots.txt; \

echo "" >> $(OUTPUT)/robots.txt; \

fi

@SITEMAP_LINE="Sitemap: https://docs.infinilabs.com/$(PRODUCT)/$(VERSION)/sitemap.xml"; \

if ! grep -Fq "$$SITEMAP_LINE" $(OUTPUT)/robots.txt; then \

echo "$$SITEMAP_LINE" >> $(OUTPUT)/robots.txt; \

echo "Added sitemap entry to robots.txt"; \

else \

echo "Sitemap entry already exists in robots.txt"; \

fi

@LLMS_LINE="X-llms-txt: https://docs.infinilabs.com/$(PRODUCT)/$(VERSION)/llms.txt"; \

if ! grep -Fq "$$LLMS_LINE" $(OUTPUT)/robots.txt; then \

echo "$$LLMS_LINE" >> $(OUTPUT)/robots.txt; \

echo "Added llms.txt entry to robots.txt"; \

else \

echo "LLMs.txt entry already exists in robots.txt"; \

fi

docs-place-redirect:

echo "<!DOCTYPE html> <html> <head> <meta http-equiv=refresh content=0;url=main /> </head> <body> <p><a href=main />REDIRECT TO THE LATEST_VERSION</a>.</p> </body> </html>" > $(OUTPUT)/$(PRODUCT)/index.html

@mkdir -p "$(OUTPUT)"

@touch "$(OUTPUT)/robots.txt"

@if [ ! -s "$(OUTPUT)/robots.txt" ]; then \

echo "Adding default robots.txt content..."; \

echo "User-agent: *" > "$(OUTPUT)/robots.txt"; \

echo "Allow: /" >> "$(OUTPUT)/robots.txt"; \

echo "Allow: /llms.txt" >> "$(OUTPUT)/robots.txt"; \

echo "" >> "$(OUTPUT)/robots.txt"; \

fi

@SITEMAP_LINE="Sitemap: https://docs.infinilabs.com/$(PRODUCT)/$(VERSION)/sitemap.xml"; \

if ! grep -Fq "$$SITEMAP_LINE" "$(OUTPUT)/robots.txt"; then \

echo "$$SITEMAP_LINE" >> "$(OUTPUT)/robots.txt"; \

echo "Added sitemap entry to robots.txt"; \

else \

echo "Sitemap entry already exists in robots.txt"; \

fi

@LLMS_LINE="X-llms-txt: https://docs.infinilabs.com/$(PRODUCT)/$(VERSION)/llms.txt"; \

if ! grep -Fq "$$LLMS_LINE" "$(OUTPUT)/robots.txt"; then \

echo "$$LLMS_LINE" >> "$(OUTPUT)/robots.txt"; \

echo "Added llms.txt entry to robots.txt"; \

else \

echo "LLMs.txt entry already exists in robots.txt"; \

fi

docs-place-redirect:

echo "<!DOCTYPE html> <html> <head> <meta http-equiv=refresh content=0;url=main /> </head> <body> <p><a href=main />REDIRECT TO THE LATEST_VERSION</a>.</p> </body> </html>" > "$(OUTPUT)/$(PRODUCT)/index.html"

Copilot · 2026-03-05T11:04:12Z

 	hugo --minify --theme book --destination="$(OUTPUT)/$(PRODUCT)/$(VERSION)" \
-        --baseURL="/$(PRODUCT)/$(VERSION)"
+        --baseURL="https://docs.infinilabs.com/$(PRODUCT)/$(VERSION)"


docs-build hardcodes --baseURL to https://docs.infinilabs.com/..., which makes the build output unsuitable for local file viewing or deploying to a staging/custom domain. Consider making the docs host/base URL configurable via a Make variable (with a sensible default) so CI/deploy can set it without changing the Makefile.

chore: update docs

9910d1a

Copilot AI review requested due to automatic review settings March 5, 2026 10:59

Copilot started reviewing on behalf of medcl March 5, 2026 11:00 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

SteveLauC approved these changes Mar 5, 2026

View reviewed changes

luohoufu approved these changes Mar 5, 2026

View reviewed changes

luohoufu merged commit 163d5c1 into main Mar 5, 2026
9 of 10 checks passed

luohoufu deleted the update_docs branch March 5, 2026 14:29

Navigation Menu

Uh oh!

chore: update docs - #47

chore: update docs#47
luohoufu merged 1 commit into
mainfrom
update_docs

medcl commented Mar 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

medcl commented Mar 5, 2026

What does this PR do

Rationale for this change

Standards checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants