Skip to content

修复多源热榜抓取稳定性并规范忽略本地补丁目录#138

Open
FlippySun wants to merge 1 commit into
imsyy:masterfrom
FlippySun:fix/source-route-fallbacks
Open

修复多源热榜抓取稳定性并规范忽略本地补丁目录#138
FlippySun wants to merge 1 commit into
imsyy:masterfrom
FlippySun:fix/source-route-fallbacks

Conversation

@FlippySun
Copy link
Copy Markdown

变更说明

本次修改将此前仅覆盖 dist/routes/*.js 的本地补丁,正式迁回源码层的 TypeScript 路由实现,避免重新构建后补丁失效,并提升多个热榜源在接口变动或风控场景下的可用性。

主要改动

  • 修复 baidu 路由:增强 s-data 解析兼容性,并补充更稳的链接兜底逻辑
  • 修复 huxiu 路由:由脆弱 API 切换为页面 HTML 解析
  • 修复 kuaishou 路由:保留 APOLLO 主链路,并增加 DOM fallback 与 TopHub fallback
  • 修复 weibo 路由:切换为移动端接口,并在失败时回退到 TopHub
  • 修复 coolapk 路由:保留官方 API,并在失败时回退到 TopHub
  • 补充 .gitignore:忽略 .planning/.pnpm-store/补丁/ 等本地工作目录产物

验证结果

已在本地完成以下验证:

  • pnpm lint
  • pnpm build
  • smoke test:baidu / huxiu / kuaishou / weibo / coolapk

备注

  • 本地验证环境未启用 Redis,日志中会出现 Redis 连接失败提示,但不影响 NodeCache 兜底与接口返回
  • weibocoolapk 在当前验证环境中主要通过 fallback 链路返回数据,说明降级逻辑已生效

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request significantly hardens the data fetching logic for several routes, including Baidu, Coolapk, Huxiu, Kuaishou, and Weibo, by implementing multi-layered fallback mechanisms and switching to more resilient endpoints or HTML parsing. Key improvements include the integration of TopHub as a secondary data source and the addition of detailed change records for better maintainability. Review feedback highlights opportunities to improve ID stability in the Coolapk fallback, ensure consistent error messaging in English, validate mandatory fields during Kuaishou parsing, and refine exception handling to ensure all fallback layers are correctly executed.

Comment thread src/routes/coolapk.ts
return;
}
data.push({
id: href.split("/").pop() || title,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

在从 TopHub 降级链路提取 ID 时,建议剥离 URL 中的查询参数。这样可以确保 ID 的纯净度,避免类似 ?from=... 的参数被包含在 ID 中,从而提高 ID 的稳定性。

Suggested change
id: href.split("/").pop() || title,
id: href.split("/").pop()?.split("?")[0] || title,

Comment thread src/routes/kuaishou.ts
@@ -74,15 +83,12 @@ const getList = async (noCache: boolean) => {
: "未知错误";
throw new Error(`快手数据解析失败: ${msg}`);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

为了保持代码中错误提示语言的一致性(该文件其他地方如第 63、71 行已改为英文),建议将此处的错误信息也统一为英文。

Suggested change
throw new Error(`快手数据解析失败: ${msg}`);
throw new Error(`Kuaishou data parsing failed: ${msg}`);

Comment thread src/routes/kuaishou.ts
allItems.forEach((item: { id: string }) => {
// 基础数据
const hotItem = jsonObject[item.id];
if (!hotItem) return;
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

在解析 APOLLO 数据时,建议增加对 hotItem.name 的存在性校验。如果该字段缺失,生成的 ListItem 将缺少必要的 title 属性(根据 types.d.ts 定义,title 是必填项),可能导致后续处理逻辑异常。

Suggested change
if (!hotItem) return;
if (!hotItem || !hotItem.name) return;

Comment thread src/routes/kuaishou.ts
Comment on lines +196 to +197
const apolloData = parseApolloList(result.data || "");
const htmlData = apolloData.length ? apolloData : parseRenderedRankList(result.data || "");
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

当前的“分层降级”逻辑存在一个小缺陷:如果 parseApolloList 抛出异常(例如页面完全没有 APOLLO 状态),执行流会直接跳到外层的 catch 块并进入 TopHub 降级,从而跳过了 parseRenderedRankList(DOM 解析)这一层。建议局部捕获 Apollo 解析的异常,以便能按预期尝试 DOM 解析。

    let apolloData: ListItem[] = [];
    try {
      apolloData = parseApolloList(result.data || "");
    } catch (e) {
      // 忽略 Apollo 解析错误,尝试后续的 DOM 解析
    }
    const htmlData = apolloData.length ? apolloData : parseRenderedRankList(result.data || "");

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant