Add search result pagination support

[sebthom]

To avoid errors like:

Error: MCP tool "find_code" response (194537 tokens) exceeds maximum allowed tokens (25000).
Please use pagination, filtering, or limit parameters to reduce the response size.

I also implemented a very simple caching mechanism to allow effective and consistent pagination, that caches only last ast-grep result on subsequent find_code calls with the same search pattern. New queries with different pattern or same query with offset=0 will result in cache invalidation.

[HerringtonDarkholme]

I think an alternative way to implement this is not to return JSON but alternative format, or to selectively return fields.

[sebthom]

I don't think changing the format to something else solves not reaching the token limit of the consuming model (in my case Opus 4).

I am using it like this with claude-code and it works fine.

Even if you selectively return fields this my not prevent you from blowing up the token limit when enough matches are found.

The problem also is the client does not know in advance how long the result is so from my view paging is the only option to handle this gracefully.

[HerringtonDarkholme]

No, I don't want to introduce additional behaviors like pagination, which requires additional maintenance and LLM understanding the tool usage.

I also don't see ripgrep tool has such behavior.

Can you check how ripgrep tool call is handling the output? Also running the command without --json is worth trying.

For example this ripgrep mcp does not have any pagination and the return is not json as well https://github.com/mcollina/mcp-ripgrep/blob/main/src/index.ts#L479-L487

[sebthom]

ripgrep mcp has a maxResults parameter.

[HerringtonDarkholme]

I think we should follow ripgrep's convention. Also ripgrep is not returning JSON.

[sebthom]

Superseeded by #14