tapdb: add pagination support for ListAssets RPC #1707
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit moves the SortDirection enum from universerpc to tapcommon to resolve circular import dependencies. This enum will be shared across multiple RPC services for pagination consistency.
Updates universerpc to import and use SortDirection from tapcommon.proto instead of defining its own. This eliminates duplication and prevents circular dependencies when adding pagination to taprootassets.proto. Also updates universe_rpc_diff.go to use the correct import path.
Adds pagination request parameters to ListAssetRequest: - offset: Starting position for results - limit: Maximum number of results to return - direction: Sort order (ASC/DESC) Adds pagination response metadata to ListAssetResponse: - total_count: Total number of assets matching the query - has_more: Boolean indicating if more results are available
Adds two new SQL queries to support efficient pagination: - QueryAssetsPaginated: Fetches assets with LIMIT/OFFSET and dynamic ORDER BY clause for ascending/descending sort - CountAssets: Returns total count of assets matching filters without fetching all rows These queries enable efficient pagination at the database level rather than in-memory filtering.
Adds FetchAllAssetsPaginated method to ActiveAssetsStore interface and implementation. This method: - Uses the new QueryAssetsPaginated SQL query for efficient pagination - Returns total count alongside paginated results - Handles witness data fetching for each asset - Supports all existing filters (min/max amount, asset ID, etc.) Also adds fetchAssetsWithWitnessPaginated helper to properly map between SQL row types and domain models.
Updates ListAssets to use the new paginated database queries: - Adds fetchRpcAssetsPaginated helper for paginated fetching - Implements backwards compatibility (default limit: 100) - Validates pagination parameters - Returns total_count and has_more metadata in response - Preserves existing in-memory filtering for unconfirmed mints The implementation maintains full backwards compatibility - clients not specifying pagination parameters get the first 100 results.
Adds test coverage for the new pagination functionality: - TestFetchAllAssetsPaginated: Tests basic pagination scenarios including offset/limit, sorting, edge cases - TestFetchAllAssetsPaginatedWithFilters: Tests pagination combined with amount filters - TestFetchAllAssetsPaginatedConsistency: Verifies paginated results match non-paginated fetching - TestFetchAllAssetsBackwardsCompatibility: Ensures original FetchAllAssets method still works correctly Test scenarios cover first/last pages, small/large page sizes, ascending/descending order, spent assets, and boundary conditions.
Adds documentation for the ListAssets pagination implementation: - issue-875-plan.md: Original implementation plan with requirements and technical specifications - listassets-pagination-progress.md: Progress tracking document with implementation status, test coverage details, and notes These documents provide context for the pagination feature implementation and track the completion status of various phases.
Adds support for pagination parameters in the tapcli assets list command: - --offset: number of results to skip - --limit: maximum number of results to return (default: 100) - --direction: sort order (asc/desc) This allows users to navigate through large asset collections efficiently from the command line, matching the new pagination support in the RPC. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements pagination for the ListAssets RPC call, addressing issue #875. The implementation follows the existing pagination patterns used in the Universe RPCs and maintains full backwards compatibility with existing clients.
The motivation for this change comes from performance issues when dealing with large asset collections. Previously, fetching 250 UTXOs could take over a second, which becomes problematic as users accumulate more assets over time. With pagination, clients can fetch results incrementally, improving both server performance and user experience.
The implementation adds offset, limit, and sort direction parameters to ListAssetRequest, along with total_count and has_more metadata in the response. Under the hood, we've added efficient SQL queries that handle pagination at the database level rather than loading everything into memory first. The default page size is set to 100 items when no limit is specified, ensuring existing clients continue to work without modification.
I've also refactored the SortDirection enum to live in tapcommon.proto to avoid circular dependencies between our proto files. This allows both the universe and taproot-assets services to share the same pagination types.
The test suite covers various pagination scenarios including edge cases like empty results, last page boundaries, and filtering combinations. The tests verify that paginated results match what you'd get from fetching everything at once, ensuring consistency across different access patterns.
One thing to note is that in-memory filtering for unconfirmed mints is still required at the RPC layer since these aren't stored in the database yet. This is a minor limitation but doesn't affect the main pagination functionality for confirmed assets.