feat(gRPC): Add transaction and read/write services for internal usage #8552

bingyanglin · 2025-09-12T09:36:20Z

Description of change

Added transaction filters for client subscription
- WASP team has provided the feedback that the transaction filter subscriber works as expected
Added node service including GetLatestCheckpoint, ExecuteTransaction, and GetObject.
- The GetLatestCheckpoint support get the latest checkpoint data or checkpoint summary
- The ExecuteTransaction is fully aligned with the current JSON-RPC design in IOTA and SUI (for the request, see the function execute_transaction_block in pub trait WriteApi; for the response, see IOTATransactionBlockResponse and SuiTransactionBlockResponse, respectively.).
  - For iota-rest-api, there is an additional field EffectsFinality in the response, but it is not used in our codebase, so we do not implement this field.
- The GetObject is also aligned with the current JSON-RPC design in IOTA and SUI (see the function get_object in the pub trait ReadApi for more details).
(deprecated) To be fully compatible with the current JSON-RPC, the return types (i.e., IotaTransactionBlockEffects, IotaTransactionBlockResponse, and IotaObjectData for transaction filter, ExecuteTransaction, and GetObject, respectively) we use serialized Json data for ease of usage and avoid additional effort, like we have the following for some of the fields. These annotations work for JSON but break BCS round-trips. In addition, to avoid repetitive work, we leveraged the existing types and functions (if available) in the JSON-RPC api.

      #[serde(skip_serializing_if = "Option::is_none")]
      #[serde(skip_serializing_if = "Vec::is_empty", default)]
      #[serde_as(as = "Base64")]
      #[serde_as(as = "Option<BigInt<u64>>")]

(NOTE: The above is deprecated, after discussion, we need to do bcs streaming for the response data, instead of using JSON string)

Moved the GrpcApiConfig to iota-config, and implemented the subscribe_events and subscribe_transactions in SubscriptionHandler of iota-node directly, so as to resolve the cyclic dependency for iota-grpc-api to use iota-core.
Implemented WriteService and ReadService to ease of future extension of more API calls.
Removed the GrpcStateReader and RestStateReaderAdapter in iota-grpc-api crate, and leveraged the existing REST API infrastructure (including the RestStateReader and RestReadStore four our use cases). The plan is after REST API is deprecated, we can directly rename them to be Grpc*, so as to avoid repetitive work in the future.
Enhanced RestStateReader and RestReadStore to access AuthorityState and AuthorityPerEpochStore for usage in ReadGrpcService and WriteGrpcServce. This is required to return more information like what JSON-RPC does.
Added test utils for cleaner code.
NOTE THAT the document will be enhanced/updated after more APIs are added and the design pattern is finalized, otherwise we might need to modify the document back and forth. (see gRPC: Document the finalised internal gRPC #8695)

Links to any relevant issues

fixes #7724

How the change has been tested

Basic tests (linting, compilation, formatting, unit/integration tests)
Patch-specific tests (correctness, functionality coverage)
I have added tests that prove my fix is effective or that my feature works
I have checked that new and existing unit tests pass locally with my changes

Release Notes

Nodes (Validators and Full nodes): For gPRC, added transaction filters for client subscription, and added 3 APIs including GetLatestCheckpoint, ExecuteTransaction, and GetObject.

vercel · 2025-09-12T09:36:24Z

The latest updates on your projects. Learn more about Vercel for GitHub.

6 Skipped Deployments

Project	Deployment	Preview	Updated (UTC)
apps-backend	Ignored	Preview	Sep 30, 2025 10:19am
apps-ui-kit	Ignored	Preview	Sep 30, 2025 10:19am
iota-evm-bridge	Ignored	Preview	Sep 30, 2025 10:19am
iota-multisig-toolkit	Ignored	Preview	Sep 30, 2025 10:19am
rebased-explorer	Ignored	Preview	Sep 30, 2025 10:19am
wallet-dashboard	Ignored	Preview	Sep 30, 2025 10:19am

crates/iota-grpc-api/proto/common.proto

semenov-vladyslav

Overall looks good. There are a few nits to improve.

semenov-vladyslav · 2025-09-30T12:14:35Z

crates/iota-types/src/storage/read_store.rs

    fn indexes(&self) -> Option<&dyn RestIndexes>;
+
+    /// Enable downcasting to concrete types for enhanced functionality
+    fn as_any(&self) -> &dyn std::any::Any;


A weird method for a trait. Should RestStateReader be extended with pub fn authority_state(&self) -> Arc<AuthorityState> and pub fn load_epoch_store_one_call_per_task(&self) -> Arc<AuthorityPerEpochStore> methods instead?

semenov-vladyslav · 2025-09-30T12:16:04Z

crates/iota-core/src/storage.rs

+    /// Get access to the underlying AuthorityState
+    pub fn authority_state(&self) -> &Arc<AuthorityState> {
+        &self.state
+    }
+
+    /// Load epoch store for transaction processing
+    pub fn load_epoch_store_one_call_per_task(&self) -> Arc<AuthorityPerEpochStore> {
+        self.state.load_epoch_store_one_call_per_task().clone()
+    }


Why not move these methods and make them a part of RestStateReader trait?

semenov-vladyslav · 2025-09-30T12:18:23Z

crates/iota-core/src/subscription_handler.rs

+    pub fn subscribe_events(
+        &self,
+        filter: EventFilter,
+    ) -> Box<dyn futures::Stream<Item = IotaEvent> + Send + Unpin> {


Should it be BoxStream<'_, IotaEvent>? Or is Unpin important here?

semenov-vladyslav · 2025-09-30T12:19:05Z

crates/iota-core/src/subscription_handler.rs

+        &self,
+        filter: EventFilter,
+    ) -> Box<dyn futures::Stream<Item = IotaEvent> + Send + Unpin> {
+        Box::new(Box::pin(self.event_streamer.subscribe(filter)))


Double Box? Doesn't look right. With BoxStream we can just use one Box::pin.

semenov-vladyslav · 2025-09-30T12:19:33Z

crates/iota-core/src/subscription_handler.rs

+    ) -> Box<dyn futures::Stream<Item = IotaTransactionBlockEffects> + Send + Unpin> {
+        Box::new(Box::pin(self.transaction_streamer.subscribe(filter)))


semenov-vladyslav · 2025-09-30T15:30:47Z

crates/iota-grpc-api/src/checkpoint_service.rs

 impl CheckpointService for CheckpointGrpcService {
    type StreamCheckpointsStream =
-        Pin<Box<dyn futures::Stream<Item = Result<crate::checkpoint::Checkpoint, Status>> + Send>>;
+        Pin<Box<dyn futures::Stream<Item = Result<Checkpoint, Status>> + Send>>;


Suggested change

Pin<Box<dyn futures::Stream<Item = Result<Checkpoint, Status>> + Send>>;

futures::BoxStream<Result<Checkpoint, Status>>;

semenov-vladyslav · 2025-09-30T15:31:16Z

crates/iota-grpc-api/src/checkpoint_service.rs

+        &self,
+        request: Request<GetLatestCheckpointRequest>,
+    ) -> Result<Response<Checkpoint>, Status> {
+        debug!("get_latest_checkpoint called");


Suggested change

debug!("get_latest_checkpoint called");

semenov-vladyslav · 2025-09-30T15:47:42Z

crates/iota-grpc-api/src/read_service.rs

+        request: Request<GetObjectRequest>,
+    ) -> Result<Response<GetObjectResponse>, Status> {
+        let req = request.into_inner();
+        debug!("get_object called");


Suggested change

debug!("get_object called");

semenov-vladyslav · 2025-09-30T15:55:40Z

crates/iota-grpc-api/src/transaction_service.rs

+#[tonic::async_trait]
+impl TransactionService for TransactionGrpcService {
+    type StreamTransactionsStream =
+        std::pin::Pin<Box<dyn futures::Stream<Item = Result<Transaction, Status>> + Send>>;


Suggested change

std::pin::Pin<Box<dyn futures::Stream<Item = Result<Transaction, Status>> + Send>>;

futures::BoxStream<Result<Transaction, Status>>;

semenov-vladyslav · 2025-09-30T15:59:21Z

crates/iota-grpc-api/src/types.rs

+        // Downcast to RestReadStore for enhanced functionality
+        self.state_reader
+            .as_any()
+            .downcast_ref::<RestReadStore>()


This downcast suggests that the only type for the dyn object state_reader is RestReadStore, all other types are "fallback" or mock.

bingyanglin self-assigned this Sep 12, 2025

github-actions bot added the ci Issues related to our CI pipeline label Sep 12, 2025

iota-ci added core-protocol node Issues related to the Core Node team labels Sep 12, 2025

bingyanglin changed the title ~~Feat/grpc transaction filter~~ gRPC: Add subscription to transaction including filtering Sep 12, 2025

bingyanglin changed the title ~~gRPC: Add subscription to transaction including filtering~~ feat(gRPC): Add subscription to transaction including filtering Sep 12, 2025

bingyanglin force-pushed the feat/grpc-transaction-filter branch 4 times, most recently from 0a5ec53 to ebbe1d4 Compare September 24, 2025 13:50

bingyanglin changed the title ~~feat(gRPC): Add subscription to transaction including filtering~~ feat(gRPC): Add transaction and API services for internal usage Sep 24, 2025

bingyanglin force-pushed the feat/grpc-transaction-filter branch 6 times, most recently from c62698b to 707e542 Compare September 26, 2025 04:40

bingyanglin changed the title ~~feat(gRPC): Add transaction and API services for internal usage~~ feat(gRPC): Add transaction and node services for internal usage Sep 26, 2025

bingyanglin force-pushed the feat/grpc-transaction-filter branch 2 times, most recently from b314d54 to d6053fd Compare September 26, 2025 13:09

bingyanglin changed the title ~~feat(gRPC): Add transaction and node services for internal usage~~ feat(gRPC): Add transaction and read/write services for internal usage Sep 28, 2025

bingyanglin force-pushed the feat/grpc-transaction-filter branch 2 times, most recently from 52aa127 to 89e7a56 Compare September 29, 2025 14:44

feat: grpc transaction, read, and write services

83033fe

bingyanglin force-pushed the feat/grpc-transaction-filter branch from 726181b to 83033fe Compare September 29, 2025 17:03

bingyanglin added 2 commits September 30, 2025 15:38

refactoring: GrpcReader clean up

508e3b4

fix: clippy

3f0a86b

bingyanglin marked this pull request as ready for review September 30, 2025 07:44

bingyanglin requested a review from a team as a code owner September 30, 2025 07:44

bingyanglin requested review from a team as code owners September 30, 2025 07:44

fix: cargo sort

f3d1bd3

Thoralf-M reviewed Sep 30, 2025

View reviewed changes

crates/iota-grpc-api/proto/common.proto Outdated Show resolved Hide resolved

refactoring: remove double versioning and unused ObjectRef

8d40b0b

bingyanglin force-pushed the feat/grpc-transaction-filter branch from 3050afb to 8d40b0b Compare September 30, 2025 10:17

semenov-vladyslav reviewed Sep 30, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(gRPC): Add transaction and read/write services for internal usage #8552

feat(gRPC): Add transaction and read/write services for internal usage #8552

bingyanglin commented Sep 12, 2025 •

edited

Loading

Uh oh!

vercel bot commented Sep 12, 2025 •

edited

Loading

Uh oh!

Uh oh!

semenov-vladyslav left a comment

Uh oh!

semenov-vladyslav Sep 30, 2025

Uh oh!

semenov-vladyslav Sep 30, 2025

Uh oh!

semenov-vladyslav Sep 30, 2025

Uh oh!

semenov-vladyslav Sep 30, 2025

Uh oh!

semenov-vladyslav Sep 30, 2025

Uh oh!

semenov-vladyslav Sep 30, 2025

Uh oh!

semenov-vladyslav Sep 30, 2025

Uh oh!

semenov-vladyslav Sep 30, 2025

Uh oh!

semenov-vladyslav Sep 30, 2025

Uh oh!

semenov-vladyslav Sep 30, 2025

Uh oh!

Uh oh!

		) -> Box<dyn futures::Stream<Item = IotaTransactionBlockEffects> + Send + Unpin> {
		Box::new(Box::pin(self.transaction_streamer.subscribe(filter)))

	Pin<Box<dyn futures::Stream<Item = Result<Checkpoint, Status>> + Send>>;
	futures::BoxStream<Result<Checkpoint, Status>>;

	std::pin::Pin<Box<dyn futures::Stream<Item = Result<Transaction, Status>> + Send>>;
	futures::BoxStream<Result<Transaction, Status>>;

feat(gRPC): Add transaction and read/write services for internal usage #8552

Are you sure you want to change the base?

feat(gRPC): Add transaction and read/write services for internal usage #8552

Conversation

bingyanglin commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of change

Links to any relevant issues

How the change has been tested

Release Notes

Uh oh!

vercel bot commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

semenov-vladyslav left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bingyanglin commented Sep 12, 2025 •

edited

Loading

vercel bot commented Sep 12, 2025 •

edited

Loading