[Task] Draft AIP for transaction verification performance microservice

[adamant-al]

Summary

Draft an AIP in Adamant-im/AIPs for improving ADAMANT Node transaction verification performance by optionally offloading cryptographic verification to a local Rust microservice, with Go evaluated as an alternative implementation.

This is a node performance feature. It must not change transaction bytes, IDs, hashes, signatures, fees, rewards, block validity, replay behavior, or consensus rules.

Related:

• [Composite] Increase max transactions per block (Spaceship) #213
• [Task] Draft AIP for node bandwidth improvement #215
• Contextually related, but different scope: [Composite] Draft AIP for transaction data verification #203

Details

Goal

Increase practical node throughput and TPS headroom by reducing CPU latency in transaction verification, especially during:

• transaction-pool admission;
• block generation and validation;
• blockchain loading, sync, and catch-up;
• high-volume tests for features such as [Composite] Increase max transactions per block (Spaceship) #213.

The AIP should specify the research, benchmark plan, security model, implementation boundary, fallback behavior, and rollout strategy for a local verification helper service.

Current node context

Preliminary code review points to transaction verification as a plausible CPU hotspot, but not the only TPS bottleneck.

Relevant current paths:

• helpers/ed.js: wraps sodium.crypto_sign_detached and sodium.crypto_sign_verify_detached.
• logic/transaction.js: serializes transactions, hashes bytes, derives IDs, verifies primary signatures, second signatures, and multisignatures.
• logic/transactionPool.js: receiveTransactions(), processBundled(), and applyUnconfirmedList() process transactions with async.eachSeries, so CPU-heavy verification blocks throughput.
• modules/blocks/verify.js: block processing validates transactions with async.eachSeries; verifyPayload() serializes every transaction again for payload hash checks.
• logic/block.js: block creation and block signature verification also use byte serialization, SHA-256, and Ed25519 verification/signing, although block signatures are lower-frequency than transaction signatures.
• modules/multisignatures.js and logic/multisignature.js: multisignature processing can call verifySignature() repeatedly.

High-probability CPU-expensive functions:

Function/path	Why it matters
Transaction.prototype.getBytes()	Allocates ByteBuffer, serializes base fields, calls type-specific getBytes(), writes bytes in JS loops, and is called repeatedly by ID, hash, signature, second-signature, multisignature, and payload-hash logic.
Transaction.prototype.getHash()	Calls getBytes() and SHA-256; used by getId() and signing.
Transaction.prototype.getId()	Recomputes transaction hash and ID during publish/process/block validation.
Transaction.prototype.verifyBytes()	Copies bytes into a new buffer byte-by-byte, hashes with SHA-256, converts signature/public key from hex, and calls sodium Ed25519 verification.
Transaction.prototype.verifySignature()	Calls getBytes(trs, true, true) and verifyBytes() for the primary signature.
Transaction.prototype.verifySecondSignature()	Calls getBytes(trs, false, true) and verifyBytes() when second signatures are required.
Transaction.prototype.verify() multisignature loop	For every transaction signature, loops over eligible multisignature keys and calls verifySignature() until one matches. Cost can grow with signatures * multisignatures.
logic/transactionPool.js::__private.processVerifyTransaction()	Calls process() and then verify(), causing repeated ID/hash/signature work during pool admission and pool re-application.
modules/blocks/verify.js::__private.checkTransaction()	Recomputes transaction ID and then runs full transaction verification for every transaction in a block.
modules/blocks/verify.js::__private.verifyPayload()	Calls transaction.getBytes() for every transaction to recompute payload hash and totals.
Block.prototype.verifySignature()	Uses block byte serialization, SHA-256, and Ed25519 verification; lower volume than transaction checks but still part of block acceptance.

Other bottlenecks still matter:

• database reads/writes in checkConfirmed(), account lookup, apply/rollback, and mem-table updates;
• sequence queues and serial processing semantics;
• block propagation and sync payload costs;
• transaction-pool ordering and expiration work;
• block apply/undo and round/account state mutation.

So the AIP should not claim that crypto offload alone will linearly increase end-to-end TPS. The expected end-to-end gain depends on how much wall-clock time is currently spent in byte serialization, hashing, signature verification, database work, and queue waiting.

Screenshot benchmark review

Source note:

Verifying a single transaction by making a request to local server written in Rust is x5-10 times faster than using only JavaScript to verify the transaction. If we verify multiple transactions at once (e.g., during blockchain loading), the performance scales up to x65.

Provided table:

Mode	Total time (ms)	Avg / Tx (ms)	Faster than JS
JavaScript, blocking	49,579	4.96	1.00x baseline
Rust over HTTP, blocking	8,257	0.83	5.97x
Rust over HTTP, non-blocking	5,421	0.54	9.19x
Rust batch, blocking	1,333	0.13	38.15x
Rust batch, parallel	759	0.076	65.26x

Preliminary assessment:

• The table appears internally plausible and seems to represent roughly 10,000 transaction verifications.
• The x faster column appears to be calculated from rounded Avg / Tx values, not directly from total times. Direct total-time ratios are close but not identical.
• A 5-10x single-transaction improvement is plausible if the current JS path pays a lot of per-call serialization, SHA-256, hex conversion, and sodium binding overhead.
• A much larger batch improvement is plausible because batching amortizes HTTP overhead and lets the Rust service use multiple CPU cores.
• The numbers should not be accepted as final TPS estimates until reproduced with real ADAMANT transaction fixtures, including transfers, chats, KVS/state, votes, second signatures, multisignatures, invalid signatures, and near-max-payload blocks.
• The benchmark must separate pure cryptographic verification from full node verification, because full node TPS also includes account state, database checks, pool sequencing, block payload verification, and apply/rollback cost.

Estimated performance impact

These are starting hypotheses for the AIP research, not final claims.

Approach	Pure crypto/signature verification estimate	Likely full-node TPS impact	Notes
Current JS path	baseline	baseline	Single Node.js event loop; verification is synchronous and currently serial in many hot paths.
Rust local service, per-transaction HTTP/IPC	5-10x faster for isolated signature checks	modest to medium	Helps latency, but per-request overhead and JS serialization still matter.
Rust local service, batch endpoint	25-65x faster for isolated batch signature checks	medium to high during loading/sync	Best case when validating many independent signatures and using multiple cores.
Go local service, per-transaction HTTP/IPC	roughly 3-8x faster for isolated signature checks	modest to medium	Go has efficient Ed25519 and goroutines, but benchmark against Rust and JS on the same fixtures.
Go local service, batch endpoint	roughly 15-45x faster for isolated batch signature checks	medium during loading/sync	May be simpler operationally for some maintainers, but likely needs direct comparison with Rust implementation.

Expected full-node improvement is likely much lower than pure crypto benchmarks. If crypto verification accounts for roughly 30-70% of the hot path under load, an optimized helper may produce a material but bounded end-to-end gain. If database apply/rollback, sequence queues, or block propagation dominate, TPS gains will be smaller.

The AIP should require profiling before and after:

• node --prof or equivalent CPU profile for transaction admission and block loading;
• event-loop delay measurements;
• wall-clock sync/catch-up time;
• CPU utilization by process and core;
• memory usage for Node.js, helper service, and queues;
• per-transaction latency and batch throughput;
• invalid transaction rejection latency.

RAM and CPU requirements

Expected resource changes:

• CPU time per signature should decrease in the helper, especially with batch verification.
• Total CPU utilization may increase when parallel verification uses more cores. This can reduce latency but may raise contention on very small VPS instances.
• RAM usage will increase because the helper is an additional process with its own runtime, buffers, request queues, and metrics. A small Rust service should have lower baseline RAM than a Go service, but both add operational footprint.
• Batch verification can temporarily increase memory pressure if large transaction batches, public keys, signatures, hashes, or serialized bytes are copied across process boundaries.
• The feature should be optional and disabled by default until benchmarks show it does not materially raise the practical low-resource node baseline.

The AIP should preserve ADAMANT's low-cost node operation goal. A performance mode that requires more CPU cores or RAM should be documented as optional, with conservative defaults and automatic fallback to JS verification.

Proposed AIP scope and boundary

Recommended AIP type: Standard AIP, category Core, because this specifies an official node feature and implementation behavior. It is not a consensus upgrade if the helper only accelerates existing verification semantics.

Recommended phase 1 boundary:

• Node.js remains the source of canonical transaction/block serialization and validation rules.
• The helper receives canonical bytes or precomputed hashes plus public keys and signatures, not arbitrary transaction objects.
• The helper returns only verification results and structured errors.
• The helper must not decide fees, balances, timestamps, sender/requester rules, multisignature threshold rules, block validity, or transaction acceptance policy.
• The node must be able to fall back to current JS verification on helper startup failure, timeout, version mismatch, malformed response, or disabled config.

The AIP should discuss whether moving getBytes()/canonical serialization into Rust or Go is a later phase. That would be higher risk because byte serialization is consensus-sensitive and must require comprehensive cross-language test vectors before production use.

Local service transport options

In the benchmark note, "Rust over HTTP" should be understood as a local sidecar service: the ADAMANT Node process runs in Node.js, the Rust verifier runs on the same host, and Node.js sends local requests such as POST /verifyBatch to 127.0.0.1 or another local-only endpoint. This is not peer-to-peer HTTP between ADAMANT nodes.

Possible Node.js-to-helper communication options:

Transport option	Expected speed	Implementation complexity	Security	Reliability	Notes
HTTP JSON over 127.0.0.1	medium	low	medium	high	Best for proof of concept. Easy to inspect and debug, but has JSON overhead and must never bind to a public interface.
HTTP over Unix domain socket	medium+	medium	high	high	Better production candidate than TCP loopback because filesystem permissions can restrict access and no TCP port is exposed.
gRPC/Protobuf over loopback	high	medium+	medium+	high	Compact schema and good batch API ergonomics, but adds code generation and versioning rules.
gRPC/Protobuf over Unix domain socket	high	high	high	high	Strong production candidate if the added tooling is acceptable.
Raw TCP binary protocol	high	high	medium	medium	Fast, but easy to make brittle; requires custom framing, compatibility, and parser-hardening work.
Child process stdin/stdout protocol	medium+	medium	high	medium	Avoids ports entirely, but requires careful framing, backpressure, restart handling, and log separation.
Rust N-API/native addon	very high	high	medium	lower	Minimizes IPC overhead, but a native crash or memory-safety bug can crash the whole node process. Higher blast radius than a sidecar.
WASM module inside Node.js	medium to high	medium	high	high	Good isolation properties, but Ed25519/SHA-256 performance and library compatibility must be benchmarked.
Node.js worker threads with current crypto	medium	medium	high	high	Reduces event-loop blocking but does not provide the same native Rust/Go crypto speedup by itself.
Shared memory plus binary protocol	very high	very high	medium	low to medium	Too complex for a first milestone; introduces difficult queueing, ownership, and failure-mode edge cases.

Recommended progression:

1. Proof of concept: HTTP JSON or a simple binary/Protobuf endpoint on loopback, with a batch verification API.
2. Production candidate: Unix domain socket plus a compact structured format such as Protobuf, with strict versioning.
3. Defer N-API/native addon until sidecar benchmarks prove the helper boundary is valuable and stable. For a blockchain node, process isolation is likely more valuable than removing the last IPC overhead at the first milestone.

Offload modes

The helper should not duplicate the entire JS validation path in normal production use. There should be distinct rollout modes:

Mode	Node.js responsibility	Rust/Go helper responsibility	Purpose
Shadow/differential mode	Verify exactly as today and also send the same verification inputs to the helper.	Verify independently and return results for comparison.	Safe rollout, benchmarking, and mismatch detection. No consensus or acceptance behavior depends on the helper.
Production offload mode	Build canonical verification inputs, apply transaction/block/account policy, enforce timeouts/versioning, and decide acceptance.	Perform expensive cryptographic checks and return deterministic results/errors.	Performance improvement after differential tests and benchmarks pass.
Fallback mode	Use the current JS verifier exactly as today.	Not used, unavailable, unhealthy, timed out, or version-incompatible.	Reliability and safe operation when the helper is disabled or fails.

Recommended first production boundary:

• Node.js keeps transaction object parsing, canonical byte construction, fee checks, balance checks, timestamp checks, requester/sender rules, multisignature policy, historical exceptions, database checks, and block validity decisions.
• Rust/Go verifies cryptographic preimages only.
• Helper results are accepted only when helper health, protocol version, timeout limits, and response schema checks pass.
• Any helper error must fail closed for that helper call and either fall back to JS verification or reject according to a documented configuration policy.

Helper input boundary

The AIP should explicitly decide what Node.js sends to the helper:

Helper input	Performance	Safety	Notes
hash + publicKey + signature	medium	highest	Rust/Go only verifies Ed25519. Node.js still performs canonical serialization and SHA-256. This is the narrowest and safest boundary, but leaves hashing cost in JS.
canonical bytes + publicKey + signature	high	high	Rust/Go performs SHA-256 plus Ed25519. Node.js still owns canonical serialization. This is likely the best first production target if batch size and byte limits are strict.
Full transaction object	potentially highest	lower	Rust/Go would duplicate canonical serialization and some validation semantics. This is consensus-sensitive and should be deferred until cross-language test vectors prove byte-for-byte equivalence for every transaction type and edge case.

The recommended phase 1 target is canonical bytes + publicKey + signature, with hash + publicKey + signature also measured as the lowest-risk baseline. Full transaction-object verification should be treated as a later phase, not as the first production design.

Security and reliability assessment

Compared with the current in-process JS verifier, a microservice can improve performance but adds new risks:

• local API/IPC attack surface;
• malformed request/response parsing bugs;
• helper crash, hang, timeout, or partial outage;
• version skew between Node.js serialization and helper verification logic;
• accidental network exposure if bound to a public interface;
• queue exhaustion and local denial-of-service under transaction floods;
• inconsistent error handling between JS and helper paths;
• difficulty debugging failures split across two processes;
• supply-chain and build/release risk for an additional binary.

Required safety requirements:

• bind only to loopback or a Unix domain socket by default;
• expose no public peer or public API route directly to the helper;
• use strict request size limits, batch size limits, timeouts, and backpressure;
• fail closed for helper errors, but allow safe fallback to JS verification where appropriate;
• treat helper results as acceleration of deterministic checks, not as authority for consensus rules;
• require byte-for-byte test vectors for every transaction type, including edge cases and historical exceptions;
• require differential tests: JS verifier and helper must agree on valid and invalid fixtures;
• include fuzzing or property tests for malformed signatures, keys, hashes, and payload sizes;
• include observability: latency histograms, error counters, fallback counters, queue depth, helper version, and health status;
• document operational failure modes and restart behavior;
• keep existing sodium/JS path as a maintained fallback at least for the first rollout.

Alternatives to evaluate

• Optimize the current JS path first: avoid byte-by-byte buffer copies in verifyBytes(), reduce repeated getBytes()/getHash() calls, and cache safe immutable preimages where possible.
• Replace or update the current sodium dependency or evaluate Node.js native/WebCrypto Ed25519 where compatible with ADAMANT's current signed preimage semantics.
• Use Node.js worker threads for CPU-heavy verification before introducing a separate service.
• Use Rust N-API/native addon instead of an HTTP/IPC microservice. This may reduce IPC overhead but increases native module coupling and crash risk inside the Node.js process.
• Use Go instead of Rust if benchmarks, maintenance cost, and deployment tooling are better for ADAMANT maintainers.
• Combine crypto offload with [Task] Draft AIP for node bandwidth improvement #215 transport/sync improvements and [Composite] Increase max transactions per block (Spaceship) #213 block-fill benchmarks.

Checklist

• Review AIPS/aip-1.md and choose the final AIP type/category.
• Search existing AIPs and AIP issues to avoid duplicating active performance, verification, or microservice proposals.
• Draft an AIP in Adamant-im/AIPs for optional transaction verification acceleration.
• Link the AIP draft to this issue, [Composite] Increase max transactions per block (Spaceship) #213, [Task] Draft AIP for node bandwidth improvement #215, and any relevant implementation PRs.
• Define non-goals explicitly: no consensus change, no byte serialization change, no transaction ID/signature semantic change.
• Define the phase 1 helper boundary: inputs, outputs, timeout behavior, batch behavior, version negotiation, and fallback.
• Compare local transport options: HTTP loopback, HTTP over Unix socket, gRPC/Protobuf, child process IPC, N-API, WASM, worker threads, and shared memory.
• Define rollout modes: shadow/differential mode, production offload mode, and fallback mode.
• Decide whether the helper receives canonical bytes, SHA-256 hashes, or full transaction objects, and document the security/performance tradeoff.
• Build benchmark fixtures from real ADAMANT transactions: transfer, chat, KVS/state, vote, delegate, second signature, multisignature, invalid signatures, and near-max-payload blocks.
• Reproduce the screenshot benchmark with the same hardware, same fixture count, same Node.js version, same Rust service, and documented methodology.
• Add a Go helper benchmark using the same fixtures and methodology.
• Profile the current JS node path during transaction-pool admission, block validation, and blockchain loading.
• Measure pure crypto verification separately from full node verification.
• Measure Node.js process CPU/RAM, helper CPU/RAM, event-loop delay, queue depth, request latency, batch throughput, and fallback count.
• Estimate low-resource operator impact on a 2 vCPU / 2 GB RAM target.
• Define security requirements for loopback/Unix socket binding, request limits, timeouts, malformed input handling, logging, and local DoS resistance.
• Define reliability requirements for helper crash/restart, health checks, fallback, and version mismatch.
• Add differential test vectors requiring JS and helper verification to agree on valid and invalid cases.
• Decide whether any docs or operator guide updates are needed in Adamant-im/docs.
• Create follow-up implementation issues for Adamant-im/adamant after the AIP direction is accepted.

Notes

This should be researched as an optimization layer around existing deterministic verification, not as a rewrite of node consensus logic.

The safest first milestone is a local batch verifier for signature/hash checks, with Node.js keeping canonical transaction bytes and all validation policy. Moving canonical serialization or transaction-object parsing into Rust/Go should be treated as a later, higher-risk milestone.

The screenshot numbers are promising enough to justify the AIP, especially for blockchain loading and sync, but they need to be reproduced under ADAMANT-specific fixtures and compared against full-node profiles.

Verification

This task is complete when:

• a draft AIP PR exists in Adamant-im/AIPs;
• the AIP links back to this issue and references [Composite] Increase max transactions per block (Spaceship) #213 and [Task] Draft AIP for node bandwidth improvement #215;
• the AIP clearly states that the proposal does not change consensus behavior;
• Rust and Go benchmark results are attached or linked;
• the benchmark methodology is reproducible;
• security and reliability requirements are explicit;
• the proposed helper boundary is narrow enough to preserve deterministic JS node behavior;
• follow-up implementation issues or PRs are created after the AIP direction is accepted.