Skip to content

StateCompute RPC timeout and max resource usage for some blocks #13553

@ziscky

Description

@ziscky

Checklist

  • This is not a security-related bug/issue. If it is, please follow please follow the security policy.
  • I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
  • I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
  • I did not make any code changes to lotus.

Lotus component

  • lotus daemon - chain sync
  • lotus fvm/fevm - Lotus FVM and FEVM interactions
  • lotus miner/worker - sealing
  • lotus miner - proving(WindowPoSt/WinningPoSt)
  • lotus JSON-RPC API
  • lotus message management (mpool)
  • Other

Lotus Version

v1.35.0-mainnet

Repro Steps

  1. Run '...'
  2. Do '...'
  3. See error '...'
    ...

Describe the Bug

RPC StateCompute to the following heights causes cpu and ram to max out regardless of how much we allocate. When use the CLI command lotus state compute-state --json it works without issue.

Heights:

  • 1594681
  • 2383682

Resources:
Cpu: 42 cores
Ram: 240Gi

Config:
[API]

Binding address for the Lotus API

type: string

env var: LOTUS_API_LISTENADDRESS

ListenAddress = "/ip4/0.0.0.0/tcp/1234/http"

type: string

env var: LOTUS_API_REMOTELISTENADDRESS

#RemoteListenAddress = ""

type: Duration

env var: LOTUS_API_TIMEOUT

Timeout = "30s"

[Backup]

When set to true disables metadata log (.lotus/kvlog). This can save disk

space by reducing metadata redundancy.

Note that in case of metadata corruption it might be much harder to recover

your node if metadata log is disabled

type: bool

env var: LOTUS_BACKUP_DISABLEMETADATALOG

#DisableMetadataLog = true

[Logging]
[Logging.SubsystemLevels]

env var: LOTUS_LOGGING_SUBSYSTEMLEVELS_EXAMPLE-SUBSYSTEM

#example-subsystem = "INFO"

[Libp2p]

Binding address for the libp2p host - 0 means random port.

Format: multiaddress; see https://multiformats.io/multiaddr/

type: []string

env var: LOTUS_LIBP2P_LISTENADDRESSES

#ListenAddresses = ["/ip4/0.0.0.0/tcp/0", "/ip6/::/tcp/0", "/ip4/0.0.0.0/udp/0/quic-v1", "/ip6/::/udp/0/quic-v1", "/ip4/0.0.0.0/udp/0/quic-v1/webtransport", "/ip6/::/udp/0/quic-v1/webtransport"]

Addresses to explicitally announce to other peers. If not specified,

all interface addresses are announced

Format: multiaddress

type: []string

env var: LOTUS_LIBP2P_ANNOUNCEADDRESSES

#AnnounceAddresses = []

Addresses to not announce

Format: multiaddress

type: []string

env var: LOTUS_LIBP2P_NOANNOUNCEADDRESSES

#NoAnnounceAddresses = []

When not disabled (default), lotus asks NAT devices (e.g., routers), to

open up an external port and forward it to the port lotus is running on.

When this works (i.e., when your router supports NAT port forwarding),

it makes the local lotus node accessible from the public internet

type: bool

env var: LOTUS_LIBP2P_DISABLENATPORTMAP

#DisableNatPortMap = false

ConnMgrLow is the number of connections that the basic connection manager

will trim down to.

type: uint

env var: LOTUS_LIBP2P_CONNMGRLOW

#ConnMgrLow = 150

ConnMgrHigh is the number of connections that, when exceeded, will trigger

a connection GC operation. Note: protected/recently formed connections don't

count towards this limit.

type: uint

env var: LOTUS_LIBP2P_CONNMGRHIGH

#ConnMgrHigh = 180

ConnMgrGrace is a time duration that new connections are immune from being

closed by the connection manager.

type: Duration

env var: LOTUS_LIBP2P_CONNMGRGRACE

#ConnMgrGrace = "20s"

[Pubsub]

Run the node in bootstrap-node mode

type: bool

env var: LOTUS_PUBSUB_BOOTSTRAPPER

#Bootstrapper = false

type: string

env var: LOTUS_PUBSUB_REMOTETRACER

#RemoteTracer = ""

Path to file that will be used to output tracer content in JSON format.

If present tracer will save data to defined file.

Format: file path

type: string

env var: LOTUS_PUBSUB_JSONTRACER

#JsonTracer = ""

Connection string for elasticsearch instance.

If present tracer will save data to elasticsearch.

Format: https://:@<elasticsearch_url>:/

type: string

env var: LOTUS_PUBSUB_ELASTICSEARCHTRACER

#ElasticSearchTracer = ""

Name of elasticsearch index that will be used to save tracer data.

This property is used only if ElasticSearchTracer propery is set.

type: string

env var: LOTUS_PUBSUB_ELASTICSEARCHINDEX

#ElasticSearchIndex = ""

Auth token that will be passed with logs to elasticsearch - used for weighted peers score.

type: string

env var: LOTUS_PUBSUB_TRACERSOURCEAUTH

#TracerSourceAuth = ""

[Client]

The maximum number of simultaneous data transfers between the client

and storage providers for storage deals

type: uint64

env var: LOTUS_CLIENT_SIMULTANEOUSTRANSFERSFORSTORAGE

#SimultaneousTransfersForStorage = 20

The maximum number of simultaneous data transfers between the client

and storage providers for retrieval deals

type: uint64

env var: LOTUS_CLIENT_SIMULTANEOUSTRANSFERSFORRETRIEVAL

#SimultaneousTransfersForRetrieval = 20

Require that retrievals perform no on-chain operations. Paid retrievals

without existing payment channels with available funds will fail instead

of automatically performing on-chain operations.

type: bool

env var: LOTUS_CLIENT_OFFCHAINRETRIEVAL

#OffChainRetrieval = false

[Wallet]

type: string

env var: LOTUS_WALLET_REMOTEBACKEND

#RemoteBackend = ""

type: bool

env var: LOTUS_WALLET_ENABLELEDGER

#EnableLedger = false

type: bool

env var: LOTUS_WALLET_DISABLELOCAL

#DisableLocal = false

[Fees]

type: types.FIL

env var: LOTUS_FEES_DEFAULTMAXFEE

#DefaultMaxFee = "0.07 FIL"

[Chainstore]

type: bool

env var: LOTUS_CHAINSTORE_ENABLESPLITSTORE

EnableSplitstore = false

[Chainstore.Splitstore]

ColdStoreType specifies the type of the coldstore.

It can be "discard" (default) for discarding cold blocks, "messages" to store only messages or "universal" to store all chain state..

type: string

env var: LOTUS_CHAINSTORE_SPLITSTORE_COLDSTORETYPE

#ColdStoreType = "discard"

HotStoreType specifies the type of the hotstore.

Only currently supported value is "badger".

type: string

env var: LOTUS_CHAINSTORE_SPLITSTORE_HOTSTORETYPE

#HotStoreType = "badger"

MarkSetType specifies the type of the markset.

It can be "map" for in memory marking or "badger" (default) for on-disk marking.

type: string

env var: LOTUS_CHAINSTORE_SPLITSTORE_MARKSETTYPE

#MarkSetType = "badger"

HotStoreMessageRetention specifies the retention policy for messages, in finalities beyond

the compaction boundary; default is 0.

type: uint64

env var: LOTUS_CHAINSTORE_SPLITSTORE_HOTSTOREMESSAGERETENTION

#HotStoreMessageRetention = 0

HotStoreFullGCFrequency specifies how often to perform a full (moving) GC on the hotstore.

A value of 0 disables, while a value 1 will do full GC in every compaction.

Default is 20 (about once a week).

type: uint64

env var: LOTUS_CHAINSTORE_SPLITSTORE_HOTSTOREFULLGCFREQUENCY

#HotStoreFullGCFrequency = 20

HotStoreMaxSpaceTarget sets a target max disk size for the hotstore. Splitstore GC

will run moving GC if disk utilization gets within a threshold (150 GB) of the target.

Splitstore GC will NOT run moving GC if the total size of the move would get

within 50 GB of the target, and instead will run a more aggressive online GC.

If both HotStoreFullGCFrequency and HotStoreMaxSpaceTarget are set then splitstore

GC will trigger moving GC if either configuration condition is met.

A reasonable minimum is 2x fully GCed hotstore size + 50 G buffer.

At this minimum size moving GC happens every time, any smaller and moving GC won't

be able to run. In spring 2023 this minimum is ~550 GB.

type: uint64

env var: LOTUS_CHAINSTORE_SPLITSTORE_HOTSTOREMAXSPACETARGET

#HotStoreMaxSpaceTarget = 650000000000

When HotStoreMaxSpaceTarget is set Moving GC will be triggered when total moving size

exceeds HotstoreMaxSpaceTarget - HotstoreMaxSpaceThreshold

type: uint64

env var: LOTUS_CHAINSTORE_SPLITSTORE_HOTSTOREMAXSPACETHRESHOLD

#HotStoreMaxSpaceThreshold = 150000000000

Safety buffer to prevent moving GC from overflowing disk when HotStoreMaxSpaceTarget

is set. Moving GC will not occur when total moving size exceeds

HotstoreMaxSpaceTarget - HotstoreMaxSpaceSafetyBuffer

type: uint64

env var: LOTUS_CHAINSTORE_SPLITSTORE_HOTSTOREMAXSPACESAFETYBUFFER

#HotstoreMaxSpaceSafetyBuffer = 50000000000

[Fevm]

EnableEthRPC enables eth_ rpc, and enables storing a mapping of eth transaction hashes to filecoin message Cids.

This will also enable the RealTimeFilterAPI and HistoricFilterAPI by default, but they can be disabled by config options above.

type: bool

env var: LOTUS_FEVM_ENABLEETHRPC

EnableEthRPC = true

EthTxHashMappingLifetimeDays the transaction hash lookup database will delete mappings that have been stored for more than x days

Set to 0 to keep all mappings

type: int

env var: LOTUS_FEVM_ETHTXHASHMAPPINGLIFETIMEDAYS

EthTxHashMappingLifetimeDays = 0

[Events]

DisableRealTimeFilterAPI will disable the RealTimeFilterAPI that can create and query filters for actor events as they are emitted.

The API is enabled when Fevm.EnableEthRPC or EnableActorEventsAPI is true, but can be disabled selectively with this flag.

type: bool

env var: LOTUS_EVENTS_DISABLEREALTIMEFILTERAPI

DisableRealTimeFilterAPI = false

DisableHistoricFilterAPI will disable the HistoricFilterAPI that can create and query filters for actor events

that occurred in the past. HistoricFilterAPI maintains a queryable index of events.

The API is enabled when Fevm.EnableEthRPC or EnableActorEventsAPI is true, but can be disabled selectively with this flag.

type: bool

env var: LOTUS_EVENTS_DISABLEHISTORICFILTERAPI

DisableHistoricFilterAPI = false

EnableActorEventsAPI enables the Actor events API that enables clients to consume events

emitted by (smart contracts + built-in Actors).

This will also enable the RealTimeFilterAPI and HistoricFilterAPI by default, but they can be

disabled by setting their respective Disable* options.

type: bool

env var: LOTUS_EVENTS_ENABLEACTOREVENTSAPI

EnableActorEventsAPI = true

FilterTTL specifies the time to live for actor event filters. Filters that haven't been accessed longer than

this time become eligible for automatic deletion.

type: Duration

env var: LOTUS_EVENTS_FILTERTTL

#FilterTTL = "24h0m0s"

MaxFilters specifies the maximum number of filters that may exist at any one time.

type: int

env var: LOTUS_EVENTS_MAXFILTERS

#MaxFilters = 100

MaxFilterResults specifies the maximum number of results that can be accumulated by an actor event filter.

type: int

env var: LOTUS_EVENTS_MAXFILTERRESULTS

MaxFilterResults = 0

MaxFilterHeightRange specifies the maximum range of heights that can be used in a filter (to avoid querying

the entire chain)

type: uint64

env var: LOTUS_EVENTS_MAXFILTERHEIGHTRANGE

#MaxFilterHeightRange = 2880

DatabasePath is the full path to a sqlite database that will be used to index actor events to

support the historic filter APIs. If the database does not exist it will be created. The directory containing

the database must already exist and be writeable. If a relative path is provided here, sqlite treats it as

relative to the CWD (current working directory).

type: string

env var: LOTUS_EVENTS_DATABASEPATH

#DatabasePath = ""

[Index]

EXPERIMENTAL FEATURE. USE WITH CAUTION

EnableMsgIndex enables indexing of messages on chain.

type: bool

env var: LOTUS_INDEX_ENABLEMSGINDEX

EnableMsgIndex = true

[FaultReporter]

EnableConsensusFaultReporter controls whether the node will monitor and

report consensus faults. When enabled, the node will watch for malicious

behaviors like double-mining and parent grinding, and submit reports to the

network. This can earn reporter rewards, but is not guaranteed. Nodes should

enable fault reporting with care, as it may increase resource usage, and may

generate gas fees without earning rewards.

type: bool

env var: LOTUS_FAULTREPORTER_ENABLECONSENSUSFAULTREPORTER

#EnableConsensusFaultReporter = false

ConsensusFaultReporterDataDir is the path where fault reporter state will be

persisted. This directory should have adequate space and permissions for the

node process.

type: string

env var: LOTUS_FAULTREPORTER_CONSENSUSFAULTREPORTERDATADIR

#ConsensusFaultReporterDataDir = ""

ConsensusFaultReporterAddress is the wallet address used for submitting

ReportConsensusFault messages. It will pay for gas fees, and receive any

rewards. This address should have adequate funds to cover gas fees.

type: string

env var: LOTUS_FAULTREPORTER_CONSENSUSFAULTREPORTERADDRESS

#ConsensusFaultReporterAddress = ""

LOTUS_DISABLE_PRE_MIGRATIONS skips the pre-migration and runs the non-cached

migration at the network upgrade epoch. This is recommended for full archival

nodes or systems that need to keep large amounts of state (RPC providers).

LOTUS_DISABLE_PRE_MIGRATIONS=1

Logging Information

2026-03-20T14:34:47.711Z	WARN	rpc	go-jsonrpc@v0.8.0/handler.go:425	error in RPC call to 'Filecoin.StateGetActor': resolution lookup failed (f1y6nkvsju4kvdkslinp5wdtzliy7shuehijyjymq):
    github.com/filecoin-project/lotus/chain/state.(*StateTree).GetActor
        /work/chain/state/statetree.go:373
  - resolve address f1y6nkvsju4kvdkslinp5wdtzliy7shuehijyjymq:
    github.com/filecoin-project/lotus/chain/state.(*StateTree).lookupInternalIDAddress
        /work/chain/state/statetree.go:338
  - actor not found
2026/03/20 14:34:47 POST /account/balance 200.893282ms

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    📌 Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions