blockchain同步处理导致死锁 #1305
Description
- blockchain事件处理限定1000容量
- 比特元节点断网恢复后大批量同步区块，由于区块处理逻辑互斥，区块处理数超过1000后，事件处理流程堵塞
- 获取到互斥锁的区块执行协程，在CheckBlock时依赖blockchain GetBlock事件处理（此时事件处理已经堵塞），导致循环依赖死锁
互斥锁1等待协程
937 @ 0x43b4b6 0x44c36c 0x44c346 0x468305 0x15894cc 0x15894a3 0x15986f4 0x15ab148 0x158fb37 0x159320b 0x46c3a1
# 0x468304 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:71
# 0x15894cb sync.(*RWMutex).RLock+0x4b /usr/local/go/src/sync/rwmutex.go:63
# 0x15894a2 github.com/33cn/chain33/blockchain.(*OrphanPool).IsKnownOrphan+0x22 github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/orphanpool.go:52
# 0x15986f3 github.com/33cn/chain33/blockchain.(*BlockChain).ProcessBlock+0x953
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:71
# 0x15ab147 github.com/33cn/chain33/blockchain.(*BlockChain).ProcAddBlockMsg+0x147
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/query_block.go:255
# 0x158fb36 github.com/33cn/chain33/blockchain.(*BlockChain).addBlock+0x2b6
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:188
# 0x159320a github.com/33cn/chain33/blockchain.(*BlockChain).processMsg+0xca
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:408
互斥锁2等待协程
62 @ 0x43b4b6 0x44c36c 0x44c346 0x468305 0x4828e5 0x483eb6 0x483e95 0x1598db8 0x1598c7c 0x15ab148 0x158fb37 0x159320b 0x46c3a1
# 0x468304 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:71
# 0x4828e4 sync.(*Mutex).lockSlow+0x164 /usr/local/go/src/sync/mutex.go:138
# 0x483eb5 sync.(*Mutex).Lock+0x35 /usr/local/go/src/sync/mutex.go:81
# 0x483e94 sync.(*RWMutex).Lock+0x14 /usr/local/go/src/sync/rwmutex.go:111
# 0x1598db7 github.com/33cn/chain33/blockchain.(*BlockChain).maybeAddBestChain+0x97 github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:108
# 0x1598c7b github.com/33cn/chain33/blockchain.(*BlockChain).ProcessBlock+0xedb
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:103
# 0x15ab147 github.com/33cn/chain33/blockchain.(*BlockChain).ProcAddBlockMsg+0x147 github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/query_block.go:255
# 0x158fb36 github.com/33cn/chain33/blockchain.(*BlockChain).addBlock+0x2b6
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:188
# 0x159320a github.com/33cn/chain33/blockchain.(*BlockChain).processMsg+0xca
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:408
互斥锁占用协程，等待blockchain事件处理
1 @ 0x43b4b6 0x44b272 0x9c61a5 0x9c6525 0xb6dc02 0xb72965 0xb71019 0x159af53 0x159af20 0x1599bc9 0x15998c5 0x158b2b3 0x1598ff5 0x1598c7c 0x15ab148 0x158fb37 0x159320b 0x46c3a1
# 0x9c61a4 github.com/33cn/chain33/queue.(*client).WaitTimeout+0x184
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/queue/client.go:155
# 0x9c6524 github.com/33cn/chain33/queue.(*client).Wait+0x24
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/queue/client.go:170
# 0xb6dc01 github.com/33cn/chain33/util.CheckBlock+0x81
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/util/exec.go:25
# 0xb72964 github.com/33cn/chain33/util.PreExecBlock+0x14a4
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/util/util.go:448
# 0xb71018 github.com/33cn/chain33/util.ExecBlock+0x258
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/util/util.go:273
# 0x159af52 github.com/33cn/chain33/blockchain.execBlock+0x172
github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/exec.go:15
# 0x159af1f github.com/33cn/chain33/blockchain.(*BlockChain).connectBlock+0x13f github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:288
# 0x1599bc8 github.com/33cn/chain33/blockchain.(*BlockChain).connectBestChain+0x128 github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:206
# 0x15998c4 github.com/33cn/chain33/blockchain.(*BlockChain).maybeAcceptBlock+0x544 github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:187
# 0x158b2b2 github.com/33cn/chain33/blockchain.(*OrphanPool).ProcessOrphans+0x6b2 github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/orphanpool.go:235
# 0x1598ff4 github.com/33cn/chain33/blockchain.(*BlockChain).maybeAddBestChain+0x2d4 github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:123
# 0x1598c7b github.com/33cn/chain33/blockchain.(*BlockChain).ProcessBlock+0xedb github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/process.go:103
# 0x15ab147 github.com/33cn/chain33/blockchain.(*BlockChain).ProcAddBlockMsg+0x147 github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/query_block.go:255
# 0x158fb36 github.com/33cn/chain33/blockchain.(*BlockChain).addBlock+0x2b6 github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:188
# 0x159320a github.com/33cn/chain33/blockchain.(*BlockChain).processMsg+0xca github.com/33cn/chain33@v1.68.1-0.20230609102640-a70aec9a3f2d/blockchain/proc.go:408