bug(v1.2.x): iavl set error "Value missing for key"

[yihuang]

We met an app hash mismatch in mantrachain testnet, there's a balance change fails to persist in the iavl db, checking the node logs leads us to this error message:

�[90m10:24PM�[0m �[31mERR�[0m �[1miavl set error�[0m �[36merror=�[0m�[31m�[1m"Value missing for key [0 0 0 0 0 98 237 164 0 0 0 41] corresponding to nodeKey 73000000000062eda400000029"�[0m�[0m �[36mmodule=�[0mserver

It's similar to a fixed bug, but different, the last bug we find is only about the root node, but this time it's an internal node.

pruning is enabled:

pruning = "custom"
pruning-keep-recent = "80000"
pruning-interval = "17"
pruning-keep-every = "4000"

[yihuang]

unconfirmed hypothesis, since BatchWithFlusher could separate the changes on the same version into multiple batches, if the batches are persisted on disk on the wrong order, then we could face loss of data:

• root node of the version is saved, but some internal nodes are not.
• when restart, we only check root node exists, and move on normally.
• trigger error down the road.

The other hypothesis is still the pruning process, some bug makes it delete the node, similar to the last time