Flaky E2E tests #176
Description
Describe the bug
Sometimes the E2E tests fail, and rerunning solves the issue. This can both be seen on pipelines and locally, usually when the tests are taking longer than expected (10+ min).
To Reproduce
Unknown, usually when first running the tests after a while (no cache). Extra info is welcomed.
Additional context
The first error usually happens on canceling unbound tokens. Here are the logs:
=== RUN TestIntegrationTestSuite/TestSlashing
=== RUN TestIntegrationTestSuite/TestSlashing/test_unjail_validator
e2e_exec_test.go:183: Executing kiichaind slashing unjail localchain_1783-339763340428243058 with options: [0x3740ac0]
e2e_exec_test.go:197: successfully unjail with options [0x3740ac0]
=== RUN TestIntegrationTestSuite/TestStakingAndDistribution
e2e_exec_test.go:437: Executing kiichaind tx staking delegate localchain_1783-339763340428243058
e2e_exec_test.go:457: kii1twdc7npjct3g28fwt7r68cqgha6g2lz3zq629m successfully delegated 500000000akii to kiivaloper1spzhzhk9lhj4nm5u522urgt33tw2a2974rwhkl
e2e_exec_test.go:520: Executing kiichaind tx staking redelegate localchain_1783-339763340428243058
e2e_exec_test.go:541: kii1twdc7npjct3g28fwt7r68cqgha6g2lz3zq629m successfully redelegated 250000000akii from kiivaloper1spzhzhk9lhj4nm5u522urgt33tw2a2974rwhkl to kiivaloper19dmck6nulxxuy8pp3la6ff030ldxe20yssjwhf
e2e_exec_test.go:464: Executing kiichaind tx staking unbond localchain_1783-339763340428243058
e2e_exec_test.go:484: kii1twdc7npjct3g28fwt7r68cqgha6g2lz3zq629m successfully undelegated 250000000akii to kiivaloper1spzhzhk9lhj4nm5u522urgt33tw2a2974rwhkl
e2e_exec_test.go:491: Executing kiichaind tx staking cancel-unbond localchain_1783-339763340428243058
e2e_exec_test.go:752:
Error Trace: /home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:752
/home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:670
/home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:510
/home/thales/github/kiichain4/tests/e2e/e2e_staking_test.go:113
/home/thales/github/kiichain4/tests/e2e/e2e_test.go:111
/home/thales/go/pkg/mod/golang.org/[email protected]/src/reflect/value.go:584
/home/thales/go/pkg/mod/golang.org/[email protected]/src/reflect/value.go:368
/home/thales/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:196
Error: Condition never satisfied
Test: TestIntegrationTestSuite/TestStakingAndDistribution
Messages: stdOut: {"height":"0","txhash":"F2EE788615DE63EB9F206DA72B6A7BDFFEAB4B47510807D250EA95004DC80243","codespace":"","code":0,"data":"","raw_log":"","logs":[],"info":"","gas_wanted":"0","gas_used":"0","tx":null,"timestamp":"","events":[]}
, stdErr:
=== RUN TestIntegrationTestSuite/TestTokenFactory
=== RUN TestIntegrationTestSuite/TestTokenFactory/create_token_mint_and_burn
e2e_tokenfactory_test.go:120: Creating denom upanda with admin kii17wf65zrd8xph9rrtwkhlu2psza4nzt4khz8k5s on chain localchain_1783-339763340428243058
e2e_exec_test.go:752:
Error Trace: /home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:752
/home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:670
/home/thales/github/kiichain4/tests/e2e/e2e_tokenfactory_test.go:122
/home/thales/github/kiichain4/tests/e2e/e2e_tokenfactory_test.go:44
/home/thales/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:115
Error: Condition never satisfied
Test: TestIntegrationTestSuite/TestTokenFactory/create_token_mint_and_burn
Messages: stdOut: {"height":"0","txhash":"97BFE8EBD10CD28D186E314D0718DC3F356A69969F0BF675BC50173518BF6A97","codespace":"","code":0,"data":"","raw_log":"","logs":[],"info":"","gas_wanted":"0","gas_used":"0","tx":null,"timestamp":"","events":[]}
, stdErr: gas estimate: 3157716
=== RUN TestIntegrationTestSuite/TestVesting
=== RUN TestIntegrationTestSuite/TestVesting/test_delayed_vesting_genesis_account
e2e_exec_test.go:437: Executing kiichaind tx staking delegate localchain_1783-339763340428243058
e2e_exec_test.go:752:
Error Trace: /home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:752
/home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:670
/home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:456
/home/thales/github/kiichain4/tests/e2e/e2e_vesting_test.go:63
/home/thales/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:115
Error: Condition never satisfied
Test: TestIntegrationTestSuite/TestVesting/test_delayed_vesting_genesis_account
Messages: stdOut: {"height":"0","txhash":"02046CD2304B3A28732F1B9BA3CB9F8724E593F95E0CF765A3731FE75A920751","codespace":"","code":0,"data":"","raw_log":"","logs":[],"info":"","gas_wanted":"0","gas_used":"0","tx":null,"timestamp":"","events":[]}
, stdErr:
=== RUN TestIntegrationTestSuite/TestVesting/test_continuous_vesting_genesis_account
e2e_exec_test.go:437: Executing kiichaind tx staking delegate localchain_1783-339763340428243058
e2e_exec_test.go:752:
Error Trace: /home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:752
/home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:670
/home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:456
/home/thales/github/kiichain4/tests/e2e/e2e_vesting_test.go:132
/home/thales/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:115
Error: Condition never satisfied
Test: TestIntegrationTestSuite/TestVesting/test_continuous_vesting_genesis_account
Messages: stdOut: {"height":"0","txhash":"E0ABCD50A2AEE2EF94B349355E296255A9D79B0F4FA55C8863907CCD56DA297F","codespace":"","code":0,"data":"","raw_log":"","logs":[],"info":"","gas_wanted":"0","gas_used":"0","tx":null,"timestamp":"","events":[]}
, stdErr:
=== RUN TestIntegrationTestSuite/TestWasmd
e2e_wasmd_test.go:145: kii1spzhzhk9lhj4nm5u522urgt33tw2a297q44yht storing wasm on host chain localchain_1783-339763340428243058
e2e_exec_test.go:752:
Error Trace: /home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:752
/home/thales/github/kiichain4/tests/e2e/e2e_exec_test.go:670
/home/thales/github/kiichain4/tests/e2e/e2e_wasmd_test.go:146
/home/thales/github/kiichain4/tests/e2e/e2e_wasmd_test.go:42
/home/thales/github/kiichain4/tests/e2e/e2e_test.go:185
/home/thales/go/pkg/mod/golang.org/[email protected]/src/reflect/value.go:584
/home/thales/go/pkg/mod/golang.org/[email protected]/src/reflect/value.go:368
/home/thales/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:196
Error: Condition never satisfied
Test: TestIntegrationTestSuite/TestWasmd
Messages: stdOut: {"height":"0","txhash":"DEC401E511CD40FD9A03498D217EA4C9C4B0E9EEF9084BBE182356E48A25EFAF","codespace":"","code":0,"data":"","raw_log":"","logs":[],"info":"","gas_wanted":"0","gas_used":"0","tx":null,"timestamp":"","events":[]}
, stdErr:
=== NAME TestIntegrationTestSuite
e2e_setup_test.go:195: tearing down e2e integration test suite...
--- FAIL: TestIntegrationTestSuite (1009.74s)
--- PASS: TestIntegrationTestSuite/TestBank (11.04s)
--- PASS: TestIntegrationTestSuite/TestBank/send_tokens_between_accounts (11.04s)
--- PASS: TestIntegrationTestSuite/TestERC20 (35.25s)
--- PASS: TestIntegrationTestSuite/TestERC20/Interact_w/_ERC20 (2.32s)
--- PASS: TestIntegrationTestSuite/TestERC20/Register_ERC20_proposal (26.05s)
--- PASS: TestIntegrationTestSuite/TestERC20/Register_ERC20_proposal/Running_tx_gov_submit-proposal (5.36s)
--- PASS: TestIntegrationTestSuite/TestERC20/Register_ERC20_proposal/Running_tx_gov_deposit (5.35s)
--- PASS: TestIntegrationTestSuite/TestERC20/Register_ERC20_proposal/Running_tx_gov_vote (15.34s)
--- PASS: TestIntegrationTestSuite/TestERC20/ConvertERC20_to_native (5.32s)
--- PASS: TestIntegrationTestSuite/TestEVM (3.13s)
--- PASS: TestIntegrationTestSuite/TestEVM/eth_blockNumber (0.00s)
--- PASS: TestIntegrationTestSuite/TestEVM/eth_chainId (0.00s)
--- PASS: TestIntegrationTestSuite/TestEVM/eth_getBalance_on_zero_address (0.00s)
--- PASS: TestIntegrationTestSuite/TestEVM/web3_clientVersion (0.00s)
--- PASS: TestIntegrationTestSuite/TestEVM/create_and_interact_w/_contract (3.12s)
--- PASS: TestIntegrationTestSuite/TestEncode (0.68s)
--- PASS: TestIntegrationTestSuite/TestEvidence (3.31s)
--- PASS: TestIntegrationTestSuite/TestEvidence/test_evidence_queries (3.31s)
--- PASS: TestIntegrationTestSuite/TestFeeGrant (22.45s)
--- PASS: TestIntegrationTestSuite/TestFeeGrant/test_fee_grant_module (22.45s)
--- PASS: TestIntegrationTestSuite/TestGov (126.10s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_tx_gov_submit-proposal (5.46s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_tx_gov_deposit (5.53s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_tx_gov_vote (15.48s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_tx_gov_submit-proposal#01 (5.40s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_tx_gov_deposit#01 (5.39s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_tx_gov_vote#01 (15.39s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_tx_gov_submit-proposal#02 (5.45s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_tx_gov_deposit#02 (5.58s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_tx_gov_vote#02 (15.41s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_expedited_tx_gov_submit-proposal (25.98s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_expedited_tx_gov_submit-proposal/Running_tx_gov_submit-proposal (5.34s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_expedited_tx_gov_submit-proposal/Running_tx_gov_deposit (5.31s)
--- PASS: TestIntegrationTestSuite/TestGov/Running_expedited_tx_gov_submit-proposal/Running_tx_gov_weighted-vote (15.32s)
--- PASS: TestIntegrationTestSuite/TestIBC (127.82s)
--- PASS: TestIntegrationTestSuite/TestIBC/send_akii_to_chainB (8.80s)
--- PASS: TestIntegrationTestSuite/TestIBC/send_successful_multihop_akii_to_chainA_from_chainA (13.24s)
--- PASS: TestIntegrationTestSuite/TestIBC/send_failed_multihop_akii_to_chainA_from_chainA (16.68s)
--- PASS: TestIntegrationTestSuite/TestIBC/register_ICA_account_and_send_tx_to_chainB (29.09s)
--- PASS: TestIntegrationTestSuite/TestOracle (67.18s)
--- PASS: TestIntegrationTestSuite/TestOracle/Running_tx_gov_submit-proposal (5.31s)
--- PASS: TestIntegrationTestSuite/TestOracle/Running_tx_gov_deposit (5.31s)
--- PASS: TestIntegrationTestSuite/TestOracle/Running_tx_gov_vote (15.31s)
--- PASS: TestIntegrationTestSuite/TestRateLimit (140.48s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_submit-proposal (5.31s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_deposit (5.34s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_vote (15.31s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_submit-proposal#01 (5.32s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_deposit#01 (5.32s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_vote#01 (15.31s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_submit-proposal#02 (5.32s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_deposit#02 (5.31s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_vote#02 (15.36s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_submit-proposal#03 (5.31s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_deposit#03 (5.32s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_vote#03 (15.31s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_submit-proposal#04 (5.32s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_deposit#04 (5.33s)
--- PASS: TestIntegrationTestSuite/TestRateLimit/Running_tx_gov_vote#04 (15.31s)
--- PASS: TestIntegrationTestSuite/TestRestInterfaces (0.05s)
--- PASS: TestIntegrationTestSuite/TestRestInterfaces/test_rest_interfaces (0.05s)
--- PASS: TestIntegrationTestSuite/TestRewards (41.35s)
--- PASS: TestIntegrationTestSuite/TestRewards/Running_tx_gov_submit-proposal (5.32s)
--- PASS: TestIntegrationTestSuite/TestRewards/Running_tx_gov_deposit (5.32s)
--- PASS: TestIntegrationTestSuite/TestRewards/Running_tx_gov_vote (15.39s)
--- PASS: TestIntegrationTestSuite/TestSlashing (5.32s)
--- PASS: TestIntegrationTestSuite/TestSlashing/test_unjail_validator (5.32s)
--- FAIL: TestIntegrationTestSuite/TestStakingAndDistribution (76.31s)
--- FAIL: TestIntegrationTestSuite/TestTokenFactory (60.31s)
--- FAIL: TestIntegrationTestSuite/TestTokenFactory/create_token_mint_and_burn (60.31s)
--- FAIL: TestIntegrationTestSuite/TestVesting (120.67s)
--- FAIL: TestIntegrationTestSuite/TestVesting/test_delayed_vesting_genesis_account (60.32s)
--- FAIL: TestIntegrationTestSuite/TestVesting/test_continuous_vesting_genesis_account (60.34s)
--- FAIL: TestIntegrationTestSuite/TestWasmd (60.34s)
FAIL
FAIL github.com/kiichain/kiichain/v6/tests/e2e 1009.858s
FAIL
make: *** [Makefile:292: run-tests] Error 1
Example pipelines:
- https://github.com/KiiChain/kiichain/actions/runs/19274887329/job/55112177132
- https://github.com/KiiChain/kiichain/actions/runs/19266577097/job/55083777232
- https://github.com/KiiChain/kiichain/actions/runs/19105696005/job/54588811605
- https://github.com/KiiChain/kiichain/actions/runs/19233509020/job/54977423890
- https://github.com/KiiChain/kiichain/actions/runs/19179037643/job/54830918292