Arby gets stuck unable to replace order

[kilrau]

Below output of my environment which is configured to issue ETH/BTC orders. As one can see, sell order is missing even though all balances and limits are fine. Arby logs show that it failed to replace an order because ALREADY_EXISTS. Restart of arby did not fix this.

simnet > listorders

Trading pair: BTC/USDT
┌────────────────────────────────────────────────┬────────────────────────────────────────────────┐
│ Buy                                            │ Sell                                           │
├───────────────┬─────────────┬──────────────────┼───────────────┬─────────────┬──────────────────┤
│ Quantity      │ Price       │ Alias            │ Quantity      │ Price       │ Alias            │
├───────────────┼─────────────┼──────────────────┼───────────────┼─────────────┼──────────────────┤
│ 0.01878902    │ 15350.3868  │ NestGrid         │ 0.29509678    │ 15935.4498  │ NestGrid         │
└───────────────┴─────────────┴──────────────────┴───────────────┴─────────────┴──────────────────┘

Trading pair: ETH/BTC
┌────────────────────────────────────────────────┬────────────────────────────────────────────────┐
│ Buy                                            │ Sell                                           │
├───────────────┬─────────────┬──────────────────┼───────────────┬─────────────┬──────────────────┤
│ Quantity      │ Price       │ Alias            │ Quantity      │ Price       │ Alias            │
├───────────────┼─────────────┼──────────────────┼───────────────┼─────────────┼──────────────────┤
│ 14.1075       │ 0.0279006   │ SheriffSolution  │               │             │                  │
└───────────────┴─────────────┴──────────────────┴───────────────┴─────────────┴──────────────────┘

Trading pair: LTC/BTC
┌────────────────────────────────────────────────┬────────────────────────────────────────────────┐
│ Buy                                            │ Sell                                           │
├───────────────┬─────────────┬──────────────────┼───────────────┬─────────────┬──────────────────┤
│ Quantity      │ Price       │ Alias            │ Quantity      │ Price       │ Alias            │
├───────────────┼─────────────┼──────────────────┼───────────────┼─────────────┼──────────────────┤
│ 10            │ 0.004       │ OzoneYellow      │ 9.4           │ 0.0042      │ OzoneYellow      │
└───────────────┴─────────────┴──────────────────┴───────────────┴─────────────┴──────────────────┘

Trading pair: USDT/DAI
┌────────────────────────────────────────────────┬────────────────────────────────────────────────┐
│ Buy                                            │ Sell                                           │
├───────────────┬─────────────┬──────────────────┼───────────────┬─────────────┬──────────────────┤
│ Quantity      │ Price       │ Alias            │ Quantity      │ Price       │ Alias            │
└───────────────┴─────────────┴──────────────────┴───────────────┴─────────────┴──────────────────┘
simnet > getbalance

Balance:
┌──────────┬───────────────┬───────────────────────────────┬────────────────────────────┐
│ Currency │ Total Balance │ Wallet Balance (Not Tradable) │ Channel Balance (Tradable) │
├──────────┼───────────────┼───────────────────────────────┼────────────────────────────┤
│ BTC      │ 1000.72965053 │ 997.9116993                   │ 2.81795123                 │
├──────────┼───────────────┼───────────────────────────────┼────────────────────────────┤
│ DAI      │ 4000          │ 2500                          │ 1500                       │
├──────────┼───────────────┼───────────────────────────────┼────────────────────────────┤
│ ETH      │ 38.55040222   │ 29.99901497                   │ 8.55138725                 │
├──────────┼───────────────┼───────────────────────────────┼────────────────────────────┤
│ LTC      │ 1013.88977649 │ 1004.9997805                  │ 8.88999599                 │
├──────────┼───────────────┼───────────────────────────────┼────────────────────────────┤
│ USDT     │ 2000          │ 1000                          │ 1000                       │
└──────────┴───────────────┴───────────────────────────────┴────────────────────────────┘
simnet > tradinglimits

Trading Limits:
┌──────────┬────────────┬────────────┬───────────────────┬──────────────────┐
│ Currency │ Max Buy    │ Max Sell   │ Reserved Outbound │ Reserved Inbound │
├──────────┼────────────┼────────────┼───────────────────┼──────────────────┤
│ BTC      │ 2.13195826 │ 2.37434352 │ 0.39360771        │ 0                │
├──────────┼────────────┼────────────┼───────────────────┼──────────────────┤
│ DAI      │ 525        │ 1500       │ 0                 │ 0                │
├──────────┼────────────┼────────────┼───────────────────┼──────────────────┤
│ ETH      │ 5.505375   │ 8.55138725 │ 0                 │ 14.1075          │
├──────────┼────────────┼────────────┼───────────────────┼──────────────────┤
│ LTC      │ 1.00964201 │ 8.78999599 │ 0                 │ 0                │
├──────────┼────────────┼────────────┼───────────────────┼──────────────────┤
│ USDT     │ 525        │ 1000       │ 0                 │ 0                │
└──────────┴────────────┴────────────┴───────────────────┴──────────────────┘
simnet > logs arby
2020-11-06 11:04:53.5353 [OpenDEX] trace: Creating ETH/BTC sell order with id arby-ETH/BTC-sell-order, quantity 8.46587337 and price 0.0290292
2020-11-06 11:04:53.5353 [OpenDEX] warn: 6 ALREADY_EXISTS: order with local id arby-ETH/BTC-sell-order already exists. Retrying in 5000ms.
2020-11-06 11:04:55.5555 [Centralized] trace: heartbeat from ETHBTC socket
2020-11-06 11:04:58.5858 [Centralized] info: Base asset balance 1000 and quote asset balance 1000
2020-11-06 11:04:58.5858 [OpenDEX] trace: Base asset balance 8.55138725 (maxinbound: 19.612875, maxoutbound: 8.55138725) and quote asset balance 2.81795123 (maxinbound: 2.13195826, maxoutbound: 2.76795123).

[kilrau]

I have arby in this state on both of my simnet environments now, restart doesn't help. xud restart probably fixes this, but I don't want to do that now in order to debug this.

[kilrau]

Confirmed: xud restart fixes this

[kilrau]

Move this issue to the xud repo? I still have on environment up where this happens. cc @sangaman

[kilrau]

Moved this over to xud since it very much looks like an xud issue.

[kilrau]

After speaking to @raladev , we think this is what happens and it's actually an issue on arby side @ERKarl (hence moving issue back to arby repo):

• ALREADY_EXISTS is arby trying to placeorder for an order that is already in the order book
• we assume this can occur if arby somehow fails to cleanly remove all orders when e.g. restarting as part of the scheduled restarts
• when coming up again it tries to place its first order but fails because the old order from pre-restart is still there

Handling this should be easy: since arby orders have rather unique id arby should switch to replace or remove+place if an order with the same id already exists on xud