Implement Automated Lightning Routing, Mining, Fee Estimation, and Transaction Details

[basim-12]

@todo

• Implement Automated Lightning Routing, Mining, Fee Estimation, and Transaction Details
  • Automated block mining for outgoing and incoming Bitcoin payments on regtest
  • Started Bitcoin payment confirmation monitoring automatically from the main UI
  • Automated channel confirmation mining for all Lightning channel opening flows
  • Implemented generic Lightning routing between configured nodes through Node 5 (Flamingo Hub)
  • Added automatic sender to hub and hub to recipient channel provisioning
  • Automatically funded nodes and expanded channels when liquidity was insufficient
  • Reused existing channels when they already had enough payment capacity
  • Added channel activation waits, gossip refresh, and route discovery retries
  • Removed hardcoded node lookups and used the currently selected wallet node
  • Displayed correct Lightning aliases, node IDs, descriptions, and transaction details
  • Displayed sub satoshi routing fees without rounding them to zero
  • Estimated fees before payment and replaced them with confirmed fees afterward
  • Debug and Testing
  • @output 📦 Flamingo-node Repo commitFrom Comment
  • @output 📦 Flamingo-ui Repo commitFrom Comment

[basim-12]

Tasks

• Automated block mining for outgoing and incoming Bitcoin payments on regtest - 1hr 30min
• Started Bitcoin payment confirmation monitoring automatically from the main UI - 45min
• Automated channel confirmation mining for all Lightning channel-opening flows - 1hr 15min
• Implemented generic Lightning routing between configured nodes through Node 5 (Flamingo Hub) - 3hr 30min
• Added automatic sender to hub and hub to recipient channel provisioning - 2hr 45min
• Automatically funded nodes and expanded channels when liquidity was insufficient - 2hr 30min
• Reused existing channels when they already had enough payment capacity - 1hr
• Added channel activation waits, gossip refresh, and route discovery retries - 2hr 30min
• Removed hardcoded node lookups and used the currently selected wallet node - 1hr
• Displayed correct Lightning aliases, node IDs, descriptions, and transaction details - 2hr
• Displayed sub satoshi routing fees without rounding them to zero - 45min
• Estimated fees before payment and replaced them with confirmed fees afterward - 1hr 30min
• Debug and Testing - 4hr
• Create issue and Worklog - 25min
  • @output 📦 Flamingo-node Repo commit
  • @output 📦 Flamingo-ui Repo commit

Worklog

Implement Automated Lightning Routing, Mining, Fee Estimation, and Transaction Details

Proposal

[ninabreznik]

@basim-12 Thank you for the worklog. Looks like a lot has been done in this one. Great job!!

1. While watching is I figured out I don't know exactly how we implemented the logic of detecting the nodes - do we check local host and see that is running and show those or how do we know which nodes to display. Also, do we need a way to add a new node (if someone runs a node on a remote server or another computer?). Do you have ideas about how we could manage the whole lifecycle of adding and removing nodes?

2. I noticed you manually copy the invoice (invoice created modal). Is this a habit or copy button doesn't work yet?

3. You mention that if you send a tx from node 4 to node 6 it auto creates a reverse liquidity. Can you please remind me how exactly does this work. Because in my head I am currently not in the lightning node channels and liquidity, I would really appreciate if you could write me down how exactly you implemented this logic, so I can really understand it. Maybe we could even create a single page docs to explain how this work (because when we will give users option to optimize things, they would want to understand what is the current default mechanics, so we will also for them need to provide this simple but to the point explanation). Would you be able to make a little write-up for this, please. Maybe you can make it in this style: summary paragraph and then glossary which explains things like routing, liquidity (inbound, outbound) etc but not as a theory but as theory and practice (default logic in our app) combined.

Ie. Routing: payments between lightning nodes (different users) goes through multiple hops (routing nodes) which chage micro fee for their service. In our case, all payments are by default routed through flamingo hub, which makes payments fast, relatively cheap and without any setup fees needed (when you want to open a chanel to every node you interact with, you would normally pay a setup fee, but if you route your payment through flamingo hub, you only open a channel with that node and then all the payments can go through it).

^^ This was now written ad hoc just from the top of my head, but just wanted to show you what I mean. We explain routing but also the logic and benefits of our approach.

Of course we will use chat GPT or similar to help with this document, but these ai models like to spit out some blabla text, while what we want here is little text and to the point explanation of our approach and benefit (and downsides) of using this approach so they can make informed decisions when to use the default option and when to start manually optimize things.

4. Lovely to see all the auto-mining logic in place. It certainly speeds up the dev process. What I wanted to ask is - did you write it so that it only does auto-mining for regtest? I see function is named auto_mine_regtest, so I assume you did, but just double checking.

5. What do you mean by 'opening a larger channel in the required payment direction' - ensure_directional_hub_channel

[basim-12]

@ninabreznik Thanks alot for the feedback.

1. When an invoice is decoded, we do not assume the recipient is Node 4 or Node 6 anymore.

The invoice contains the recipient’s Lightning public key. The backend:

• Decodes the invoice and reads its payee public key.
• Calls getinfo for every configured node.
• Compares each node’s public key with the invoice payee.
• Identifies the matching recipient automatically.
• Uses the currently selected account as the sender.
• Prepares the required sender -> Node 5 -> recipient route.

It also reads each node’s active host and port from getinfo, rather than hard coding the connection port.
So payments are dynamically detected and routed between any nodes already registered in LIGHTNING_DIRS. We no longer have payment logic specifically saying “Node 4 pays Node 6.”

2. Yes you're right, Copy Invoice button doesn't work right now, I already have it in my notes, will fix it in the next worklog.

3. For a payment from Node 4 to Node 6, the default route is:
   Node 4 -> Node 5 Flamingo Hub -> Node 6

Before paying, the backend decodes the invoice to identify Node 6 and the payment amount. It then checks two separate pieces of liquidity:

• Does Node 4 have enough outbound liquidity to Node 5?
• Does Node 5 have enough outbound liquidity to Node 6?

If both are available, the existing channels are reused.

If Node 4 cannot send enough to Node 5, Node 4 opens an additional channel to Node 5. If Node 5 cannot send enough to Node 6, Node 5 opens an additional channel to Node 6. Funding the Node 5->Node 6 channel gives Node 5 outbound liquidity and Node 6 inbound liquidity.

The channels themselves are bidirectional. This refers to which side currently owns enough channel balance to send the payment, not to a permanently one way channel.

After a payment, balances move across those channels. This can create useful liquidity for a later payment in the opposite direction. If Node 6 later pays Node 4, the backend checks:
Node 6 -> Node 5 -> Node 4

It reuses the shifted balances when possible, otherwise it creates the missing liquidity in that direction.

The main benefit is that users do not need a separate channel with every person they pay. They mainly connect through the Flamingo Hub, which handles routing and earns a small forwarding fee.

Yes, I can definitely make the document as you suggested.

4. Yes, automatic mining only runs on regtest.

auto_mine_regtest checks the configured Bitcoin network first. On any other network, it skips mining. Automatic node funding for channel creation is also blocked outside regtest.
For testnet or mainnet, confirmations must happen naturally.

5. “Opening a larger channel in the required payment direction” means ensuring that the node which must send at a particular hop owns enough outbound balance.

Like, for Node 4->Node 6:

Node 4 needs outbound liquidity to Node 5
Node 5 needs outbound liquidity to Node 6

ensure_directional_hub_channel checks the active channels between each pair and totals the funder’s available outbound balance.

If the balance is sufficient, it reuses those channels. If it is insufficient, the funder opens an additional, larger channel. The new channel is currently sized to at least the default channel size or roughly twice the required payment capacity, giving some room for future payments.
So “directional” refers to the required flow of liquidity for that payment, while the actual Lightning channel remains bidirectional.

[ninabreznik]

@basim-12 sorry for the delay, last week I had my parents visiting, so I was basically on holidays.

It also reads each node’s active host and port from getinfo, rather than hard coding the connection port.
So payments are dynamically detected and routed between any nodes already registered in LIGHTNING_DIRS. We no longer have payment logic specifically saying “Node 4 pays Node 6.”

Thanks for the explanation, so for node detection (so users can select them and switch between them) we use getInfo (does that show all the available nodes?). I actually try to imagine when would user run many nodes? Would that be even needed/helpful/typical usecase? Now we test using multiple nodes, but in real life, will anyone use it this way? What do you think? And if, will they maybe run some nodes locally, some over a server? And in this case I wonder how will they add/remove different nodes to Flamingo?

If Node 4 cannot send enough to Node 5, Node 4 opens an additional channel to Node 5. If Node 5 cannot send enough to Node 6, Node 5 > opens an additional channel to Node 6. Funding the Node 5->Node 6 channel gives Node 5 outbound liquidity and Node 6 inbound liquidity.

Now we can do this because we control all the nodes, but I wonder what will happen if Node 4 and Node 6 are different users. Will we prompt them to Confirm opening new channels? The question is if Node 4 wants to send to Node 6, and some other nodes further down the line don't have enough liquidity, does it mean their payment will be on halt until all the Nodes (users) open required channels (aka fix their liquidity)? Like, how will this look like in the real world with different users?

[basim-12]

@ninabreznik Apologies, was out of town too

1. getinfo does not show all available nodes. It only gives information about one node that Flamingo already knows how to contact.

Currently, Node 4, Node 5, and Node 6 are registered in the backend. Flamingo calls getinfo for each configured node to check whether it is online and read its public key, alias, host, and port.

In real life, most users will probably run only one node. Multiple nodes would mainly be useful for advanced cases, for example:

• Separate personal and business nodes
• Multiple stores or companies
• One local node and one hosted server node

So the normal experience should focus on one node, with multi node support available as an advanced option.

To support this properly, we will eventually need an Add Node screen. The user would enter the local or remote connection details, Flamingo would verify the node using getinfo, and then save it in a node registry. Nodes could then be renamed, disabled, or removed.

Remote nodes would need a secure connection method, such as an auth Flamingo agent, VPN, or SSH tunnel. Removing a node from Flamingo should only remove the connection.

2. Yes, the current automatic channel setup works because we control all nodes and funds in regtest. With real users, Flamingo cannot spend their funds or open funded channels from their nodes without permission.

From what I researched, For a real Node 4 -> Node 6 payment:

• If Node 4 needs a channel to the Flamingo Hub, Node 4 is paying the on chain fee and locking funds. We should show the cost and ask Node 4 for confirmation.
• If Node 6 needs inbound liquidity, the Flamingo Hub can open a channel to Node 6 using the hub’s own funds. Node 6 would normally not pay the opening transaction directly, but this would be a Flamingo liquidity service with its own limits or fee.
• Flamingo cannot make Node 6 fund a channel without Node 6’s approval.

Other routing nodes further down the network are not asked to open channels for that payment. Core Lightning first tries other routes and may split the payment across multiple routes.

If no route has enough liquidity, the payment does not wait for every user to fix their channels. It eventually fails with an insufficient liquidity or no route error. The user can then retry, choose another route, rebalance, or request liquidity.

Our hub model simplifies this by managing only:
Sender -> Flamingo Hub -> Recipient
For users registered with Flamingo, we can monitor and offer liquidity on both sides. For external recipients, we cannot automatically manage every channel further along the network, we must use the routes that already exist.

This way, In production, the flow would be:

1. Reuse existing routes first.
2. Try alternative routes if the first one lacks liquidity.
3. Ask the sender before opening a channel with their funds.
4. Offer recipient liquidity through the Flamingo Hub when available.
5. Show a clear failure if no usable route can be prepared.

[ninabreznik]

Hi, @basim-12, thank you

1. Yes, that's what I was imagining too. So, some sort of screen and logic to add/remove a node and I agree, focus on one node but ability to support more. And running a node is separate from this where we just connect/disconnect basically.

If Node 6 needs inbound liquidity, the Flamingo Hub can open a channel to Node 6 using the hub’s own funds. Node 6 would normally not >pay the opening transaction directly, but this would be a Flamingo liquidity service with its own limits or fee.

If during the process you have some more ideas about how to set up the service and fees, please make notes so later when we get to this part, we can use all the valuable ideas/suggestions :)

This way, In production, the flow would be:

Reuse existing routes first.
Try alternative routes if the first one lacks liquidity.
Ask the sender before opening a channel with their funds.
Offer recipient liquidity through the Flamingo Hub when available.
Show a clear failure if no usable route can be prepared.

This sounds like a realistic and feasible approach. We will later just need to define the Flamingo Hub service and then offer it to users just-in-time (so next to routing we will also offer liquidity services and maybe more, let's just gather ideas in the notes. Should we make a Flamingo hub document and just trow in all the ideas and knowledge we collect along the way? Could you do that (nothing fancy, jsut to have one spot where we collect this so we make it easier for ourselves later when we start solving this part).

[basim-12]

@ninabreznik Thanks, so I'll add the Flamingo_hub_service_note in Flamingo_node/docs, and add the current details as well as future plans like 'payment routing', 'inbound liquidity', 'just-in-time liquidity', 'channel monitoring' etc.
I have also created the previous document explaining the default mechanics of Flamingo hub for users, will post both in the next worklog.

[ninabreznik]

@basim-12 awesome, thank you 🙏