Autonat doesn't expire addresses

[nazar-pc]

I noticed that in our network we have many peers that think their local addresses are using unreachable ports, which looks something like this:

2023-11-14T05:12:40.890493Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/30533/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.890581Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/30533/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.890651Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/30533/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.890689Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/30533/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.890727Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/31657/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.890762Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/31657/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.890807Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/53250/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.890841Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/53250/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.890877Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/11115/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.890912Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/11115/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.890949Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/62453/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.890982Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/62453/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.891016Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/1653/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.891050Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/1653/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.891094Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/25653/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.891126Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using libp2p_core::transport::dummy::DummyTransport<(libp2p_identity::peer_id::PeerId, libp2p_core::muxing::boxed::StreamMuxerBox)> address=/ip4/38.242.132.116/udp/25653/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg
2023-11-14T05:12:40.891281Z TRACE tokio-runtime-worker libp2p_core::transport::choice: Attempting to dial address using subspace_networking::constructor::transport::CustomTransportWrapper<libp2p_core::transport::map::Map<libp2p_quic::transport::GenTransport<libp2p_quic::provider::tokio::Provider>, subspace_networking::constructor::transport::build_transport::{{closure}}>> address=/ip4/38.242.132.116/udp/30533/quic-v1/p2p/12D3KooWDg2oMW14keFVdiNQrRXZZUZpV1xVPR16pbvQ1gJXeYUg

As you can see many ports that are different from default port 30533 and are unreachable. I didn't understand why though, but now I think I do:

2023-11-14T15:15:14.029284Z Public address status changed. old=Private new=Public("/ip4/a.b.c.d/udp/1998/quic-v1/p2p/12D3KooWHkpEArN18NZEjNkeYSNuwP1q7aPNJqGegAAyeovsDstH")
2023-11-14T15:15:14.029314Z Confirmed external address address=/ip4/a.b.c.d/udp/1998/quic-v1/p2p/12D3KooWHkpEArN18NZEjNkeYSNuwP1q7aPNJqGegAAyeovsDstH
...
2023-11-14T15:16:54.321423Z Public address status changed. old=Public("/ip4/a.b.c.d/udp/1998/quic-v1/p2p/12D3KooWHkpEArN18NZEjNkeYSNuwP1q7aPNJqGegAAyeovsDstH") new=Private

Here node discovered external address candidate with Identify, then confirmed with Autonat, Swarm then captures ToSwarm::ExternalAddrConfirmed and does self.add_external_address().

However, when Autonat notices that address is no longer reachable it doesn't fire ToSwarm::ExternalAddrExpired and previously added external address is not removed, so these invalid addresses keep piling up. At least that is my understanding right now.

I believe Autonat should start firing ToSwarm::ExternalAddrExpired to do proper cleanup and close the loop here.

[mxinden]

I believe Autonat should start firing ToSwarm::ExternalAddrExpired to do proper cleanup and close the loop here.

Yes, good catch!

Out of curiosity, why is your node continuously changing its address?

Either way, this should be fixed.

[nazar-pc]

Out of curiosity, why is your node continuously changing its address?

My node has static IP and doesn't change address. In fact IP address is always correct, it is port that varies. I also recall seeing in code comments about ephemeral TCP ports of outgoing connections being handled carefully, my current suspicion is that something like that is not taken into consideration for UDP/QUIC, which we use as primary transport now, but I didn't look into it very closely yet.

[nazar-pc]

I looked into it a bit and it is tricky to do without negative side-effects. The thing is that when we get error response from a peer, we would have to mark all external addresses as expired, which seems too damaging in case it is an adversarial response. I guess this is another thing that'll have to wait for Antonat v2?

[guillaumemichel]

@umgefahren is this still relevant?

[umgefahren]

As it stands, we don't expire addresses. This is connected to the issue of IPv4 and IPv6. Very basically: It's way more difficult to correctly state that an address is unreachable then to say that it's reachable. The goal of the version we shipped was to avoid false positives.

However this can be shipped as a future improvement of the protocol.