Skip to content

Conversation

rodrigo-o
Copy link
Collaborator

@rodrigo-o rodrigo-o commented Oct 10, 2025

Motivation

We have some WARN logs spamming the node while on Sepolia or Mainnet, we want to remove them given taht they don't require attention.

Description

This PR move the most common of these errors (the one that are spammed during most of mainnet executions) to debug:

  • Attempted to replace a pooled transaction with an underpriced transaction
  • Nonce for account too low
  • Account does not have enough balance to cover the tx cost

We could move them all to debug instead but wanted some feedback first. It also adds information to the peer logs related to the client. We checked that these logs doesn't come from the same source of clients, here is a summary of a node in mainnet running for a couple of hours

 Qty.         Client                  Mempool Warned error
 398542  client=Geth        Attempted to replace a pooled transaction with an underpriced transaction
  32357  client=reth        Attempted to replace a pooled transaction with an underpriced transaction
  26634  client=ethrex      Nonce for account too low
  24225  client=Geth        Nonce for account too low
  24179  client=besu        Attempted to replace a pooled transaction with an underpriced transaction
   8028  client=Nethermind  Attempted to replace a pooled transaction with an underpriced transaction
   2625  client=ethrex      Attempted to replace a pooled transaction with an underpriced transaction
   2241  client=reth        Nonce for account too low
   1812  client=r           Attempted to replace a pooled transaction with an underpriced transaction
   1279  client=Geth        Account does not have enough balance to cover the tx cost
    356  client=besu        Account does not have enough balance to cover the tx cost
    276  client=besu        Nonce for account too low
    253  client=Nethermind  Nonce for account too low
    172  client=erigon      Attempted to replace a pooled transaction with an underpriced transaction
    136  client=reth        Account does not have enough balance to cover the tx cost
     32  client=erigon      Nonce for account too low
     24  client=Nethermind  Account does not have enough balance to cover the tx cost
     15  client=ethrex      Account does not have enough balance to cover the tx cost
      9  client=Nethermind  Transaction blob base fee too low
      6  client=r           Account does not have enough balance to cover the tx cost
      1  client=erigon      Transaction intrinsic gas overflow

This is how a typical mainnet log looked like before:

2025-10-10T21:26:44.430287Z  WARN ethrex_p2p::rlpx::utils: erigon/[0xbf33…8627(49.12.130.92:30010)]: Error adding transaction: Nonce for account too low                                                                                                                                                                                                                    
2025-10-10T21:26:44.430605Z  WARN ethrex_p2p::rlpx::utils: erigon/[0xbf33…8627(49.12.130.92:30010)]: Error adding transaction: Nonce for account too low                                                                                                                                                                                                                    
2025-10-10T21:26:44.430889Z  WARN ethrex_p2p::rlpx::utils: erigon/[0xbf33…8627(49.12.130.92:30010)]: Error adding transaction: Nonce for account too low                                                                                                                                                                                                                    
2025-10-10T21:26:44.431204Z  WARN ethrex_p2p::rlpx::utils: erigon/[0xbf33…8627(49.12.130.92:30010)]: Error adding transaction: Nonce for account too low                                                                                                                                                                                                                    
2025-10-10T21:26:44.431671Z  WARN ethrex_p2p::rlpx::utils: erigon/[0xbf33…8627(49.12.130.92:30010)]: Error adding transaction: Nonce for account too low                                                                                                                                                                                                                    
2025-10-10T21:26:44.557946Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x9ef4…bfc0(208.91.111.153:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                    
2025-10-10T21:26:44.605518Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x8920…2618(54.209.213.104:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                    
2025-10-10T21:26:44.629493Z  WARN ethrex_p2p::rlpx::utils: Geth/[0xd464…a086(3.238.31.12:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                       
2025-10-10T21:26:44.652605Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x8cb3…bf1f(202.90.244.19:30303)]: Error adding transaction common: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                              
2025-10-10T21:26:44.761911Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x0965…3f16(136.49.53.169:30303)]: Error adding transaction: Nonce for account too low                                                                                                                                                                                                                     
2025-10-10T21:26:44.765399Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Nonce for account too low                                                                                                                                                                                                                     
2025-10-10T21:26:44.768167Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                     
2025-10-10T21:26:44.770387Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                     
2025-10-10T21:26:44.770697Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                     
2025-10-10T21:26:44.771131Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Nonce for account too low                                                                                                                                                                                                                     
2025-10-10T21:26:44.771583Z  WARN ethrex_p2p::rlpx::utils: Geth/[0xd54b…0100(99.246.19.166:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                     
2025-10-10T21:26:44.773184Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Nonce for account too low                                                                                                                                                                                                                     
2025-10-10T21:26:44.775012Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                     
2025-10-10T21:26:44.776434Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1cba…1319(67.220.95.8:50122)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                       
2025-10-10T21:26:44.777130Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                     
2025-10-10T21:26:44.777380Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                     
2025-10-10T21:26:44.777695Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                     
2025-10-10T21:26:44.777992Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                     
2025-10-10T21:26:44.778375Z  WARN ethrex_p2p::rlpx::utils: Geth/[0x1575…a597(5.181.157.139:30303)]: Error adding transaction: Attempted to replace a pooled transaction with an underpriced transaction                                                                                                                                                                                                          

We may want to further investigate if they are a constant influx of new transactions that we need to discard or if we are recieving the same transactions multiple times, for that I created #4842

@rodrigo-o rodrigo-o changed the title Adding client name to peer logs chore(l1): move WARN logs to DEBUG when a tx is not able to be added to the mempool Oct 10, 2025
@github-actions github-actions bot added the L1 Ethereum client label Oct 10, 2025
Comment on lines 807 to 825
match e {
// Some of the more common mempool errors are logged at debug level, we might want to move all of them to debug in the future
ethrex_blockchain::error::MempoolError::NonceTooLow
| ethrex_blockchain::error::MempoolError::NotEnoughBalance
| ethrex_blockchain::error::MempoolError::UnderpricedReplacement => {
log_peer_debug(
&state.node,
&format!("Error adding transaction common: {e}"),
);
continue;
}
_ => {
log_peer_warn(
&state.node,
&format!("Error adding transaction: {e}"),
);
continue;
}
}
Copy link
Collaborator Author

@rodrigo-o rodrigo-o Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we agree we can move them all to debug, otherwise this tackles 99% of all logs that are seen in the current sepolia and mainnet executions

PS: the errors aren't matched rn, Im looking into it

Copy link

Lines of code report

Total lines added: 28
Total lines removed: 0
Total lines changed: 28

Detailed view
+--------------------------------------------------------+-------+------+
| File                                                   | Lines | Diff |
+--------------------------------------------------------+-------+------+
| ethrex/crates/networking/p2p/rlpx/connection/server.rs | 878   | +16  |
+--------------------------------------------------------+-------+------+
| ethrex/crates/networking/p2p/types.rs                  | 508   | +12  |
+--------------------------------------------------------+-------+------+

@rodrigo-o rodrigo-o marked this pull request as ready for review October 10, 2025 21:59
@rodrigo-o rodrigo-o requested a review from a team as a code owner October 10, 2025 21:59
@Copilot Copilot AI review requested due to automatic review settings October 10, 2025 21:59
@ethrex-project-sync ethrex-project-sync bot moved this to In Review in ethrex_l1 Oct 10, 2025
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR addresses log spam by downgrading common mempool-related transaction errors from WARN to DEBUG level. The changes reduce noise in production logs when transactions fail to be added to the mempool for common, expected reasons.

  • Move three specific mempool errors (NonceTooLow, NotEnoughBalance, UnderpricedReplacement) from WARN to DEBUG logging
  • Add client name information to peer logging functions for better debugging context
  • Implement client_name() method to extract client information from node version strings

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
crates/networking/p2p/types.rs Adds client_name() method to extract client name from node version
crates/networking/p2p/rlpx/utils.rs Updates logging functions to include client name in log format
crates/networking/p2p/rlpx/connection/server.rs Changes specific mempool errors from WARN to DEBUG level logging

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

L1 Ethereum client

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

1 participant