Skip to content

Comments

feat(buffer-handler): add buffering support for external loggers#7994

Open
acascell wants to merge 3 commits intoaws-powertools:developfrom
acascell:feat/add-buffering-support-ext-loggers
Open

feat(buffer-handler): add buffering support for external loggers#7994
acascell wants to merge 3 commits intoaws-powertools:developfrom
acascell:feat/add-buffering-support-ext-loggers

Conversation

@acascell
Copy link

@acascell acascell commented Feb 14, 2026

Issue number: closes #7918

Summary

This PR implements log buffering support for external libraries when using AWS Lambda Powertools Logger with buffering enabled.

Changes

  1. New BufferingHandler class (aws_lambda_powertools/logging/buffer/handler.py):

    • Custom logging handler that intercepts logs from external libraries extending logging.Handler
    • Override and extends emit() method delegating to the source Logger's _add_log_record_to_buffer() function
    • Shares the same buffer cache and tracer_id as the Powertools Logger
    • Ensures external library logs are buffered and flushed together with application logs
  2. Enhanced copy_config_to_registered_loggers() function (aws_lambda_powertools/logging/utils.py):

    • Added new include_buffering parameter (default: False)
    • Maintains backward compatibility - existing behavior unchanged when parameter is False
  3. Updated configuration logic (-):

    • adds BufferingHandler on include_buffering flag
  4. Included unit and functional tests

  • tests/functional/logger/required_dependencies/test_logger_utils.py
  • tests/unit/logger/required_dependencies/test_logger_buffer_handler.py

User Experience

Before this change:
When using Logger with buffering enabled, external library logs (botocore, mangum, etc.) were NOT buffered - they appeared immediately in CloudWatch, defeating the purpose of buffering:

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import utils
from aws_lambda_powertools.logging.config import LoggerBufferConfig

# Enable buffering on Powertools Logger
buffer_config = LoggerBufferConfig(
    max_bytes=20480,
    buffer_at_verbosity="DEBUG",
    flush_on_error_log=True
)
LOGGER = Logger(service="payment", buffer_config=buffer_config)

# Configure external loggers
utils.copy_config_to_registered_loggers(source_logger=LOGGER)

# Get external logger
botocore_logger = logging.getLogger("botocore")

def handler(event, context):
    LOGGER.debug("Processing payment")              # Buffered
    LOGGER.info("Validating request")               # Buffered
    botocore_logger.info("Making API call")         # NOT buffered (appears immediately)
    botocore_logger.debug("Request headers set")    # NOT buffered (appears immediately)
    LOGGER.error("Payment failed")                  # Flushes only LOGGER's logs

Result: External library logs appeared immediately in CloudWatch, while application logs were buffered.

After this change:
External library logs are now buffered alongside application logs when include_buffering=True:

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import utils
from aws_lambda_powertools.logging.config import LoggerBufferConfig

# Enable buffering on Powertools Logger
buffer_config = LoggerBufferConfig(
    max_bytes=20480,
    buffer_at_verbosity="DEBUG",
    flush_on_error_log=True
)
LOGGER = Logger(service="payment", buffer_config=buffer_config)

# Configure external loggers WITH buffering
utils.copy_config_to_registered_loggers(
    source_logger=LOGGER,
    include_buffering=True  # Enable buffering for external loggers
)

# Get external logger
botocore_logger = logging.getLogger("botocore")

def handler(event, context):
    LOGGER.debug("Processing payment")              # Buffered
    LOGGER.info("Validating request")               # Buffered
    botocore_logger.info("Making API call")         # Buffered
    botocore_logger.debug("Request headers set")    # Buffered
    LOGGER.error("Payment failed")                  # Flushes ALL logs (app + external)

Result: All logs (application + external libraries) are buffered together and flushed as a unit


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

@acascell acascell requested a review from a team as a code owner February 14, 2026 22:56
@acascell acascell requested a review from ConnorKirk February 14, 2026 22:56
@boring-cyborg
Copy link

boring-cyborg bot commented Feb 14, 2026

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@pull-request-size pull-request-size bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 14, 2026
@sonarqubecloud
Copy link

@acascell acascell changed the title feat(buffer-handler): add buffering support ext loggers feat(buffer-handler): add buffering support for external loggers Feb 15, 2026
@dreamorosi dreamorosi requested review from leandrodamascena and removed request for ConnorKirk February 25, 2026 11:42
@dreamorosi
Copy link
Contributor

Hi @acascell - thanks for the PR and sorry for the delayed review.

@leandrodamascena will most likely review this next week once he's back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logger size/L Denotes a PR that changes 100-499 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: introduce handler-based buffering for external python loggers in powertools Logger

2 participants