Skip to content

TLS: structured certificate selection result for async certificate callbacks #838

@dnstelecom

Description

@dnstelecom

Currently async TLS certificate callbacks rely entirely on mutating SslRef.
When certificate selection fails or is rejected, the resulting TLS accept error
can be difficult to diagnose because the callback cannot explicitly communicate
its outcome.

This is a small proposal to improve diagnostics for async certificate selection.

This proposal introduces a structured result for certificate selection:

enum TlsCertificateSelection {
    Selected,
    Rejected { reason: String },
    NoSelection,
}

TlsAccept would gain an optional method:

async fn certificate_callback_result(
    &self,
    ssl: &mut TlsRef,
) -> TlsCertificateSelection

New implementations could return explicit outcomes, while existing
certificate_callback() implementations would remain supported.

Legacy callbacks mutate SslRef directly, so the TLS server handshake layer
would infer success when certificate material was installed even if the
callback returned NoSelection.

This would allow TLS accept to produce clearer diagnostics for cases such as:

  • explicit callback rejection
  • callback returning without selecting a certificate
  • callback reporting success without installing certificate material

The listener API remains backend-agnostic; certificate inspection and legacy
inference stay in the TLS server handshake implementation.

Scope

This change currently applies only to the OpenSSL/BoringSSL TLS server
implementation (openssl_derived). Other TLS backends (rustls, s2n, noop)
are unaffected because async certificate selection currently operates on
SslRef in the OpenSSL path.

If this direction looks reasonable, I can open a PR with a working implementation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions