Fluentd opensearch plugin does not send full certificate chain

[shubher13]

Problem description

Fluentd cannot communicate with OpenSearch because, during the SSL handshake, the Fluentd OpenSearch plugin sends only its microservice certificate instead of the full certificate chain, including the intermediate CA. However, when using the same certificate with Curl to communicate with OpenSearch, it works fine because Curl sends the complete certificate chain.

Steps to replicate

1. Create issuer with intermediate CA

#!/bin/bash
mkdir -p out
function generate_root_certificate() {
  root_CA_config="[req]
        default_bits = 4096
        prompt = no
        default_md = sha256
        distinguished_name = dn
        [dn]
        CN = Root CA
        O = MyOrg
        [v3_ca]
        basicConstraints = CA:TRUE
        keyUsage = keyCertSign, cRLSign
        authorityKeyIdentifier = keyid,issuer:always"
  openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out ./out/root_ca.key
  openssl req -x509 -new -key ./out/root_ca.key \
    -out ./out/root_ca.crt \
    -days 7300 \
    -config /dev/stdin <<< "$root_CA_config" \
    -extensions v3_ca
}
function generate_certificates() {
  openssl genrsa -out "./out/$1.key" 4096
  siteConfig="[req]
        default_bits = 4096
        prompt = no
        default_md = sha256
        distinguished_name = dn
        [dn]
        CN = $1 Intermediate CA
        O = MyOrg
        [v3_ca]
        basicConstraints = CA:TRUE
        keyUsage = keyCertSign, cRLSign
        authorityKeyIdentifier = keyid,issuer:always"
  openssl req -new -key "./out/$1.key" \
    -out "./out/$1.csr" \
    -config /dev/stdin <<< "$siteConfig" \
    -days 3650 \
    -extensions v3_ca
  openssl x509 -req -in "./out/$1.csr" \
    -CA ./out/root_ca.crt -CAkey ./out/root_ca.key \
    -CAcreateserial -out "./out/$1.crt" \
    -days 3650 \
    -extensions v3_ca \
    -extfile /dev/stdin <<< "$siteConfig"
}
generate_root_certificate
generate_certificates site-a
cat ./out/site-a.crt ./out/root_ca.crt > combined-site-a.crt

2. Create Secret for the ClusterIssuer

kubectl create secret tls bug-report-ca-secret --cert=./combined-site-a.crt --key=./out/site-a.key -n ncms 

3. Create ClusterIssuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: bug-report-ca-issuer
spec:
  ca:
    secretName: bug-report-ca-secret

 kubectl apply -f clusterIssuer.yaml
clusterissuer.cert-manager.io/bug-report-ca-issuer created
 
[abc@machine user]$ k get clusterissuer
NAME                   READY   AGE
bug-report-ca-issuer   True    39s

4. Replace all certificates old ClusterIssuer to new ClusterIssuer

CERTIFICATES=$(kubectl get certificates  -o jsonpath='{.items[*].metadata.name}')
NEW_ISSUER=bug-report-ca-issuer
for CERT in $CERTIFICATES; do   kubectl patch certificate $CERT -n $NAMESPACE --type=json -p="[{'op': 'replace', 'path': '/spec/issuerRef/name', 'value': '$NEW_ISSUER'}]";   echo "Updated $CERT to use $NEW_ISSUER"; done

5. Delete secrets to enforce new one generation

kubectl get certificates | awk 'NR > 1' | awk '{print $3}' | xargs kubectl delete secret

6. switch opensearch to print SSL handshake logs (Optional)
   Edit configmap-

kubectl edit cm bp23-btel-belk-elasticsearch-jvmopt

and add -Djavax.net.debug=ssl:handshake
Provide example config and message

Plugin configuration

<match org.logging.**>
     @type copy
     <store>
       @type opensearch
       host bp23-opensearch.test_ns.svc.cluster.local
       port 9200
       resurrect_after 5s
       id_key _hash
       type_name fluentd
       time_key time
       utc_index true
       time_key_exclude_timestamp true
       logstash_format true
       logstash_prefix fluentd-${tag[2]}-${tag[3]}
       reload_connections false
       reconnect_on_error true
       reload_on_failure true
       bulk_message_request_threshold 8MB
       request_timeout 30s
       ca_file /etc/td-agent/sharedMountFiles/isroot_cert
       client_cert /etc/td-agent/oscerts/tls.crt
       client_key /etc/td-agent/oscerts/tls.key
       scheme https
       ssl_verify true
       ssl_version TLSv1_2
       suppress_type_name true
       <buffer tag, time, namespace, type>
         @type file
         path /var/log/td-agent/opensearch-buffer-test_ns/org.logging.all.all
         flush_mode interval
         flush_interval 30s
         timekey 3600
         retry_forever true
         chunk_limit_size 8MB
         retry_max_interval 5s
         overflow_action block
         total_limit_size 1024m
       </buffer>
     </store>
   </match>
   # Suppress all non matching tags at the end of this label
   <match **>

Logs

The handshake begins and the server presents its full certificate

opensearch µservice certificate
intermediate certificate

as seen in

Log1

"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.055 UTC|null:-1|Produced ServerHello handshake message ("}}
"message":""ServerHello": {"}}
"message":" "server version" : "TLSv1.2","}}
"message":" "random" : "60 4C B6 F3 1C 20 A1 44 11 41 0B F9 9B 7D 95 64 51 2E B2 2F 3D 5C 4F AE 50 43 F7 35 3D 2D 19 D6","}}
"message":" "session id" : "","}}
"message":" "cipher suite" : "TLS_AES_256_GCM_SHA384(0x1302)","}}
"message":" "compression methods" : "00","}}
"message":" "extensions" : ["}}
"message":" "supported_versions (43)": {"}}
"message":" "selected version": [TLSv1.3]"}}
"message":" },"}}
"message":" "key_share (51)": {"}}
"message":" "server_share": {"}}
"message":" "named group": x25519"}}
"message":" "key_exchange": {"}}
"message":" 0000: 84 CA 67 73 D6 EE CB 41 FB 6F D3 93 4C DD FD 33 ..gs...A.o..L..3"}}
"message":" 0010: 24 87 85 54 91 DB BC ED 17 B2 24 3C 69 81 20 31 $..T......$<i. 1"}}
"message":" }"}}
"message":" },"}}
"message":" }"}}
"message":" ]"}}
"message":"}"}}
"message":")"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.056 UTC|null:-1|KeyLimit read side: algorithm = AES/GCM/NOPADDING:KEYUPDATE"}}
"message":"countdown value = 137438953472"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.057 UTC|null:-1|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE"}}
"message":"countdown value = 137438953472"}}
"message":"javax.net.ssl|ALL|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.057 UTC|null:-1|No expected server name indication response"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.057 UTC|null:-1|Ignore, context unavailable extension: server_name"}}
"message":"javax.net.ssl|ALL|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.057 UTC|null:-1|Ignore unavailable max_fragment_length extension"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.057 UTC|null:-1|Ignore, context unavailable extension: max_fragment_length"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.057 UTC|null:-1|Ignore unavailable extension: application_layer_protocol_negotiation"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.057 UTC|null:-1|Ignore, context unavailable extension: application_layer_protocol_negotiation"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.057 UTC|null:-1|Produced EncryptedExtensions message ("}}
"message":""EncryptedExtensions": ["}}
"message":" "supported_groups (10)": {"}}
"message":" "versions": [x25519, secp256r1, secp384r1, secp521r1, x448, ffdhe2048, ffdhe3072, ffdhe4096, ffdhe6144, ffdhe8192]"}}
"message":" }"}}
"message":"]"}}
"message":")"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.058 UTC|null:-1|Produced CertificateRequest message ("}}
"message":""CertificateRequest": {"}}
"message":" "certificate_request_context": "","}}
"message":" "extensions": ["}}
"message":" "signature_algorithms (13)": {"}}
"message":" "signature schemes": [ed25519, ed448, ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp521r1_sha512, rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pss_pss_sha256, rsa_pss_pss_sha384, rsa_pss_pss_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, ecdsa_sha1, rsa_pkcs1_sha1]"}}
"message":" },"}}
"message":" "signature_algorithms_cert (50)": {"}}
"message":" "signature schemes": [ed25519, ed448, ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp521r1_sha512, rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pss_pss_sha256, rsa_pss_pss_sha384, rsa_pss_pss_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, ecdsa_sha1, rsa_pkcs1_sha1]"}}
"message":" },"}}
"message":" "certificate_authorities (47)": {"}}
"message":" "certificate authorities": ["}}
"message":" O=MyOrg, CN=Root CA]"}}
"message":" }"}}
"message":" ]"}}
"message":"}"}}
"message":")"}}
"message":"javax.net.ssl|ALL|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.058 UTC|null:-1|No X.509 cert selected for EC"}}
"message":"javax.net.ssl|WARNING|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.058 UTC|null:-1|Unavailable authentication scheme: ecdsa_secp256r1_sha256"}}
"message":"javax.net.ssl|ALL|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.059 UTC|null:-1|No X.509 cert selected for EC"}}
"message":"javax.net.ssl|WARNING|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.059 UTC|null:-1|Unavailable authentication scheme: ecdsa_secp384r1_sha384"}}
"message":"javax.net.ssl|ALL|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.059 UTC|null:-1|No X.509 cert selected for EC"}}
"message":"javax.net.ssl|WARNING|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.059 UTC|null:-1|Unavailable authentication scheme: ecdsa_secp521r1_sha512"}}
"message":"javax.net.ssl|WARNING|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.059 UTC|null:-1|Unsupported authentication scheme: ed25519"}}
"message":"javax.net.ssl|WARNING|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.059 UTC|null:-1|Unsupported authentication scheme: ed448"}}
"message":"javax.net.ssl|ALL|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.059 UTC|null:-1|No X.509 cert selected for RSASSA-PSS"}}
"message":"javax.net.ssl|WARNING|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.059 UTC|null:-1|Unavailable authentication scheme: rsa_pss_pss_sha256"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.059 UTC|null:-1|Staping disabled or is a resumed session"}}
"message":"javax.net.ssl|ALL|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.060 UTC|null:-1|Stapling is disabled for this connection"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.060 UTC|null:-1|Ignore, context unavailable extension: status_request"}}
"message":"javax.net.ssl|ALL|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.060 UTC|null:-1|Stapling is disabled for this connection"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.060 UTC|null:-1|Ignore, context unavailable extension: status_request"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.061 UTC|null:-1|Produced server Certificate message ("}}
"message":""Certificate": {"}}
"message":" "certificate_request_context": "","}}
"message":" "certificate_list": [ "}}
"message":" {"}}
"message":" "certificate" : {"}}
"message":" "version" : "v3","}}
"message":" "serial number" : "00 AB 3B 3E 62 81 9A E7 CF 73 D8 B7 EA A6 E9 3E 72","}}
"message":" "signature algorithm": "SHA256withRSA","}}
"message":" "issuer" : "O=MyOrg, CN=site-a Intermediate CA","}}
"message":" "not before" : "2024-04-23 12:01:58.000 UTC","}}
"message":" "not after" : "2025-04-23 12:01:58.000 UTC","}}
"message":" "subject" : "CN=bp23-elasticsearch, O=Company","}}
"message":" "subject public key" : "RSA","}}
"message":" "extensions" : ["}}
"message":" {"}}
"message":" ObjectId: 2.5.29.19 Criticality=true"}}
"message":" BasicConstraints:["}}
"message":" CA:false"}}
"message":" PathLen: undefined"}}
"message":" ]"}}
"message":" },"}}
"message":" {"}}
"message":" ObjectId: 2.5.29.37 Criticality=false"}}
"message":" ExtendedKeyUsages ["}}
"message":" serverAuth"}}
"message":" clientAuth"}}
"message":" ]"}}
"message":" },"}}
"message":" {"}}
"message":" ObjectId: 2.5.29.17 Criticality=false"}}
"message":" SubjectAlternativeName ["}}
"message":" DNSName: bp23-elasticsearch.test_ns"}}
"message":" DNSName: bp23-elasticsearch.test_ns.svc.cluster.local"}}
"message":" ]"}}
"message":" }"}}
"message":" ]}"}}
"message":" "extensions": {"}}
"message":" "}}
"message":" }"}}
"message":" },"}}
"message":" {"}}
"message":" "certificate" : {"}}
"message":" "version" : "v3","}}
"message":" "serial number" : "00 AE E0 F5 8E D2 2E 02 2E","}}
"message":" "signature algorithm": "SHA256withRSA","}}
"message":" "issuer" : "O=MyOrg, CN=Root CA","}}
"message":" "not before" : "2024-04-23 11:50:13.000 UTC","}}
"message":" "not after" : "2034-04-21 11:50:13.000 UTC","}}
"message":" "subject" : "O=MyOrg, CN=site-a Intermediate CA","}}
"message":" "subject public key" : "RSA","}}
"message":" "extensions" : ["}}
"message":" {"}}
"message":" ObjectId: 2.5.29.35 Criticality=false"}}
"message":" AuthorityKeyIdentifier ["}}
"message":" [O=MyOrg, CN=Root CA]"}}
"message":" SerialNumber: [ b1cd36c9 82249830]"}}
"message":" ]"}}
"message":" },"}}
"message":" {"}}
"message":" ObjectId: 2.5.29.19 Criticality=false"}}
"message":" BasicConstraints:["}}
"message":" CA:true"}}
"message":" PathLen:2147483647"}}
"message":" ]"}}
"message":" },"}}
"message":" {"}}
"message":" ObjectId: 2.5.29.15 Criticality=false"}}
"message":" KeyUsage ["}}
"message":" Key_CertSign"}}
"message":" Crl_Sign"}}
"message":" ]"}}
"message":" }"}}
"message":" ]}"}}
"message":" "extensions": {"}}
"message":" "}}
"message":" }"}}
"message":" },"}}
"message":"]"}}
"message":"}"}}
"message":")"}}

but the client presents only its fluentd µservice certificate
as in

Log2

UTC|null:-1|Produced server Finished handshake message ("}}
"message":""Finished": {"}}
"message":" "verify data": {"}}
"message":" 0000: A9 31 B9 E8 1C 38 2B E4 1A 95 A1 E9 3B 20 C0 4F .1...8+.....; .O"}}
"message":" 0010: C9 AF 4F 61 5A 3E FA 53 4A BB 8E E7 7E C6 87 68 ..OaZ>.SJ......h"}}
"message":" 0020: 31 1A 9C 0E 64 6C 58 75 DC 20 20 45 55 8A 2B 17 1...dlXu. EU.+."}}
"message":" }'}"}}
"message":")"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.066 UTC|null:-1|KeyLimit write side: algorithm = AES/GCM/NOPADDING:KEYUPDATE"}}
"message":"countdown value = 137438953472"}}
"message":"javax.net.ssl|DEBUG|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.070 UTC|null:-1|Consuming client Certificate handshake message ("}}
"message":""Certificate": {"}}
"message":" "certificate_request_context": "","}}
"message":" "certificate_list": [ "}}
"message":" {"}}
"message":" "certificate" : {"}}
"message":" "version" : "v3","}}
"message":" "serial number" : "00 EB F1 0F 16 01 05 4B B0 85 2B 44 18 61 40 66 A4","}}
"message":" "signature algorithm": "SHA256withRSA","}}
"message":" "issuer" : "O=MyOrg, CN=site-a Intermediate CA","}}
"message":" "not before" : "2024-04-23 12:02:10.000 UTC","}}
"message":" "not after" : "2025-04-23 12:02:10.000 UTC","}}
"message":" "subject" : "CN=bp23-fluentd, O=Company","}}
"message":" "subject public key" : "RSA","}}
"message":" "extensions" : ["}}
"message":" {"}}
"message":" ObjectId: 2.5.29.19 Criticality=true"}}
"message":" BasicConstraints:["}}
"message":" CA:false"}}
"message":" PathLen: undefined"}}
"message":" ]"}}
"message":" },"}}
"message":" {"}}
"message":" ObjectId: 2.5.29.37 Criticality=false"}}
"message":" ExtendedKeyUsages ["}}
"message":" serverAuth"}}
"message":" clientAuth"}}
"message":" ]"}}
"message":" },"}}
"message":" {"}}
"message":" ObjectId: 2.5.29.17 Criticality=false"}}
"message":" SubjectAlternativeName ["}}
"message":" DNSName: bp23-fluentd.test_ns.svc.cluster.local"}}
"message":" ]"}}
"message":" }"}}
"message":" ]}"}}
"message":" "extensions": {"}}
"message":" "}}
"message":" }"}}
"message":" },"}}
"message":"]"}}
"message":"}"}}
"message":")"}}
"message":"javax.net.ssl|ERROR|15|opensearch[bp23-""-xyz-opensearch-client-654799dd7f-gwn5v][transport_worker][T#1]|2024-04-25 09:22:40.072 UTC|null:-1|Fatal (CERTIFICATE_UNKNOWN): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target ("}}
"message":""throwable" : {"}}
"message":" sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}}

Expected Behavior or What you need to ask

Plugin should be able to provide full certificate chain.
...

Using Fluentd and OpenSearch plugin versions

• Bare Metal or within Docker or Kubernetes or others?
  deployed in Kubernetes

• Fluentd v1.0 or later

  • paste result of fluentd --version or td-agent --version

   bash-4.4# /usr/sbin/td-agent --version
   td-agent 4.4.2 fluentd 1.15.3 (e89092ce1132a933c12bb23fe8c9323c07ca81f5)

• OpenSearch plugin version

  • paste boot log of fluentd or td-agent
    Boot log

    [abc@machine ~]$ kubectl logs fluentd-daemonset-8j6nj -n st -c mysidecar ### Thu May 16 07:42:04 UTC 2024 2024-05-15 08:27:38 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil {"time":"2024-05-15T08:27:38+0000","level":"info","message":"parsing config file is succeeded path=\"/etc/td-agent/td-agent.conf\""} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-amqp' version '0.14.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-brevity-control' version '0.1.1'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-calyptia-monitoring' version '0.1.3'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-clog' version '0.1.3'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-concat' version '2.5.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-cvea-log' version '0.0.3'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-elasticsearch' version '5.2.4'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-flowcounter-simple' version '0.1.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-forest' version '0.3.3'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-genhashvalue' version '1.1'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-grafana-loki' version '1.2.20'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-grok-parser' version '2.6.2'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-kafka' version '0.18.1'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-kubernetes_metadata_filter' version '3.2.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-metrics-cmetrics' version '0.1.2'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-multi-format-parser' version '1.0.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-opensearch' version '1.0.8'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-out-http' version '1.3.4'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-parser-cri' version '0.1.1'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-postgres' version '0.1.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-prometheus' version '2.0.300001'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-record-modifier' version '2.1.1'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-remote_syslog' version '1.1.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-route' version '1.0.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-s3' version '1.7.2'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-sd-dns' version '0.1.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-splunk-hec' version '1.3.2'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-systemd' version '1.0.5'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-td' version '1.2.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-utmpx' version '0.5.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluent-plugin-webhdfs' version '1.5.0'"} {"time":"2024-05-15T08:27:40+0000","level":"info","message":"gem 'fluentd' version '1.15.3'"} {"time":"2024-05-15T08:27:44+0000","level":"warn","message":"define to capture fluentd logs in top level is deprecated. Use instead"} {"time":"2024-05-15T08:27:44+0000","level":"info","message":"using configuration file: \n \n \n format json\n time_format \"%Y-%m-%dT%H:%M:%S%z\"\n \n \n \n @type tail\n path \"/tmp/mainContainerLogs/*.log\"\n pos_file \"/tmp/td-agent.pos\"\n read_from_head true\n tag \"fluentd-main-container-logs\"\n \n @type \"json\"\n time_key \"time\"\n time_format \"%iso8601\"\n keep_time_key true\n unmatched_lines \n time_type string\n \n \n \n @type record_transformer\n enable_ruby true\n renew_record true\n remove_keys $.extension.time,$.extension.message,$.extension.level\n \n log ${ { message: record[\"message\"] } }\n extension ${require 'json';record.merge(JSON.parse(ENV[\"EXTENSION_FIELDS\"]))}\n type log\n level ${record.has_key?(\"level\") ? record[\"level\"]: \"unavailable\" }\n timezone ${ ENV[\"TZ\"] }\n system ${ ENV[\"SYSTEM\"] }\n systemid ${ ENV[\"SYSTEMID\"] }\n host ${ ENV[\"HOSTNAME\"]}.${ ENV[\"NAMESPACE\"] || '' }\n time ${record.has_key?(\"time\") ? record[\"time\"]: time.strftime('%Y-%m-%dT%H:%M:%S%z') }\n \n \n \n @type record_modifier\n enable_ruby true\n remove_keys \"dummy\"\n \n dummy ${require 'json';if record[\"extension\"].empty?; record.delete(\"extension\"); end}\n \n \n \n @type copy\n \n @type \"stdout\"\n \n @type \"json\"\n \n \n \n"} {"time":"2024-05-15T08:27:44+0000","level":"info","message":"starting fluentd-1.15.3 pid=10 ruby=\"2.7.6\""} {"time":"2024-05-15T08:27:44+0000","level":"info","message":"spawn command to main: cmdline=[\"/opt/td-agent/bin/ruby\", \"-Eascii-8bit:ascii-8bit\", \"/usr/sbin/td-agent\", \"--under-supervisor\"]"} 2024-05-15 08:27:44 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil 2024-05-15 08:28:09 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil {"time":"2024-05-15T08:28:09+0000","level":"info","message":"adding filter pattern=\"**\" type=\"record_transformer\""} {"time":"2024-05-15T08:28:12+0000","level":"info","message":"adding filter pattern=\"**\" type=\"record_modifier\""} {"time":"2024-05-15T08:28:12+0000","level":"info","message":"adding match pattern=\"**\" type=\"copy\""} {"time":"2024-05-15T08:28:14+0000","level":"info","message":"adding source type=\"tail\""} {"time":"2024-05-15T08:28:15+0000","level":"warn","message":"define to capture fluentd logs in top level is deprecated. Use instead","worker_id":0} {"time":"2024-05-15T08:28:15+0000","level":"warn","message":"parameter 'enable_ruby' in \n @type record_modifier\n enable_ruby true\n remove_keys \"dummy\"\n \n dummy ${require 'json';if record[\"extension\"].empty?; record.delete(\"extension\"); end}\n \n is not used."} {"time":"2024-05-15T08:28:15+0000","level":"info","message":"starting fluentd worker pid=15 ppid=10 worker=0","worker_id":0} {"time":"2024-05-15T08:28:15+0000","level":"info","message":"following tail of /tmp/mainContainerLogs/fluentd.log","worker_id":0} {"time":"2024-05-15T08:28:15+0000","level":"warn","message":"pattern not matched: \"# Logfile created on 2024-05-15 08:27:18 +0000 by logger.rb/v1.4.2\"","worker_id":0}

  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock

    Installed gems

    bash-4.4# /opt/td-agent/bin/fluent-gem list

    *** LOCAL GEMS ***

    activemodel (7.0.5)
    activesupport (7.0.5)
    addressable (2.8.1)
    aes_key_wrap (1.1.0)
    amq-protocol (2.3.2)
    async (1.30.3)
    async-http (0.59.2)
    async-io (1.34.0)
    async-pool (0.3.12)
    attr_required (1.0.1)
    aws-eventstream (1.2.0)
    aws-partitions (1.781.0)
    aws-sdk-cloudwatchlogs (1.65.0)
    aws-sdk-core (3.175.0)
    aws-sdk-kms (1.58.0)
    aws-sdk-s3 (1.116.0)
    aws-sdk-sqs (1.51.1)
    aws-sigv4 (1.5.2)
    base91 (0.0.1)
    benchmark (default: 0.1.0)
    bigdecimal (default: 2.0.0)
    bindata (2.4.14)
    bundler (2.3.18, default: 2.1.4)
    bunny (2.22.0)
    cgi (default: 0.1.0.1)
    cmetrics (0.3.3)
    concurrent-ruby (1.1.10)
    connection_pool (2.4.1)
    console (1.16.2)
    cool.io (1.7.1)
    csv (default: 3.1.2)
    date (default: 3.0.3)
    delegate (default: 0.1.0)
    did_you_mean (default: 1.4.0)
    digest-crc (0.6.4)
    digest-murmurhash (1.1.1)
    domain_name (0.5.20190701)
    elastic-transport (8.1.0)
    elasticsearch (8.4.0)
    elasticsearch-api (8.4.0)
    etc (default: 1.1.0)
    excon (0.93.1)
    faraday (1.10.2)
    faraday-em_http (1.0.0)
    faraday-em_synchrony (1.0.0)
    faraday-excon (1.1.0)
    faraday-httpclient (1.0.1)
    faraday-multipart (1.0.4)
    faraday-net_http (1.0.1)
    faraday-net_http_persistent (1.2.0)
    faraday-patron (1.0.0)
    faraday-rack (1.0.0)
    faraday-retry (1.0.3)
    faraday_middleware (1.2.0)
    faraday_middleware-aws-sigv4 (0.6.1)
    fcntl (default: 1.0.0)
    ffi (1.15.5)
    ffi-compiler (1.0.1)
    fiber-local (1.0.0)
    fiddle (default: 1.0.0)
    fileutils (1.6.0, default: 1.4.1)
    fluent-config-regexp-type (1.0.0)
    fluent-diagtool (1.0.1)
    fluent-logger (0.9.0)
    fluent-plugin-amqp (0.14.0)
    fluent-plugin-brevity-control (0.1.1)
    fluent-plugin-calyptia-monitoring (0.1.3)
    fluent-plugin-clog (0.1.3)
    fluent-plugin-cloudwatch-logs (0.14.3)
    fluent-plugin-concat (2.5.0)
    fluent-plugin-cvea-log (0.0.3)
    fluent-plugin-elasticsearch (5.2.4)
    fluent-plugin-flowcounter-simple (0.1.0)
    fluent-plugin-forest (0.3.3)
    fluent-plugin-genhashvalue (1.1)
    fluent-plugin-grafana-loki (1.2.20)
    fluent-plugin-grok-parser (2.6.2)
    fluent-plugin-kafka (0.18.1)
    fluent-plugin-kubernetes_metadata_filter (3.2.0)
    fluent-plugin-metrics-cmetrics (0.1.2)
    fluent-plugin-multi-format-parser (1.0.0)
    fluent-plugin-opensearch (1.0.8)
    fluent-plugin-out-http (1.3.4)
    fluent-plugin-parser-cri (0.1.1)
    fluent-plugin-postgres (0.1.0)
    fluent-plugin-prometheus (2.0.300001)
    fluent-plugin-prometheus_pushgateway (0.1.0)
    fluent-plugin-record-modifier (2.1.1)
    fluent-plugin-remote_syslog (1.1.0)
    fluent-plugin-rewrite-tag-filter (2.4.0)
    fluent-plugin-route (1.0.0)
    fluent-plugin-s3 (1.7.2)
    fluent-plugin-sd-dns (0.1.0)
    fluent-plugin-splunk-hec (1.3.2)
    fluent-plugin-systemd (1.0.5)
    fluent-plugin-td (1.2.0)
    fluent-plugin-utmpx (0.5.0)
    fluent-plugin-webhdfs (1.5.0)
    fluentd (1.15.3)
    forwardable (default: 1.3.1)
    getoptlong (default: 0.1.0)
    hirb (0.7.3)
    http (5.1.1)
    http-accept (1.7.0)
    http-cookie (1.0.5)
    http-form_data (2.3.0)
    http_parser.rb (0.8.0)
    httpclient (2.8.3)
    i18n (1.14.1)
    io-console (default: 0.5.6)
    ipaddr (default: 1.2.2)
    irb (default: 1.2.6)
    jmespath (1.6.1)
    json (2.6.2, default: 2.3.0)
    json-jwt (1.15.3)
    jsonpath (1.1.3)
    kubeclient (4.11.0)
    linux-utmpx (0.3.0)
    llhttp-ffi (0.4.0)
    logger (default: 1.4.2)
    lru_redux (1.1.0)
    ltsv (0.1.2)
    mail (2.8.1)
    matrix (default: 0.2.0)
    mime-types (3.4.1)
    mime-types-data (3.2023.0218.1)
    mini_mime (1.1.2)
    mini_portile2 (2.8.0)
    minitest (5.13.0)
    msgpack (1.6.0)
    multi_json (1.15.0)
    multipart-post (2.2.3)
    murmurhash3 (0.1.7)
    mutex_m (default: 0.1.0)
    net-http-persistent (4.0.2)
    net-imap (0.3.6)
    net-pop (default: 0.1.0)
    net-protocol (0.2.1)
    net-smtp (default: 0.1.0)
    net-telnet (0.2.0)
    netrc (0.11.0)
    nio4r (2.5.8)
    observer (default: 0.1.0)
    oj (3.13.17)
    open3 (default: 0.1.0)
    openid_connect (1.1.8)
    opensearch-api (2.0.2)
    opensearch-ruby (2.0.3)
    opensearch-transport (2.0.1)
    openssl (default: 2.1.3)
    ostruct (default: 0.2.0)
    parallel (1.22.1)
    pg (1.5.3)
    power_assert (1.1.7)
    prime (default: 0.1.1)
    prometheus-client (4.0.000002)
    protocol-hpack (1.4.2)
    protocol-http (0.23.12)
    protocol-http1 (0.14.6)
    protocol-http2 (0.14.2)
    pstore (default: 0.1.0)
    psych (default: 3.1.0)
    public_suffix (5.0.0)
    racc (default: 1.4.16)
    rack (3.0.8)
    rack-oauth2 (1.21.3)
    rake (13.0.6, 13.0.1)
    rbtree (0.4.6)
    rdkafka (0.11.1)
    rdoc (default: 6.2.1.1)
    readline (default: 0.0.2)
    readline-ext (default: 0.1.0)
    recursive-open-struct (1.1.3)
    reline (default: 0.1.5)
    remote_syslog_sender (1.2.2)
    rest-client (2.1.0)
    rexml (default: 3.2.3.1)
    rss (default: 0.2.8)
    ruby-kafka (1.5.0)
    ruby-progressbar (1.11.0)
    ruby2_keywords (0.0.5)
    rubyzip (1.3.0)
    sdbm (default: 1.0.0)
    serverengine (2.3.0)
    set (1.0.3)
    sigdump (0.2.4)
    singleton (default: 0.1.0)
    sorted_set (1.0.3)
    stringio (default: 0.1.0)
    strptime (0.2.5)
    strscan (default: 1.0.3)
    swd (1.3.0)
    syslog_protocol (0.9.2)
    systemd-journal (1.4.2)
    td (0.16.9)
    td-client (1.0.8)
    td-logger (0.3.28)
    test-unit (3.3.4)
    timeout (default: 0.1.0)
    timers (4.3.5)
    tracer (default: 0.1.0)
    traces (0.7.0)
    tzinfo (2.0.5)
    tzinfo-data (1.2022.5)
    unf (0.1.4)
    unf_ext (0.0.8.2)
    uri (default: 0.10.0)
    validate_email (0.1.6)
    validate_url (1.0.15)
    webfinger (2.0.0)
    webhdfs (0.10.2)
    webrick (1.7.0, default: 1.6.1)
    xmlrpc (0.3.0)
    yajl-ruby (1.4.3)
    yaml (default: 0.1.0)
    zip-zip (0.3)
    zlib (default: 1.1.0)

[waza-ari]

I can confirm the same behaviour. Did you ever find any workaround?

[waza-ari]

I did some investigation and found both the cause and a workaround. This is not directly a limitation of the opensearch plugin, but rather the underlying (default) excon HTTP client. Sending the full chain using excon requires a separate parameter and files for intermediate CAs, it apparently cannot handle the chain in the actual client_certificate. See the relevant section from their Readme:

Optionally, you can also pass the whole chain by passing the extra certificates through client_chain:

connection = Excon.new('https://example.com',
                      client_cert: 'mycert.pem',
                      client_chain: 'mychain.pem',
                      client_key: 'mycert.key')

You can see the implementation in those lines, where clearly the cert_chain parameter is not propagated (and not exposed in the first place):

fluent-plugin-opensearch/lib/fluent/plugin/out_opensearch.rb

Lines 514 to 518 in 5bf8450

	when :excon
	{ client_key: @client_key, client_cert: @client_cert, client_key_pass: @client_key_pass, nonblock: @http_backend_excon_nonblock }
	when :typhoeus
	require 'faraday/typhoeus'
	{ sslkey: @client_key, sslcert: @client_cert, keypasswd: @client_key_pass }

One workaround is to switch to the other supported HTTP backend typhoeus. The last line is important:

<match **>
    @type opensearch
    host ...
    port 9200
    scheme https
    ca_file /etc/fluent/root-ca/ca.crt
    client_cert /etc/fluent/fluentd-certs/tls.crt
    client_key /etc/fluent/fluentd-certs/tls.key
    http_backend typhoeus
</match>

Note that you need to install faraday-typhoeus (and therefore typhoeus), and note that typhoeus requires libcurl. Personally, I'm installing fluentd via their Helm chart, which in turn references Debian based images. A simple Dockerfile like this installed all required things:

FROM fluent/fluentd-kubernetes-daemonset:v1.17.1-debian-opensearch-1.0

# Install faraday-typhoeus and its libcurl4 dependency 
RUN apt-get update \
     && apt-get upgrade -y \
     && apt-get install -y --no-install-recommends libcurl4 \
     && fluent-gem install faraday-typhoeus \
     && rm -rf /var/lib/apt/lists/* \
     && gem sources --clear-all \
     && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

ENTRYPOINT ["tini", "--", "/fluentd/entrypoint.sh"]

[aggarwalShivani]

Hi @waza-ari
Thanks a lot for your inputs. So to summarize, in order to solve this issue, there are two ways. Pls confirm if this aligns.

1. Addition in fluent-plugin-opensearch plugin to expose and propagate the client_chain parameter.
2. Or (workaround) - we install the faraday-typhoeus gem and configure 'http_backend typhoeus'.

Requesting fluentd community maintainers for their feedback too.

[aggarwalShivani]

Hi @waza-ari
Can you confirm if my understanding is right?

Hi Fluentd maintainers,
Requesting confirmation if the change suggested in point 1 is acceptable from the design perspective?

1. Addition in fluent-plugin-opensearch plugin to expose and propagate the client_chain parameter.

[waza-ari]

Hi @aggarwalShivani, it's some time ago, but if I remember correctly, your understanding is correct. It should be enough to expose the client_chain parameter to be configured by the user of the plugin.