Fluentd’s forward input and output plugins can secure their communication using mutual TLS, ensuring both the client and server authenticate each other before exchanging logs.
Here’s Fluentd forwarding logs between two instances, fluentd-server and fluentd-client, with mutual TLS enabled.
First, let’s set up our certificates. We’ll need a Certificate Authority (CA) to sign both the server and client certificates.
# Create CA private key
openssl genrsa -out ca.key 2048
# Create CA certificate
openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 -out ca.crt \
-subj "/CN=MyFluentdCA"
# Create server private key
openssl genrsa -out server.key 2048
# Create server certificate signing request (CSR)
openssl req -new -key server.key -out server.csr \
-subj "/CN=fluentd-server.example.com"
# Sign server CSR with CA
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out server.crt -days 365 -sha256
# Create client private key
openssl genrsa -out client.key 2048
# Create client certificate signing request (CSR)
openssl req -new -key client.key -out client.csr \
-subj "/CN=fluentd-client.example.com"
# Sign client CSR with CA
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out client.crt -days 365 -sha256
Now, we need to combine the certificate and private key for each side.
On fluentd-server:
cat server.crt server.key > server.pem
On fluentd-client:
cat client.crt client.key > client.pem
Ensure ca.crt is available on both fluentd-server and fluentd-client.
Fluentd Server Configuration (fluentd-server.conf)
The server listens for incoming connections on a specific port, configured to require client authentication.
<source>
@type forward
port 24224
@id input_forward
<security>
shared_key "changeme" # This is for basic auth if mTLS is NOT used.
# For mTLS, it's not strictly necessary but good to have
# as a fallback or for non-mTLS clients.
</security>
<transport tls>
cert_each false
cert /etc/fluentd/certs/server.pem
private_key /etc/fluentd/certs/server.pem
ca_cert /etc/fluentd/certs/ca.crt
verify_mode 1 # 1 means verify client certificate
</transport>
</source>
<match **>
@type stdout
@id output_stdout
</match>
cert: Path to the server’s combined certificate and private key.private_key: Same ascertwhen using a combined file.ca_cert: Path to the CA certificate used to verify client certificates.verify_mode 1: This is crucial. It tells the server to request a client certificate and verify it against the providedca_cert.
Fluentd Client Configuration (fluentd-client.conf)
The client connects to the server, presenting its own certificate for authentication.
<source>
@type dummy
dummy {"message":"Hello from client!"}
@id input_dummy
rate 1
</source>
<match **>
@type forward
host fluentd-server.example.com
port 24224
@id output_forward
<security>
shared_key "changeme" # Same as server, for fallback/non-mTLS.
</security>
<transport tls>
cert /etc/fluentd/certs/client.pem
private_key /etc/fluentd/certs/client.pem
ca_cert /etc/fluentd/certs/ca.crt
insecure_skip_verify false # Ensure we are verifying the server cert
</transport>
</match>
cert: Path to the client’s combined certificate and private key.private_key: Same ascertfor a combined file.ca_cert: Path to the CA certificate used to verify the server’s certificate.insecure_skip_verify false: This ensures the client verifies the server’s certificate against the providedca_cert.
Running Fluentd
-
On
fluentd-server: Placeserver.pemandca.crtin/etc/fluentd/certs/. Start Fluentd:fluentd -c fluentd-server.conf -
On
fluentd-client: Placeclient.pemandca.crtin/etc/fluentd/certs/. Start Fluentd:fluentd -c fluentd-client.conf
You should see the "Hello from client!" message appear in the fluentd-server’s stdout.
The verify_mode parameter on the server is what enforces mutual TLS. When set to 1, the server will not only present its certificate but also request one from the client. If the client doesn’t provide a valid certificate (signed by the same CA), the connection will be rejected. Conversely, on the client, insecure_skip_verify false ensures it trusts the server’s identity.
After successfully configuring mutual TLS, the next common hurdle is ensuring proper certificate rotation and management, especially in production environments where certificates have expiration dates.