Fluentd, the log collector, is failing to send logs to New Relic, your observability platform, because the New Relic ingest endpoint is rejecting the log data.
Common Causes and Fixes
-
Incorrect New Relic License Key: This is the most common culprit. Fluentd needs a valid license key to authenticate with New Relic.
- Diagnosis: Check your Fluentd configuration file (often
fluent.confor a file inplugin/conf.d/). Look for thelicense_keyparameter within your New Relic output plugin configuration. Compare this key to the one found in your New Relic account settings (under Admin > Account settings > License key). - Fix: Update the
license_keyin your Fluentd configuration to the correct, active key. For example, if your New Relic output section looks like this:
Change it to:<match **> @type newrelic license_key YOUR_OLD_OR_INCORRECT_KEY # ... other parameters </match><match **> @type newrelic license_key YOUR_CORRECT_LICENSE_KEY # ... other parameters </match> - Why it works: The New Relic API requires a valid license key for every ingestion request. An incorrect key results in an immediate authentication failure, and logs are dropped.
- Diagnosis: Check your Fluentd configuration file (often
-
Network Connectivity Issues/Firewall Blocking: Fluentd might not be able to reach the New Relic ingest endpoint due to network misconfigurations or firewalls.
- Diagnosis: From the server where Fluentd is running, try to
curlthe New Relic ingest endpoint. The primary endpoint islog-api.newrelic.com/log/v1.
If this command times out, returns an error (likecurl -v https://log-api.newrelic.com/log/v1connection refusedortimeout), or shows a certificate error, you have a network issue. Also, check your firewall rules on the Fluentd host and any intermediate network devices. - Fix: Ensure that outbound connections on port 443 (HTTPS) are allowed from your Fluentd server to
log-api.newrelic.com. If you’re using a proxy, configure Fluentd to use it. - Why it works: Logs can only be sent if Fluentd can establish a TCP connection and complete the TLS handshake with the New Relic API server.
- Diagnosis: From the server where Fluentd is running, try to
-
Incorrect
endpointConfiguration (for specific regions or custom endpoints): Whilelog-api.newrelic.comis the default, if you’re in a specific New Relic region or using a custom endpoint, this must be configured correctly.- Diagnosis: Verify the
endpointparameter in your Fluentd New Relic output plugin configuration. If it’s missing or incorrect, Fluentd will try to send logs to the wrong place. Check the New Relic documentation for the correct endpoint for your region (e.g.,log-api.eu.newrelic.com/log/v1for European regions). - Fix: Update the
endpointparameter in your Fluentd configuration.<match **> @type newrelic license_key YOUR_LICENSE_KEY endpoint https://log-api.eu.newrelic.com/log/v1 # Example for EU # ... other parameters </match> - Why it works: The
endpointparameter explicitly tells Fluentd where to send the logs. An incorrect endpoint means the logs are sent to a non-existent or incorrect API, leading to rejection.
- Diagnosis: Verify the
-
Outdated Fluentd New Relic Plugin Version: Older versions of the
fluent-plugin-newrelicmight have bugs or be incompatible with recent changes in the New Relic API.- Diagnosis: Check the version of the
fluent-plugin-newrelicgem installed on your Fluentd server.
Compare this to the latest version available on RubyGems.gem list | grep fluent-plugin-newrelic - Fix: Update the plugin to the latest version:
Replace# Stop Fluentd first gem install fluent-plugin-newrelic -v <latest_version> # Restart Fluentd<latest_version>with the actual latest version number. - Why it works: Newer plugin versions often include fixes for API compatibility, performance improvements, and bug resolutions that could be preventing successful log transmission.
- Diagnosis: Check the version of the
-
TLS/SSL Certificate Issues: While less common, Fluentd might be having trouble verifying the SSL certificate of the New Relic ingest endpoint. This can happen with custom CA bundles or strict system SSL configurations.
- Diagnosis: Look for errors related to SSL, certificate verification, or TLS handshake failures in your Fluentd logs. You might need to increase Fluentd’s log level to
debugto see these. - Fix: Ensure your system’s CA certificate store is up-to-date. If you are using a custom
ssl_client_certorssl_client_keyin your Fluentd config (which is unusual for sending logs to New Relic, but possible), ensure they are valid and correctly configured. Sometimes, explicitly settingssl_verifytotrue(it’s the default) can help if it was accidentally set tofalse. - Why it works: Successful TLS negotiation is crucial for secure communication. If Fluentd cannot trust the New Relic server’s certificate, it will refuse to connect.
- Diagnosis: Look for errors related to SSL, certificate verification, or TLS handshake failures in your Fluentd logs. You might need to increase Fluentd’s log level to
-
Rate Limiting or Quota Exceeded: If you’re sending an exceptionally high volume of logs, you might hit New Relic’s ingest rate limits.
- Diagnosis: Check the Fluentd logs for any messages indicating
429 Too Many Requestsor similar HTTP status codes from the New Relic endpoint. Also, monitor your New Relic account for any notifications about ingest limits. - Fix: Reduce the volume of logs being sent by:
- Filtering logs at the source or in Fluentd.
- Batching logs more effectively (though the plugin usually handles this).
- Contacting New Relic support to discuss increasing your ingest quota if your volume is legitimately high and expected.
- Why it works: Adhering to API rate limits is essential for stable service operation. Reducing the rate of requests prevents the API from rejecting your data due to overload.
- Diagnosis: Check the Fluentd logs for any messages indicating
After fixing these, you might encounter issues with log parsing or enrichment if your Fluentd configuration is malformed or if the New Relic Logs UI isn’t displaying data as expected due to incorrect field mapping.