Fluentd, the log collector, is failing to send logs to New Relic, your observability platform, because the New Relic ingest endpoint is rejecting the log data.

Common Causes and Fixes

  1. Incorrect New Relic License Key: This is the most common culprit. Fluentd needs a valid license key to authenticate with New Relic.

    • Diagnosis: Check your Fluentd configuration file (often fluent.conf or a file in plugin/conf.d/). Look for the license_key parameter within your New Relic output plugin configuration. Compare this key to the one found in your New Relic account settings (under Admin > Account settings > License key).
    • Fix: Update the license_key in your Fluentd configuration to the correct, active key. For example, if your New Relic output section looks like this:
      <match **>
        @type newrelic
        license_key YOUR_OLD_OR_INCORRECT_KEY
        # ... other parameters
      </match>
      
      Change it to:
      <match **>
        @type newrelic
        license_key YOUR_CORRECT_LICENSE_KEY
        # ... other parameters
      </match>
      
    • Why it works: The New Relic API requires a valid license key for every ingestion request. An incorrect key results in an immediate authentication failure, and logs are dropped.
  2. Network Connectivity Issues/Firewall Blocking: Fluentd might not be able to reach the New Relic ingest endpoint due to network misconfigurations or firewalls.

    • Diagnosis: From the server where Fluentd is running, try to curl the New Relic ingest endpoint. The primary endpoint is log-api.newrelic.com/log/v1.
      curl -v https://log-api.newrelic.com/log/v1
      
      If this command times out, returns an error (like connection refused or timeout), or shows a certificate error, you have a network issue. Also, check your firewall rules on the Fluentd host and any intermediate network devices.
    • Fix: Ensure that outbound connections on port 443 (HTTPS) are allowed from your Fluentd server to log-api.newrelic.com. If you’re using a proxy, configure Fluentd to use it.
    • Why it works: Logs can only be sent if Fluentd can establish a TCP connection and complete the TLS handshake with the New Relic API server.
  3. Incorrect endpoint Configuration (for specific regions or custom endpoints): While log-api.newrelic.com is the default, if you’re in a specific New Relic region or using a custom endpoint, this must be configured correctly.

    • Diagnosis: Verify the endpoint parameter in your Fluentd New Relic output plugin configuration. If it’s missing or incorrect, Fluentd will try to send logs to the wrong place. Check the New Relic documentation for the correct endpoint for your region (e.g., log-api.eu.newrelic.com/log/v1 for European regions).
    • Fix: Update the endpoint parameter in your Fluentd configuration.
      <match **>
        @type newrelic
        license_key YOUR_LICENSE_KEY
        endpoint https://log-api.eu.newrelic.com/log/v1 # Example for EU
        # ... other parameters
      </match>
      
    • Why it works: The endpoint parameter explicitly tells Fluentd where to send the logs. An incorrect endpoint means the logs are sent to a non-existent or incorrect API, leading to rejection.
  4. Outdated Fluentd New Relic Plugin Version: Older versions of the fluent-plugin-newrelic might have bugs or be incompatible with recent changes in the New Relic API.

    • Diagnosis: Check the version of the fluent-plugin-newrelic gem installed on your Fluentd server.
      gem list | grep fluent-plugin-newrelic
      
      Compare this to the latest version available on RubyGems.
    • Fix: Update the plugin to the latest version:
      # Stop Fluentd first
      gem install fluent-plugin-newrelic -v <latest_version>
      # Restart Fluentd
      
      Replace <latest_version> with the actual latest version number.
    • Why it works: Newer plugin versions often include fixes for API compatibility, performance improvements, and bug resolutions that could be preventing successful log transmission.
  5. TLS/SSL Certificate Issues: While less common, Fluentd might be having trouble verifying the SSL certificate of the New Relic ingest endpoint. This can happen with custom CA bundles or strict system SSL configurations.

    • Diagnosis: Look for errors related to SSL, certificate verification, or TLS handshake failures in your Fluentd logs. You might need to increase Fluentd’s log level to debug to see these.
    • Fix: Ensure your system’s CA certificate store is up-to-date. If you are using a custom ssl_client_cert or ssl_client_key in your Fluentd config (which is unusual for sending logs to New Relic, but possible), ensure they are valid and correctly configured. Sometimes, explicitly setting ssl_verify to true (it’s the default) can help if it was accidentally set to false.
    • Why it works: Successful TLS negotiation is crucial for secure communication. If Fluentd cannot trust the New Relic server’s certificate, it will refuse to connect.
  6. Rate Limiting or Quota Exceeded: If you’re sending an exceptionally high volume of logs, you might hit New Relic’s ingest rate limits.

    • Diagnosis: Check the Fluentd logs for any messages indicating 429 Too Many Requests or similar HTTP status codes from the New Relic endpoint. Also, monitor your New Relic account for any notifications about ingest limits.
    • Fix: Reduce the volume of logs being sent by:
      • Filtering logs at the source or in Fluentd.
      • Batching logs more effectively (though the plugin usually handles this).
      • Contacting New Relic support to discuss increasing your ingest quota if your volume is legitimately high and expected.
    • Why it works: Adhering to API rate limits is essential for stable service operation. Reducing the rate of requests prevents the API from rejecting your data due to overload.

After fixing these, you might encounter issues with log parsing or enrichment if your Fluentd configuration is malformed or if the New Relic Logs UI isn’t displaying data as expected due to incorrect field mapping.

Want structured learning?

Take the full Fluentd course →