New Relic data retention isn’t just about how long you can keep data; it’s about defining what data you need to keep and why, and then configuring the system to do precisely that, no more, no less.
Let’s see this in action. Imagine you’ve got a critical microservice, user-auth-service, and you need its transaction traces for the last 30 days, but its error data only needs to be kept for 7 days. Here’s how you’d configure that in New Relic, assuming you’re using the modern data model:
resource "newrelic_data_retention" "user_auth_service_retention" {
account_id = 1234567
rules {
name = "user-auth-service-traces"
type = "TRACE"
days = 30
filter {
attribute = "service.name"
operator = "EQUALS"
value = "user-auth-service"
}
}
rules {
name = "user-auth-service-errors"
type = "ERROR"
days = 7
filter {
attribute = "service.name"
operator = "EQUALS"
value = "user-auth-service"
}
}
}
This Terraform code defines two specific retention rules for the user-auth-service. The first rule, user-auth-service-traces, targets data of TRACE type and keeps it for 30 days, but only for events where the service.name attribute is exactly user-auth-service. The second rule, user-auth-service-errors, does the same but for ERROR data and keeps it for 7 days.
The core problem New Relic data retention solves is the exponential cost and complexity of storing all telemetry data indefinitely. By default, New Relic has a standard retention policy. For most data types (like metrics, events, and logs), this is 30 days. For others, like traces or error events, it might be shorter. However, this default might not align with your compliance requirements, debugging needs, or cost optimization strategies. You might need to retain specific, high-cardinality data for longer periods for forensic analysis or regulatory audits, while other, more granular data can be purged sooner to save on storage costs.
Internally, New Relic’s data retention is managed through a set of configurable rules. When data streams into New Relic, it’s tagged with various attributes. These rules act as filters, identifying specific data subsets based on these attributes and then applying a defined retention period to that subset. The system then periodically prunes data older than the configured retention period for each matching rule. This granular control allows you to create a tiered retention strategy.
The key levers you control are:
account_id: The New Relic account where these rules will apply.rules: A block defining individual retention policies.name: A human-readable identifier for the rule.type: The data type to which the rule applies (e.g.,METRIC,EVENT,TRACE,LOG,ERROR). This is crucial as different data types have different default retentions and storage costs.days: The number of days to retain data matching this rule. This is the primary control for how long data lives.filter: A block that specifies conditions for matching data.attribute: The data attribute to filter on (e.g.,service.name,host.name,deployment.id).operator: The comparison operator (EQUALS,NOT_EQUALS,IN,NOT_IN,STARTS_WITH,ENDS_WITH,CONTAINS,GREATER_THAN,LESS_THAN, etc.).value: The value(s) to compare the attribute against. You can specify multiple values forINandNOT_INoperators.
It’s important to note that these rules are additive. If a data point matches multiple retention rules, the longest retention period among the matching rules will be applied. This can sometimes lead to unexpected data retention if not carefully managed. For example, if you have a general rule to keep all METRIC data for 30 days, and a specific rule to keep METRIC data for a particular service for 60 days, the specific service’s metrics will be retained for 60 days.
When configuring retention, be mindful of the interplay between data types and your filtering. For instance, if you filter on service.name for TRACE data, and you also want to retain LOG data for the same service, you’ll need a separate rule for LOG data with the same filter criteria. The system doesn’t automatically infer relationships between data types based on filters.
The next concept you’ll likely encounter is how these retention rules interact with New Relic’s data ingestion pipeline, particularly concerning custom attributes and their impact on query performance and storage costs when defining broad retention policies.