InfluxDB shards are only ever deleted, never truncated or modified.
Let’s see InfluxDB in action. Imagine you’re monitoring a fleet of IoT devices, each sending temperature readings every 10 seconds. That’s 6 readings per minute, per device. If you have 1,000 devices, you’re looking at 6,000 data points per minute, or 360,000 per hour. Over a day, that’s 8.64 million points. InfluxDB needs to organize this data efficiently for writes and reads. It does this by grouping data into "shards."
Each shard is a directory on disk containing data for a specific time range. When data is written to InfluxDB, it’s assigned to a shard based on its timestamp. The crucial parameter here is the "shard duration," which defines the time window each shard covers.
Think of it like filing cabinets. If you have a small filing cabinet and put too many documents in it, it gets unwieldy. If you have too many small filing cabinets, you spend a lot of time opening and closing them. Shard duration is about finding the right size for your filing cabinets.
InfluxDB’s default shard duration is often 7d (7 days). This means all data points within a 7-day window will reside in the same shard directory. When that 7-day window passes, InfluxDB creates a new shard for the next 7 days. The old shard becomes "immutable" (read-only) and will eventually be deleted based on your data retention policies.
The problem arises when your data rate is very high or very low relative to the default shard duration.
Too Many Small Shards: If you have a very high data ingestion rate and a short shard duration (e.g., 1h with millions of points per hour), you’ll end up with an enormous number of shards. This can overwhelm the InfluxDB metadata store, leading to slow query performance and even write failures. The system spends too much time managing the sheer volume of shard directories and their associated metadata.
Too Few Large Shards: Conversely, if you have a low data ingestion rate and a very long shard duration (e.g., 30d with only a few data points per hour), each shard will grow very large. While you won’t have a metadata management problem, queries that span a long time range will have to sift through massive amounts of data within a single shard, degrading read performance. Furthermore, if you need to delete old data, InfluxDB has to scan the entire large shard to remove it, which is inefficient.
The Sweet Spot: The goal is to have shards that are large enough to minimize metadata overhead but small enough that queries and data deletion operations are efficient. A common recommendation is to aim for shards that contain between 100,000 and 1,000,000 data points.
Let’s say you’re ingesting 100,000 data points per minute.
- If your shard duration is
1h(60 minutes), a single shard would accumulate 6,000,000 data points. This is likely too large. - If your shard duration is
1m(1 minute), a single shard would accumulate 100,000 data points. This is a good target. - If your shard duration is
10m(10 minutes), a single shard would accumulate 1,000,000 data points. This is also a good target.
So, for 100,000 points/minute, a shard duration of 10m or 1h (if you can tolerate slightly larger shards) would be a better choice than the default 7d.
How to Choose and Configure:
- Estimate Your Data Rate: Determine the average number of data points per second, minute, or hour your system will ingest.
- Target Shard Size: Aim for 100,000 to 1,000,000 points per shard.
- Calculate Shard Duration:
Shard Duration = Target Shard Size / Data Rate.- Example: Data rate = 100,000 points/minute. Target shard size = 1,000,000 points.
Shard Duration = 1,000,000 points / 100,000 points/minute = 10 minutes.- So, you’d set your shard duration to
10m.
Configuration: Shard duration is set at the database level, within the CREATE DATABASE or ALTER DATABASE command. You can also configure it via the influxd configuration file under [storage.sharding].
Here’s how you’d set it when creating a database:
CREATE DATABASE my_iot_db WITH DURATION 10m REPLICATION 1 SHARD DURATION 10m
If you’re altering an existing database:
ALTER DATABASE my_iot_db DURATION 10m SHARD DURATION 10m
Important Note on DURATION vs. SHARD DURATION:
In InfluxDB v1.x, DURATION referred to the retention period, and SHARD DURATION was the time window for shards. In InfluxDB v2.x, the concept is slightly different. You set the retention period for a bucket, and the shard group duration is automatically calculated based on the data rate and retention period. However, the principle of optimizing for shard size remains critical for performance. For v2.x, you’d configure retention policies on buckets:
influx bucket create --name my_bucket --org your-org-id --retention 30d
InfluxDB then manages shard group sizes internally. The advice to consider data rate and desired shard group size (even if not directly configurable in v2.x as a fixed duration) still applies to understanding how InfluxDB will organize data for optimal performance. If you’re on v1.x, the direct SHARD DURATION setting is what you’ll manipulate.
The actual physical location of shard files on disk is within your InfluxDB data directory. For example, a shard directory might look like /var/lib/influxdb/data/your_database/autogen/123/456, where 123 is the shard ID and 456 is the TSM file ID.
The one thing most people don’t realize is that InfluxDB doesn’t just pick an arbitrary shard duration. It dynamically creates and manages shard groups (which are collections of shards covering a specific time range) based on your retention policy and ingestion rate. The goal is to balance the number of shard groups and the size of individual shards. If your data rate fluctuates wildly, InfluxDB might adjust shard group sizes to try and maintain optimal performance.
When you change the shard duration for an existing database, InfluxDB doesn’t immediately reorganize existing shards. It only applies the new duration to newly created shards. Existing shards will remain with their original duration until they expire and are re-created.