InfluxDB doesn’t actually delete data when you drop a measurement or series; it just marks it for deletion, and that space isn’t reclaimed until a manual compaction process runs.
Let’s see this in action. Imagine you have an InfluxDB instance and you’ve been aggressively dropping old data, maybe a daily cleanup of a retention policy. You check your disk usage, and it’s barely budged. This is because InfluxDB stores data in Time Structured Merge (TSM) files. When you delete data, it’s not immediately removed from these files. Instead, a tombstone marker is written, indicating that this data point is no longer valid. The actual space reclamation happens when InfluxDB compacts these TSM files.
Here’s how you trigger that compaction manually.
First, you need to identify the shard that holds the data you want to compact. You can get a list of shards by querying InfluxDB itself. Connect to your InfluxDB instance using the influx CLI:
influx
Then, run the following query:
SHOW SHARDS
This will output something like:
id,database,retention_policy,shard_group,start_time,end_time,expiry_time,owner_database,owner_retention_policy
1,mydb,autogen,1550822400000000000,2019-02-22T00:00:00Z,2019-02-23T00:00:00Z,<nil>,mydb,autogen
2,mydb,autogen,1550908800000000000,2019-02-23T00:00:00Z,2019-02-24T00:00:00Z,<nil>,mydb,autogen
3,mydb,autogen,1550995200000000000,2019-02-24T00:00:00Z,2019-02-25T00:00:00Z,<nil>,mydb,autogen
Note the id column. This is your shard ID. You’ll also need the database and retention_policy associated with that shard ID. For this example, let’s assume we want to compact shard ID 2, which belongs to database mydb and retention policy autogen.
Next, you need to find the directory where InfluxDB stores its data. This is configured in your influxdb.conf file, typically under the [data] section, with the dir parameter. A common default location on Linux systems is /var/lib/influxdb/data.
Within that data directory, each shard has its own subdirectory named after its ID. So, for shard ID 2, you’d look in /var/lib/influxdb/data/mydb/autogen/2/. Inside this directory, you’ll find .tsm files. These are the files containing your time-series data.
Now, to trigger the compaction, you’ll use the influxd command-line tool with the run-compaction subcommand. You specify the shard ID, the database name, the retention policy name, and optionally, the shard group duration. The shard group duration is crucial because it tells InfluxDB how to group TSM files for compaction. It’s usually derived from your retention policy’s GROUP BY time() clause. For daily retention policies, this is often 168h (7 days) or 24h depending on your configuration. If you’re unsure, check your retention policy definition:
SHOW RETENTION POLICIES ON mydb
This might show something like:
name,duration,shardGroupDuration,replica,default
autogen,168h0m0s,24h0m0s,1,true
Here, the shardGroupDuration is 24h0m0s.
The command to run manual compaction looks like this:
influxd run-compaction --shard-id 2 --database mydb --retention-policy autogen --shard-group-duration 24h
When you run this command, InfluxDB will:
- Identify all TSM files associated with the specified shard.
- Read the data from these files, respecting the tombstone markers.
- Write new, smaller TSM files that contain only the active data.
- Replace the old TSM files with the newly compacted ones.
- Delete the old, now-unused TSM files, thereby reclaiming disk space.
The run-compaction command is synchronous, meaning it will block until the compaction for that shard is complete. For very large shards, this can take a significant amount of time and consume considerable I/O and CPU resources. You might want to run this during off-peak hours.
After the compaction finishes successfully, you should see a reduction in the disk space used by the shard’s directory.
The next error you’ll likely encounter after successfully reclaiming disk space is related to max-series-per-database or max-series-per-tag-value if your data model is inefficient and you’re hitting those limits, or perhaps a too many open files error if you have a very large number of shards and TSM files.