Neo4j stores its data on disk in a format that has evolved over time, and understanding these formats is key to optimizing performance and managing your database effectively.
Let’s look at Neo4j in action, specifically how it writes and reads data from disk. Imagine you’re running a simple query to find all users connected to a specific user:
MATCH (u:User {userId: "user123"})-[:FOLLOWS]->(friend:User)
RETURN friend.name
When this query runs, Neo4j doesn’t just magically pull data from RAM. It needs to access the data on disk. The way it structures that data on disk is determined by the "store format." Neo4j has historically used different store formats, primarily evolving from an "aligned" format to a "high-limit" format.
The core problem Neo4j’s store format addresses is efficient storage and retrieval of graph data, which is inherently non-linear and highly interconnected. Traditional relational databases struggle with deep graph traversals because they rely on joins, which can become prohibitively expensive. Neo4j’s property graph model and its on-disk format are designed to make these traversals fast.
Internally, Neo4j’s store files are organized into a series of blocks. These blocks contain different types of data: nodes, relationships, properties, and the pointers that link them together. The store format dictates how these pieces of information are laid out within these blocks and how the blocks themselves are managed.
The older "aligned" store format was simpler but had limitations. It tended to allocate space in a more rigid, aligned manner. This could lead to inefficiencies, especially with variable-sized data like property values or when dealing with a large number of relationships. For instance, if a node had many relationships, the space allocated for those relationship pointers might not be perfectly utilized, leading to fragmentation or wasted space.
The "high-limit" store format, introduced in later Neo4j versions (primarily Neo4j 3.x and onwards), is a significant improvement. It’s designed to be more flexible and efficient. The "high-limit" moniker refers to its ability to handle larger numbers of elements (nodes, relationships, properties) within its storage structures. It uses more dynamic allocation strategies and improved data compression techniques. This means that Neo4j can store more data in less disk space and access it more quickly. For example, instead of fixed-size chunks for relationship lists, it uses more adaptive structures that grow and shrink as needed, reducing internal fragmentation.
The main levers you control are indirectly through the Neo4j version you use and the configuration settings that influence the store. While you don’t directly "choose" the store format after installation in the same way you’d pick a file format in a word processor, the version of Neo4j you run dictates which format is active. Neo4j versions 3.0 and later primarily use the high-limit format. Older versions (like 2.x) used the aligned format. When upgrading, Neo4j performs a migration to convert the store to the new format.
The surprising efficiency of the high-limit format comes from its intelligent handling of variable-length data and its reduced overhead for managing these structures. It’s not just about storing more data; it’s about organizing it so that traversing relationships, finding nodes by ID, or accessing properties is as close to a direct lookup as possible, minimizing disk seeks and maximizing cache utilization. This is achieved through more sophisticated indexing strategies within the store files and a more compact representation of relationship and property data.
Upgrading to a Neo4j version that supports the high-limit store format is the primary way to benefit from its improvements.