NATS Object Store is a distributed, eventually consistent key-value store that can handle large files, built on top of JetStream.
Let’s see it in action. Imagine you have a service that needs to store user avatars. Instead of managing a separate object storage system, you can use NATS Object Store.
First, ensure you have a NATS server running with JetStream enabled. You’ll also need a NATS client library. Here’s a conceptual Python example:
import asyncio
from nats.aio.client import Client as NATS
async def main():
nc = NATS()
await nc.connect("nats://localhost:4222")
# Create an object store if it doesn't exist
try:
os = await nc.object_store("avatars")
except Exception as e:
print(f"Object store 'avatars' already exists: {e}")
os = await nc.object_store("avatars")
# Upload a file
with open("user_avatar.jpg", "rb") as f:
await os.put({"name": "user_avatar.jpg", "content": f.read()})
print("Avatar uploaded.")
# Download a file
obj = await os.get("user_avatar.jpg")
with open("downloaded_avatar.jpg", "wb") as f:
f.write(obj.data)
print("Avatar downloaded.")
await nc.close()
if __name__ == '__main__':
asyncio.run(main())
This code snippet demonstrates the core functionality: creating an object store named "avatars" and then putting (uploading) and getting (downloading) a file. The put operation takes a dictionary with at least name and content. The get operation returns an object that contains the file’s data.
The problem NATS Object Store solves is unifying your messaging and large file storage needs. Traditionally, you’d have a message broker like NATS for ephemeral data and a separate system like S3 or MinIO for persistent, large objects. Object Store brings these together. It leverages JetStream’s persistence and replication capabilities, meaning your objects are stored reliably and can be accessed even if a NATS server goes down.
Internally, each object store is a JetStream stream. When you put an object, NATS creates a new message in that stream. The message payload contains the object’s metadata and its content. For large files, NATS doesn’t store the entire file in a single message in the traditional sense; instead, it manages chunks and references efficiently. When you get an object, NATS reconstructs it from these stored chunks. The name you provide for the object acts as the key for retrieval.
The exact levers you control are primarily around the stream configuration that underlies the object store. When you create an object store, you can specify parameters that mirror JetStream stream configuration, such as retention policies (limits, interest), storage type (file, memory), and replication factors. For instance, to ensure high availability, you can configure the underlying stream to have a replication factor of 3.
nats context save my-cluster
nats stream add avatars --replicas 3 --storage file --max-age 7d
nats server start -js
This sets up an "avatars" stream that replicates data across 3 NATS servers, stores data on disk, and automatically purges objects older than 7 days.
When you upload an object, NATS doesn’t just dump the raw bytes into a message. It intelligently breaks down larger files into manageable chunks and stores them as individual messages within the underlying JetStream stream. Each chunk message includes metadata that allows NATS to reconstruct the original file. This chunking mechanism is crucial for handling files that exceed typical message size limits and for enabling efficient retrieval and partial reads if needed. The put operation on the client side handles this chunking transparently; you provide the whole file, and NATS takes care of the rest.
The next concept you’ll likely explore is managing object metadata and versions.