A NATS cluster upgrade via rolling restart is actually a process of graceful degradation where nodes momentarily operate with a reduced quorum, not a true zero-downtime event for all operations.
Let’s see what this looks like in practice. Imagine we have a 3-node NATS cluster (nodes A, B, and C). We want to upgrade node A.
# On node A, before the restart
nats-server --config nats.conf
[1] 2023/10/27 10:00:00.123456 [INF] Starting NATS Server - version 2.9.17, cluster nats-cluster, id 1
[1] 2023/10/27 10:00:00.123457 [INF] Listening for clients on 0.0.0.0:4222
[1] 2023/10/27 10:00:00.123458 [INF] Server is configured with cluster URL: nats://0.0.0.0:6222
[1] 2023/10/27 10:00:00.123459 [INF] Server is configured with routes: [nats://nodeB:6222 nats://nodeC:6222]
[1] 2023/10/27 10:00:00.123460 [INF] Cluster configuration: nats://0.0.0.0:6222
[1] 2023/10/27 10:00:00.123461 [INF] Cluster state: LEARNING -> LEADER (quorum: 2)
[1] 2023/10/27 10:00:00.123462 [INF] Node "1" is LEADER
Now, we initiate the upgrade on node A by restarting the nats-server process. This might involve systemctl restart nats-server or simply killing and re-launching the process.
# On node A, after the restart (new process starts with upgraded binary)
nats-server --config nats.conf
[1] 2023/10/27 10:01:00.789012 [INF] Starting NATS Server - version 2.10.0, cluster nats-cluster, id 1
[1] 2023/10/27 10:01:00.789013 [INF] Listening for clients on 0.0.0.0:4222
[1] 2023/10/27 10:01:00.789014 [INF] Server is configured with cluster URL: nats://0.0.0.0:6222
[1] 2023/10/27 10:01:00.789015 [INF] Server is configured with routes: [nats://nodeB:6222 nats://nodeC:6222]
[1] 2023/10/27 10:01:00.789016 [INF] Cluster configuration: nats://0.0.0.0:6222
[1] 2023/10/27 10:01:00.789017 [INF] Cluster state: LEARNING -> follower (quorum: 2)
[1] 2023/10/27 10:01:00.789018 [INF] Node "1" is FOLLOWER
Notice the change in cluster state: from LEADER to FOLLOWER. During this brief period, the cluster has a reduced quorum. If there were only 2 nodes, this would be a problem. With 3 nodes, the quorum is (3/2) + 1 = 2. When node A restarts, the cluster temporarily has only 2 active nodes (B and C). This is still sufficient to meet the quorum requirement. Node A, now running the new version, rejoins the cluster and participates in consensus. It might become a follower or, depending on timing and election logic, could even become the leader if it’s the first to establish sufficient connections.
The key is that the cluster maintains its quorum throughout the restart of a single node. Clients connected to node A will experience a brief interruption as the connection drops and they reconnect to another available node (B or C). Clients connected to B or C will continue to operate normally, though they might see a slight delay in routing messages through node A if they were previously using it.
This strategy allows you to upgrade each node one by one. After node A is upgraded, you’d then move to node B, and finally node C, each time restarting a single node while the others maintain quorum.
The core mechanism is the Raft consensus algorithm. NATS uses Raft for cluster state management, including leader election and cluster membership. When a node restarts, it drops out of the consensus group. The remaining nodes continue to operate, and if they still constitute a majority (quorum), they can elect a new leader if the old one was the departed node. The restarted node then rejoins the cluster, re-syncs its state, and participates in the consensus process again.
The cluster configuration block in your nats.conf is critical here.
cluster {
name = "nats-cluster"
listen = "0.0.0.0:6222"
routes = [
"nats://nodeB:6222",
"nats://nodeC:6222"
]
# No specific "graceful shutdown" settings here,
# it's implicit in the Raft protocol's handling of node loss/rejoin.
}
The routes directive tells each server how to find its peers. When a node is down, the remaining nodes will attempt to reconnect to it. The listen directive is the address other nodes will use to connect to this one.
The "downtime" experienced is primarily for clients directly connected to the node being restarted. They will lose their connection and must reconnect. This reconnection is usually handled by client libraries automatically, but it’s not instantaneous. For internal cluster operations, like leader election or state updates, the impact is minimized as long as quorum is maintained.
A subtle point often missed is the impact on high-availability configurations within a single NATS client connection. If a client is using multiple URLs for its connection attempt (e.g., nats://nodeA:4222,nats://nodeB:4222), and node A is restarted, the client will automatically attempt to connect to node B. However, if the client library is single-threaded or has a slow reconnection timeout, there can still be a perceived "pause" in message delivery from that client’s perspective.
The next challenge you’ll likely face after mastering rolling restarts is understanding how NATS handles network partitions within the cluster and how to recover from them.