Finding out that your production database isn’t functioning is the most terrible and disastrous thing that can happen. The most challenging part comes when you need to upgrade the node in a MongoDB Cluster without downtime while keeping everything else running smoothly.
Imagine a scenario where you have 3 Node Database Cluster like MongoDB and you receive an alert saying that the node is OutofSync. Although the system will continue to operate normally because a replica was set up and two members were still operational. Since you have a 3 Node Setup, so there are less chances of any downtime but still, due to high traffic, there will always be a chance where our whole cluster can come down as it will be running on just 2 nodes.
Let’s see how we can upgrade/recover the node in the MongoDB Cluster without downtime.
To upgrade a node in a MongoDB cluster without downtime, we must use the replica set functionality to do a “rolling upgrade,” which involves stepping down the primary node, upgrading it, and then allowing the replica set to choose a new primary from the remaining active 2 nodes. This will enable us to upgrade one node at a time while the other nodes continue to function and serve requests, thus preventing disruption to our application.
To ensure a seamless node upgrade and maintain high availability, it is crucial to monitor your MongoDB cluster’s performance throughout the process. For detailed insights, check out this MongoDB monitoring guide.
Step-by-Step Guide
Step 1: Check the Replica Set Status.
Log in to one of the nodes and check the status of the replica set:
mongo -u <USERNAME> -p
rs.status()Ensure all nodes are healthy and synchronized. Look for the following:
- Primary: The primary node is marked as the leader.
- Secondaries: Both secondary nodes are in the “SECONDARY” state.
Step 2: Upgrade the Secondary Nodes
Secondary nodes can be upgraded one at a time without impacting the cluster’s availability. Follow these steps:
Step 2: Identify a Secondary Node
Use the rs.status() output to identify one of the secondary nodes.
Step 3: Lower the votes and priority for the Secondary Node
Lower the votes and priority for the Secondary Node you are upgrading so that it cannot participate in the election process and then remove the node using the Shutdown command.
Please note that in MongoDB, the cfg.members array uses zero-based indexing. This means the first member in the array is accessed with cfg.members[0], the second with cfg.members[1], and so on.
Please select the index of the secondary node you are upgrading and run these commands from inside your primary node.
cfg = rs.conf()
cfg.members[1].priority = 0
cfg.members[1].votes = 0
rs.reconfig(cfg)Now, go to SECONDARY NODE and run the following commands to shut down the Server.
use admin;
db.shutdownServer();Also, stop the mongo serve using:
systemctl stop mongod.serviceSince the SECONDARY has been shut down, you can now upgrade the VM or perform the actions you want.
Once that is done, Add the votes and priority again for the SECONDARY NODES, so that they are eligible to participate in the election and restart the mongod service.
cfg = rs.conf()
cfg.members[1].priority = 1
cfg.members[1].votes = 1
rs.reconfig(cfg)Wait for the lag to clear out and then proceed above steps again for the other node.
Now, perform the same steps for another Secondary Node as well.
Step 4: Step Down the Primary Node
Step down the primary node by increasing the vote of OTHER SECONDARY NODE so that PRIMARY can be elected as SECONDARY.
Run the below commands in PRIMARY NODE.
mongo
use admin
cfg = rs.conf()
cfg.members[1].priority = <HIGHER PRIORITY>
cfg.members[1].votes = 1
rs.reconfig(cfg)In the above command, we are raising the Priority higher and adding votes for SECONDARY Node, so the SECONDARY can now become PRIMARY.
Once we have apply the above config, the PRIMARY should now have become the SECONDARY Node, you can also check the same using rs.status() command.
Now we can initiate the shutdown for the OLD PRIMARY node which has become a NEW SECONDARY NODE.
Also, stop the mongod service as well for NEW SECONDARY NODE.
use admin;
db.shutdownServer();systemctl stop mongod.serviceNow you can perform the steps as needed and follow the above same steps to add the priority and votes back for this node.
Now, since all three 3 Nodes have been upgraded we can ever the Primary status, and elect the OLDER PRIMARY again.
Step 5: Revert Primary Status:
Once, we have upgraded both the nodes, we can promote the CURRENT SECONDARY to PRIMARY back by setting its priority higher.
mongo
use admin
cfg = rs.conf()
cfg.members[0].priority = 3
cfg.members[0].votes = 1
rs.reconfig(cfg)Conclusion
By following the steps in this guide, you can upgrade the node in the MongoDB Cluster without downtime. MongoDB is a great option for high-availability applications because of its replica set architecture, which guarantees ongoing availability during maintenance.
👍 Please share this article if you found it helpful.
Please feel free to share your ideas for improvement with us in the Comment Section.
🤞 Stay tuned for future posts.
Feel free to contact us for any more conversations regarding Cloud Computing, DevOps, etc.
I’m a DevOps Engineer with 3 years of experience, passionate about building scalable and automated infrastructure. I write about Kubernetes, cloud automation, cost optimization, and DevOps tooling, aiming to simplify complex concepts with real-world insights. Outside of work, I enjoy exploring new DevOps tools, reading tech blogs, and play badminton.
