Setup a MongoDB Sharded Cluster

"Sharding = Shared + Nothing"
Database Sharding is a type of database partitioning that separates very large databases into smaller, faster, more easily managed horizontal partitions called data shards.

Sharding in MongoDB

"MongoDB’s sharding system allows users to partition a collection within a database to distribute the collection’s documents across a number of mongod instances or shards. Sharding increases write capacity, provides the ability to support larger working sets, and raises the limits of total data size beyond the physical resources of a single node." as per the MongoDB Manual.

So when the amount of data that needs to be stored exceeds the storage limit of a single physical resource, a sharded cluster needs to be deployed. Every horizontal partition called "shard" stores a part of the entire dataset. Sharding enables distribution of data, but to have this distributed data replicated as well, each shard should be a Replica Set. For more details on Replica Set, refer to my previous post.

Sample Document

 

A Sharded Cluster in MongoDB

The figure above denotes a Sharded Cluster, where each Shard is a Replica Set which ensures that each portion of the data lying on different physical devices has a copy for backup in case of failures. As depicted, here the Document 1 is distributed across all shards.

Deploying a Sharded Cluster

A Sharded Cluster consists of

• 1 config server(3 for production cluster)
• 1 mongos connecting the config server(s) (1 or more for production cluster)
• 1 shard i.e. a replica set of a standalone machine(2+ for production cluster)

For testing purpose you can also deploy all the above three daemons, on the same machine keeping in mind that the ports on which they would run have to be different. Following setups will guide how to assign a different port to each daemon process. Here we are assuming that you are using a single node(for eg. 10.10.10.10) to run all the daemons, in case of multiple machines use their respective ip addresses.

Step 1: Config Server Setup
Follow the Step 1 and 2 as given in the MongoDB Standalone Setup post.
Config Server requires a config db location which can be specified in the startup command using the --dbpath option. Before this, we need to create it and change its permissions and ownership to the user running MongoDB. Use the following command to setup the config db path of your choice. Here taking "/data/configdb"

mkdir -p /data/configdb sudo chmod -R 777 /data/configdb sudo chown groupName:userName /data/configdb -R

Start the Config Server by running the following command on 10.10.10.10

mongod --configsvr --dbpath /data/configdb --port 27018

This would start the config server on the machine at port 27018.

Step 2: MongoS Instance Setup
Here we only need to follow the Step 1 and 2 as given in the MongoDB Standalone Setup post. No other MongoS specific configuration needs to be done. To start MongoS Server run

mongos --configdb 10.10.10.10:27018 --port 27019

Here, the --configdb option lists the ip:port of the config servers that the mongos server is supposed to connect to and on what port they are running. In case of multiple config servers the command would be like:

mongo --configdb 10.10.10.10:27018,11.11.11.11:27017 --port 27019

Step 3: Setting up the Shards
As mentioned above, each shard is a Replica Set and every member of a Replica Set is a MongoD Instance. For setting up a Shard you need to create a Replica Set first, follow my previous post for achieving the same.
For testing purpose, we can also setup a standalone MongoD Server to add to the cluster as a shard. Follow the steps for setting up a standalone MongoD Instance as given in this post.
Having started the MongoD daemon using

/bin/mongod --config /pathToFile/mongod.conf

on 10.10.10.10 all you need is to add this node as a shard to the cluster.

Step 4: Adding Shards to the Cluster
For adding shards, you need a MongoS terminal which can be opened by hitting the following command:

mongo mongos_instance_ip:mongos_port

where the "mongos_instance_ip" is the ip address of the machine on which MongoS daemon is running at port "mongos_port".
In our example the command would be

mongo 10.10.10.10:27019

which would open a mongos terminal.
In the terminal fire commands like 

sh.addShard("shard_ip:mongod_port");

to add as many shards you want.
For eg, here since we have a standalone mongod instance as a shard, we will use

sh.addShard("10.10.10.10:27017");

You can check the successful execution of the above command using 

sh.status()

in the mongos terminal.

Note: After version 2.0.3, when adding a Replica Set as a shard to the cluster, it is not mandatory to add all nodes in the replica set using sh.addShard(). Adding one of the nodes of the Replica Set would enable mongos to discover all other members and add them to the shard automatically.
For eg. if you have a replica set named "replSet01" which has 2 members 10.10.10.1 and 10.10.10.2, to add this replica set as a shard to the above cluster you would use

sh.addShard("replSet01/10.10.10.1:27017");

and this would automatically add both 10.10.10.1 and 10.10.10.2 as a member of the shard being added.

Before version 2.0.3, it is required to specify all the replica set members to be added to the sharded cluster like

sh.addShard("replSet01/10.10.10.1:27017,10.10.10.2:27017");

Hope this post was a helping hand to you in testing out Sharding in MongoDB.

All the Best.

Setup a MongoDB Replica Set

Database replication is a service for shipping changes to your database, off to a copy housed on another server, potentially even in another data center.It ensures redundancy, backup, and automatic failover.
Replication occurs through groups of servers known as replica sets, each of which stores the same dataset as the other members of the replica set.

Replication in MongoDB

In MongoDB, replication is implemented in two forms:

1. Replica Sets: Group of replication nodes containing one primary node and multiple secondary nodes. Primary accepts write operations, and secondaries are the replicating members. MongoDB’s replica sets provide automated failover. If a primary fails, the remaining members will automatically try to elect a new primary.
2. Master-Slave Architecture: When the number of the replica nodes need to be more than 12, then Master-Slave architecture comes in. It is similar to Replica Sets except that it does not provide automated failover.

The figure below showcases the concept of Replica Sets in MongoDB, where one node is primary and rest all are secondaries. Also all of them are storing the same documents i.e. Doc 1 and 2

 

 Replication in MongoDB

Deploying a Replica Set

With trivial knowledge gathered about Replica Sets in MongoDB, try deploying one using the steps below:

Step 1: Follow Step 1,2 and 3 for setting up MongoDB on all the nodes of the Replica Set as given in my previous post.

Step 2: Create a conf file (for example "mongod.conf") at a location owned and accessible by the user with the following content. You can configure "port" and name your replica set by setting the field "replSet" to that name which would be same on all the nodes of the replica set.

dbpath=/var/lib/mongodb logpath=/var/log/mongodb/mongodb.log logappend=true nojournal = true port = 27017 bind_ip = machine_ip (for eg. 100.11.11.11) fork = true replSet = rsName

Step 3: Start the MongoD daemon on each node of the Replica Set using:

/bin/mongod --config /pathToFile/mongod.conf

Step 4: Open a mongo terminal on the node which you want to be the primary node using: 

bin/mongo --host ip_of_the_machine

To make this node the primary run the following command in the opened mongo terminal :

rs.initiate()

Step 5: To add the other nodes, run the add commands in the opened mongo terminal on the primary node:

rs.add("ip_of_secondary_node_1") rs.add("ip_of_secondary_node_2") and so on.

Check the status of the replica set using:

rs.conf()

Add secondaries to the Replica Set using a JavaScript

After initializing the primary using "rs.initiate()" add the secondaries using :

mongo primary_ip:mongod_port /path_to_js_file

where the .js file contains 

rs.add('ip_of_sec1:port_of_mongod'); rs.add('ip_of_sec2:port_of_mongod'); . . . rs.add('ip_of_secN:port_of_mongod');

To check, you can open the mongo terminal on the primary  node and run the "rs.conf()" command to see the status post running the JavaScript file.

Hope that the above steps helped you create a Replica Set successfully. My next post would be on "Creating a MongoDB Sharded Cluster".

MongoDB Installation: Standalone Mode

"Mongo" is a short for "Humongous" meaning "Huge".
By the name it goes that, MongoDB is a database phenomenal at dealing with "huge" piles of data.

As per the MongoDB Documentation,
"MongoDB is a scalable, high-performance, open source NoSQL database."

Installing MongoDB in standalone is guileless. It is also achievable using apt-get command in case of Linux. This post is for you if you wish to know what configuration files are used in MongoDB and if you intend to go a longer way rather than just a standalone installation and a casual try out of MongoDB.

Tested On
I have tested the following steps on Ubuntu 10.04 and MongoDB 2.2.2. 

Steps of Installation Step 1: Every version of MongoDB setups comes in two flavors:

• 32-bit
• 64-bit

The 32-bit builds run on both 32 and 64-bit machines while 64-bit build runs only on the 64-bit machines. However, as per the MongoDB manual,
"32-bit builds are limited to around 2GB of data. In general you should use the 64 bit builds. The 32 bit binaries are ok for replica set arbiters and mongos but not for production mongod's."

So it would be good to figure out the machine configuration before deployment.
Run the following command to know your machine

uname -m

it would return "i686" for 32 bit machines and "x86_64" for 64 bit machines.

Step 2: According to the Step 1 results download the setup from
http://www.mongodb.org/downloads
Untar the downloaded setup using 

tar  -zxvf  location_of_setup

Step 3: MongoDB requires 2 directories for it's functioning. They are:
1. Directory for storing the logs
2. Another directory for storing the data i.e. dbpath (by default /data/db)
You can use any suitable location on your machine to create these directories. For example, here we'll be taking
Log Directory: /var/log/mongodb
DBPath: /var/lib/mongodb

Create the above 2 directories as below:

mkdir /var/lib/mongodb mkdir /var/log/mongodb

Change their permissions as below:

sudo chmod -R 777 /var/lib/mongodb sudo chmod -R 777 /var/log/mongodb

Change their ownership to the user (root or non-root) as below:

sudo chown groupName:userName /var/lib/mongodb -R sudo chown groupName:userName /var/log/mongodb -R

Also change the ownership of the directory where mongodb's setup is copied using similar commands as above.

Step 4: Create a conf file (for example "mongod.conf") at a location owned and accessible by the user.

dbpath=/var/lib/mongodb logpath=/var/log/mongodb/mongodb.log                     logappend=true nojournal=true

Step 5: Start the MongoD daemon using:

/bin/mongod --config /path_to_conf_file/mongod.conf

  
Step 6: Verify whether the MongoD daemon is running or not using:

sudo netstat -nlp | grep mongod

 Step 7: Start using mongodb by opening the mongo shell using:

/bin/mongo

This completes the installation process of MongoDB in standalone mode. In my next post, I'll be catering steps on how to setup a MongoDB Replica Set.