Showing posts with label Zookeeper. Show all posts
Showing posts with label Zookeeper. Show all posts

Friday, June 26, 2015

Start-up script for an installed Apache Zookeeper Cluster

If you have an installed Zookeeper-3.3.5 cluster, this script will save you from manually visiting each node and starting the zkServer there. All you have to do is grab a remote machine and run the script. The script will ask for the required information and your cluster would be up. Also, the script should work equally fine for both Ubuntu and CentOS.


#!/bin/bash

# Directory Name
echo "Enter the zookeeper setup location :"
read -e zookeeperSetupLocation

# Read the ips and the usernames of all the machines in the zookeeper cluster
echo "Enter the number of machines in the cluster";
read -e n;
for ((  i = 1 ;  i <= n;  i++  ))
do
echo "Enter the IP of cluster machine $i:"
read -e zkServer;
zkServerIPs[i]=$zkServer
echo "Enter the username of machine ${zkServerIPs[i]}"
read -e username
zkServerUsernames[i]=$username
done

# Start up the cluster
for ((  i = 1 ;  i <= n;  i++  ))
do
ssh ${zkServerUsernames[i]}@${zkServerIPs[i]} "$zookeeperSetupLocation/zookeeper-3.3.5/bin/zkServer.sh start;"
done


Thanks !!

Installation Script for Apache Zookeeper-3.3.5 on Linux

One of my previous blogs describes how to setup a Zookeeper cluster manually. Here's a quick fix: an installation script for the same. You need to run the following script (after storing the content in a .sh file) on your machine and you can install a zookeeper cluster on a set of remote machines. All you need as prerequisite on those machines is the zookeeper-3.3.5 setup at a common location. You can also use this script for other versions, with a bit of modification to the script(replacing the version used in the script to yours. It should have not been hard coded, I know.. My bad).



#!/bin/bash

zkServerEntryPart1="server.";
zkServerEntryPart2="=zoo";
zkServerEntryPart3=":2888:3888NEW_LINE";
zkServerEntry="";

# Local FS Setup location
echo "Enter the path of the folder in which the zookeeper setup is stored. For example '/home/abc/setups'"
read -e setupsLocation
# Directory Name
echo "Enter the directory path where zookeeper is to be installed : "
read -e zookeeperSetupLocation

# Read the number of zookeeper servers
echo "Enter the number of machines in the zookeeper cluster : ";
read -e n;

# Read the zookeeper server details
for ((  i = 1 ;  i <= n;  i++  ))
do
# obtain the zookeeper server ips of all machines in the cluster
echo "Enter the IP of zookeeper machine $i:"
read -e zookeeperServer;
zookeeperServerIPList[i]=$zookeeperServer
temp=$zkServerEntryPart1""$i""$zkServerEntryPart2""$i""$zkServerEntryPart3;
zkServerEntry=$zkServerEntry""$temp;

# obtain the usernames for all the zookeeper servers
echo "Enter the username of machine ${zookeeperServerIPList[i]}"
read -e username
userNameList[i]=$username
done

# Copy the setup to all the storm machines
for ((  i = 1 ;  i <= n;  i++  ))
do
echo "Enter the data directory location of zookeeper for the machine ${zookeeperServerIPList[i]}"
read -e dataDir
     
# create the required folders on the machines
ssh -t ${userNameList[i]}@${zookeeperServerIPList[i]} "if [ ! -d $zookeeperSetupLocation ]; then
sudo mkdir $zookeeperSetupLocation;
sudo chown -R ${userNameList[i]}: $zookeeperSetupLocation;
fi
if [ ! -d $dataDir ]; then
sudo mkdir $dataDir;
sudo chown -R ${userNameList[i]}: $dataDir;
fi"

# copy the zookeeper setup at the specified location on the machines
scp -r -q $setupsLocation/zookeeper-3.3.5 ${userNameList[i]}@${zookeeperServerIPList[i]}:$zookeeperSetupLocation/zookeeper-3.3.5

# create and configure the 'zoo.cfg' and 'myid' files
ssh -t ${userNameList[i]}@${zookeeperServerIPList[i]} "touch $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
echo -e "dataDir=$dataDir" >> $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
echo -e "syncLimit=2" >> $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
echo -e "initLimit=5" >> $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
echo -e "clientPort=2181" >> $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
echo  "$zkServerEntry" >> $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
sed -i 's/NEW_LINE/\n/g' $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg
touch $dataDir/myid;                           
echo -e $i >> $dataDir/myid;"

# update the /etc/hosts file
hostFileEntry=${zookeeperServerIPList[i]}" zoo"$i;
for ((  j = 1 ;  j <= n;  j++  ))
do
ssh -t ${userNameList[j]}@${zookeeperServerIPList[j]} "sudo cp /etc/hosts /etc/hosts.bak;
sudo cp /etc/hosts /etc/hosts1;
sudo chmod 777 /etc/hosts1;
sudo echo -e "$hostFileEntry" >> /etc/hosts1;
sudo mv /etc/hosts1 /etc/hosts;"
done
done

Hope it helped. My next blog post is about a small script to start-up the installed zookeeper cluster.

Wednesday, November 21, 2012

Setting up a Zookeeper Cluster

ZooKeeper is a distributed, open-source and high-performance coordination service for distributed applications. Setting up a running Zookeeper cluster is a prerequisite to kick off many installations.
This post would direct you on how to setup a Zookeeper Cluster.

 

Prerequisites

Java JDK installed on all nodes of the cluster.

 

Setting up the cluster

The following installation steps have been tested for 10.04, 10.10, 11.04, 12.04 versions of Ubuntu.

1. Obtain the zookeeper setup at some location. Setup can be downloaded from :
http://hadoop.apache.org/zookeeper/releases.html

2. Create a file with any name (eg. zoo.cfg) in the conf folder of the copied setup and write in it


dataDir=/var/zookeeper/                                                                   
clientPort=2181
initLimit=5
syncLimit=2
server.server1=zoo1:2888:3888                               
server.server2=zoo2:2888:3888
server.server3=zoo3:2888:3888                                                         

Here 2888 and 3888 ports cannot be modified but the server.id(server.server1) and the zkServerName(zoo1) can be changed by the user. Using the above entries as sample entries, next it is required that a file named “myid” be created in the path specified in dataDir which contains just one entry which is of the server id. So the first system of the cluster would have a file named "myid" created at the path specified in dataDir containing server1 and so on i.e.
To make it more clear, if we are using 3 systems with IP 192.192.192.191, 192, 193
and zoo1 would designate 192.192.192.191, zoo2 would designate 192.192.192.192, zoo3 would designate 192.192.192.193
then
the machine 192.192.192.191 should contain a file called myid at /var/zookeeper/ (or the value of dataDir specified in zoo.cfg) containing the following entry
server1
Similarly machines 192.192.192.192 and 192.192.192.193 should have entries server2 and server3 respectively.

3. Update the /etc/hosts file on each machine to add the host names being used in the zookeeper configuration. This is needed so as to make it understandable that zoo1, zoo2 and zoo3 refer to which systems.
Post-updation the /etc/hosts file on each system in the cluster would have a similar set of entries like :

192.192.192.191   zoo1
192.192.192.192   zoo2                                                                     
192.192.192.193   zoo3

4. This completes the configuration part, next cd to the zookeeper home and start the cluster by running

bin/zkServer.sh start                                                                          
command on each system.
And you have a running Zookeeper cluster at your disposal. 
Good Luck !!!