Showing posts with label Shell Script. Show all posts
Showing posts with label Shell Script. Show all posts

Friday, June 26, 2015

Start-up script for an installed Apache Storm Cluster

If you have installed a Storm cluster using my shell scripts in the previous blogs or even otherwise, this script will save you from manually visiting each node and starting the appropriate service(nimbus/supervisor/ui) there. All you have to do is grab a remote machine and run the script. The script will ask for the required information and your cluster would be up. Also, the script should work equally fine for both Ubuntu and CentOS.


#!/bin/bash

# Directory Name
echo "Enter the relative path to the storm setup on the machines (For example, /opt/stormSetup/storm-0.9.0-wip4):"
read -e stormDir

# Read usernames and ips of all the storm cluster nodes
echo "Enter the IP of nimbus machine :"
read -e stormNimbus;
clusterMachineIPs[0]=$stormNimbus

echo "Enter the username of nimbus :"
read -e usernameNimbus
clusterMachineUsernames[0]=$usernameNimbus

# Read supervisor
echo "Enter the number of supervisor machines";
read -e n;
for ((  i = 1 ;  i <= n;  i++  ))
do
echo "Enter the IP of storm supervisor machine $i:"
read -e stormSupervisor;
clusterMachineIPs[i]=$stormSupervisor
echo "Enter the username of machine ${clusterMachineIPs[i]}"
read -e username
clusterMachineUsernames[i]=$username
done

sshpass -p root ssh -o StrictHostKeyChecking=no  $usernameNimbus@$stormNimbus $stormDir/bin/storm nimbus&

# Start the supervisor nodes
for ((  i = 1 ;  i <= n;  i++  ))
do
sshpass -p root ssh -o StrictHostKeyChecking=no  ${clusterMachineUsernames[i]}@${clusterMachineIPs[i]} $stormDir/bin/storm supervisor&
done

# Start the UI on the nimbus machine
sshpass -p root ssh -o StrictHostKeyChecking=no  $usernameNimbus@$stormNimbus $stormDir/bin/storm ui&


Visit your UI on the browser after a few minutes. Hope it shows up fine. Cheers!

Installation Script for Apache Storm on CentOS

CentOS and Ubuntu and two famous Linux distribution used pretty widely. My last post shares an installation script for Storm cluster over Ubuntu machines and this one is for CentOS. The few usage rules are just the same as for Ubuntu. I'll recite here. Although the script should work for older versions of Apache Storm, it has been tested for storm-0.9.0-wip7. The script has embedded descriptive messages for each input it expects from you. The installation would be done in the '/opt' folder of the machines in a sub-directory of your choice. Make sure the user installing the cluster has admin rights on the /opt folder. The script also takes care of installing all the required dependencies. To use the script for versions other than the supported one, you need to make changes to the script and replace the "storm-0.9.0-wip7" occurrence with your storm version.


#!/bin/bash
# Local FS Setups location
echo "Enter the location of the setups folder. For example '/home/abc/storminstallation/setups'"
read -e setupsLocation
# Directory Name
echo "Enter the directory name"
read -e realTimePlatformDir
rtpLocalDir="\/opt\/$realTimePlatformDir\/storm\/storm_temp"
rtpLocalDirMake=/opt/$realTimePlatformDir/storm/storm_temp
echo $rtpLocalDir;
echo "Enter the IP of nimbus machine :"
read -e stormNimbus;
array[0]=$stormNimbus


# Read supervisor
echo "Enter the number of supervisor machines";
read -e n;
for ((  i = 1 ;  i <= n;  i++  ))
do
echo "Enter the IP of storm supervisor machine $i:"
read -e stormSupervisor;
array[i]=$stormSupervisor
done

# Read zookeeper
echo "Enter the number of machines in the zookeeper cluster";
read -e m;
for ((  i = 1 ;  i <= m;  i++  ))
do
echo "Enter the IP of zookeeper machine $i:"
read -e zkServer;
zkEntry="- \""$zkServer"\""
zKArray=$zKArray","$zkEntry
done

# Copy the required setups to all the storm machines
for ((  i = 1 ;  i <= n+1;  i++  ))
do
echo "Enter the username of machine ${array[i-1]}"
read -e username
echo "Username:"$username
if [ $username == 'root' ]; then
echo 'root';
yamlFilePath="/root/.storm";
else
echo $username;
yamlFilePath="/home/$username/.storm";
fi
echo "the storm.yaml file would be formed at : $yamlFilePath";
echo "Enter the value for JAVA_HOME to be set on the machine ${array[i-1]}"
read -e javaHome;
echo 'JAVA_HOME would be set to :'$javaHome;
ssh -t $username@${array[i-1]} "if [ ! -d /opt/$realTimePlatformDir ]; then
    sudo mkdir /opt/$realTimePlatformDir;
    sudo chown -R $username: /opt/$realTimePlatformDir;
    mkdir /opt/$realTimePlatformDir/storm;
    mkdir $rtpLocalDirMake;
    mkdir $yamlFilePath;
  fi"
     
scp -r -q $setupsLocation/storm-0.9.0-wip7 $username@${array[i-1]}:/opt/$realTimePlatformDir/storm/storm-0.9.0-wip7
     
ssh -t $username@${array[i-1]} "sed -i 's/ZOOKEEPER_IPS/$zKArray/g' /opt/$realTimePlatformDir/storm/storm-0.9.0-wip7/conf/storm.yaml;
sed -i 's/,/\n/g' /opt/$realTimePlatformDir/storm/storm-0.9.0-wip7/conf/storm.yaml;
sed -i 's/NIMBUS_IP/$stormNimbus/g' /opt/$realTimePlatformDir/storm/storm-0.9.0-wip7/conf/storm.yaml;
sed -i 's/LOCAL_DIR/$rtpLocalDir/g' /opt/$realTimePlatformDir/storm/storm-0.9.0-wip7/conf/storm.yaml;
cp /opt/$realTimePlatformDir/storm/storm-0.9.0-wip7/conf/storm.yaml $yamlFilePath;
sudo yum install git;
sudo yum install libuuid-devel;"

ssh -t $username@${array[i-1]} "cd /opt/$realTimePlatformDir/storm;
wget http://download.zeromq.org/zeromq-2.1.7.tar.gz;
tar -xzf zeromq-2.1.7.tar.gz
cd zeromq-2.1.7                    
./configure
make
sudo make install

cd ..
export JAVA_HOME=$javaHome;
echo $JAVA_HOME;
git clone https://github.com/nathanmarz/jzmq.git
cd jzmq
./autogen.sh
./configure
make
sudo make install"

done


Hope this helps. My next post shares small start-up script for the installed Storm cluster.

Start-up script for an installed Apache Zookeeper Cluster

If you have an installed Zookeeper-3.3.5 cluster, this script will save you from manually visiting each node and starting the zkServer there. All you have to do is grab a remote machine and run the script. The script will ask for the required information and your cluster would be up. Also, the script should work equally fine for both Ubuntu and CentOS.


#!/bin/bash

# Directory Name
echo "Enter the zookeeper setup location :"
read -e zookeeperSetupLocation

# Read the ips and the usernames of all the machines in the zookeeper cluster
echo "Enter the number of machines in the cluster";
read -e n;
for ((  i = 1 ;  i <= n;  i++  ))
do
echo "Enter the IP of cluster machine $i:"
read -e zkServer;
zkServerIPs[i]=$zkServer
echo "Enter the username of machine ${zkServerIPs[i]}"
read -e username
zkServerUsernames[i]=$username
done

# Start up the cluster
for ((  i = 1 ;  i <= n;  i++  ))
do
ssh ${zkServerUsernames[i]}@${zkServerIPs[i]} "$zookeeperSetupLocation/zookeeper-3.3.5/bin/zkServer.sh start;"
done


Thanks !!

Installation Script for Apache Zookeeper-3.3.5 on Linux

One of my previous blogs describes how to setup a Zookeeper cluster manually. Here's a quick fix: an installation script for the same. You need to run the following script (after storing the content in a .sh file) on your machine and you can install a zookeeper cluster on a set of remote machines. All you need as prerequisite on those machines is the zookeeper-3.3.5 setup at a common location. You can also use this script for other versions, with a bit of modification to the script(replacing the version used in the script to yours. It should have not been hard coded, I know.. My bad).



#!/bin/bash

zkServerEntryPart1="server.";
zkServerEntryPart2="=zoo";
zkServerEntryPart3=":2888:3888NEW_LINE";
zkServerEntry="";

# Local FS Setup location
echo "Enter the path of the folder in which the zookeeper setup is stored. For example '/home/abc/setups'"
read -e setupsLocation
# Directory Name
echo "Enter the directory path where zookeeper is to be installed : "
read -e zookeeperSetupLocation

# Read the number of zookeeper servers
echo "Enter the number of machines in the zookeeper cluster : ";
read -e n;

# Read the zookeeper server details
for ((  i = 1 ;  i <= n;  i++  ))
do
# obtain the zookeeper server ips of all machines in the cluster
echo "Enter the IP of zookeeper machine $i:"
read -e zookeeperServer;
zookeeperServerIPList[i]=$zookeeperServer
temp=$zkServerEntryPart1""$i""$zkServerEntryPart2""$i""$zkServerEntryPart3;
zkServerEntry=$zkServerEntry""$temp;

# obtain the usernames for all the zookeeper servers
echo "Enter the username of machine ${zookeeperServerIPList[i]}"
read -e username
userNameList[i]=$username
done

# Copy the setup to all the storm machines
for ((  i = 1 ;  i <= n;  i++  ))
do
echo "Enter the data directory location of zookeeper for the machine ${zookeeperServerIPList[i]}"
read -e dataDir
     
# create the required folders on the machines
ssh -t ${userNameList[i]}@${zookeeperServerIPList[i]} "if [ ! -d $zookeeperSetupLocation ]; then
sudo mkdir $zookeeperSetupLocation;
sudo chown -R ${userNameList[i]}: $zookeeperSetupLocation;
fi
if [ ! -d $dataDir ]; then
sudo mkdir $dataDir;
sudo chown -R ${userNameList[i]}: $dataDir;
fi"

# copy the zookeeper setup at the specified location on the machines
scp -r -q $setupsLocation/zookeeper-3.3.5 ${userNameList[i]}@${zookeeperServerIPList[i]}:$zookeeperSetupLocation/zookeeper-3.3.5

# create and configure the 'zoo.cfg' and 'myid' files
ssh -t ${userNameList[i]}@${zookeeperServerIPList[i]} "touch $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
echo -e "dataDir=$dataDir" >> $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
echo -e "syncLimit=2" >> $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
echo -e "initLimit=5" >> $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
echo -e "clientPort=2181" >> $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
echo  "$zkServerEntry" >> $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg;
sed -i 's/NEW_LINE/\n/g' $zookeeperSetupLocation/zookeeper-3.3.5/conf/zoo.cfg
touch $dataDir/myid;                           
echo -e $i >> $dataDir/myid;"

# update the /etc/hosts file
hostFileEntry=${zookeeperServerIPList[i]}" zoo"$i;
for ((  j = 1 ;  j <= n;  j++  ))
do
ssh -t ${userNameList[j]}@${zookeeperServerIPList[j]} "sudo cp /etc/hosts /etc/hosts.bak;
sudo cp /etc/hosts /etc/hosts1;
sudo chmod 777 /etc/hosts1;
sudo echo -e "$hostFileEntry" >> /etc/hosts1;
sudo mv /etc/hosts1 /etc/hosts;"
done
done

Hope it helped. My next blog post is about a small script to start-up the installed zookeeper cluster.