Showing posts with label Ubuntu. Show all posts
Showing posts with label Ubuntu. Show all posts

Friday, June 26, 2015

Start-up script for an installed Apache Storm Cluster

If you have installed a Storm cluster using my shell scripts in the previous blogs or even otherwise, this script will save you from manually visiting each node and starting the appropriate service(nimbus/supervisor/ui) there. All you have to do is grab a remote machine and run the script. The script will ask for the required information and your cluster would be up. Also, the script should work equally fine for both Ubuntu and CentOS.


#!/bin/bash

# Directory Name
echo "Enter the relative path to the storm setup on the machines (For example, /opt/stormSetup/storm-0.9.0-wip4):"
read -e stormDir

# Read usernames and ips of all the storm cluster nodes
echo "Enter the IP of nimbus machine :"
read -e stormNimbus;
clusterMachineIPs[0]=$stormNimbus

echo "Enter the username of nimbus :"
read -e usernameNimbus
clusterMachineUsernames[0]=$usernameNimbus

# Read supervisor
echo "Enter the number of supervisor machines";
read -e n;
for ((  i = 1 ;  i <= n;  i++  ))
do
echo "Enter the IP of storm supervisor machine $i:"
read -e stormSupervisor;
clusterMachineIPs[i]=$stormSupervisor
echo "Enter the username of machine ${clusterMachineIPs[i]}"
read -e username
clusterMachineUsernames[i]=$username
done

sshpass -p root ssh -o StrictHostKeyChecking=no  $usernameNimbus@$stormNimbus $stormDir/bin/storm nimbus&

# Start the supervisor nodes
for ((  i = 1 ;  i <= n;  i++  ))
do
sshpass -p root ssh -o StrictHostKeyChecking=no  ${clusterMachineUsernames[i]}@${clusterMachineIPs[i]} $stormDir/bin/storm supervisor&
done

# Start the UI on the nimbus machine
sshpass -p root ssh -o StrictHostKeyChecking=no  $usernameNimbus@$stormNimbus $stormDir/bin/storm ui&


Visit your UI on the browser after a few minutes. Hope it shows up fine. Cheers!

Installation Script for Apache Storm on Ubuntu

One of my blogs here, describes steps for manual installation of a Storm cluster. To intensify the convenience factor for you, here's an installation script that you can use for setting up a Storm cluster on Linux machines. Although the script should work for older versions of Apache Storm, it has been tested for storm-0.9.0-wip4. The script has embedded descriptive messages for each input it expects from you. The installation would be done in the '/opt' folder of the machines in a sub-directory of your choice. Make sure the user installing the cluster has admin rights on the /opt folder. The script also takes care of installing all the required dependencies. To use the script for versions other than the supported one, you need to make changes to the script and replace the "storm-0.9.0-wip4" occurrence with your storm version.


#!/bin/bash
# Local FS Setups location
echo "Enter the location of the setups folder. For example '/home/abc/storminstallation/setups'"
read -e setupsLocation
# Directory Name
echo "Enter the directory name"
read -e realTimePlatformDir
rtpLocalDir="\/opt\/$realTimePlatformDir\/storm\/storm_temp"
rtpLocalDirMake=/opt/$realTimePlatformDir/storm/storm_temp
echo $rtpLocalDir;
echo "Enter the IP of nimbus machine :"
read -e stormNimbus;
array[0]=$stormNimbus


# Read supervisor
echo "Enter the number of supervisor machines";
read -e n;
for ((  i = 1 ;  i <= n;  i++  ))
do
echo "Enter the IP of storm supervisor machine $i:"
read -e stormSupervisor;
array[i]=$stormSupervisor
done

# Read zookeeper
echo "Enter the number of machines in the zookeeper cluster";
read -e m;
for ((  i = 1 ;  i <= m;  i++  ))
do
echo "Enter the IP of zookeeper machine $i:"
read -e zkServer;
zkEntry="- \""$zkServer"\""
zKArray=$zKArray","$zkEntry
done

# Copy the required setups to all the storm machines
for ((  i = 1 ;  i <= n+1;  i++  ))
do
echo "Enter the username of machine ${array[i-1]}"
read -e username
echo "Username:"$username
if [ $username == 'root' ]; then
echo 'root';
yamlFilePath="/root/.storm";
else
echo $username;
yamlFilePath="/home/$username/.storm";
fi
echo "the storm.yaml file would be formed at : $yamlFilePath";
echo "Enter the value for JAVA_HOME to be set on the machine ${array[i-1]}"
read -e javaHome;
echo 'JAVA_HOME would be set to :'$javaHome;
ssh -t $username@${array[i-1]} "if [ ! -d /opt/$realTimePlatformDir ]; then
           sudo mkdir /opt/$realTimePlatformDir;
           sudo chown -R $username: /opt/$realTimePlatformDir;
           mkdir /opt/$realTimePlatformDir/storm;
           mkdir $rtpLocalDirMake;
           mkdir $yamlFilePath;
        fi"
     
scp -r -q $setupsLocation/storm-0.9.0-wip4 $username@${array[i-1]}:/opt/$realTimePlatformDir/storm/storm-0.9.0-wip4
     
ssh -t $username@${array[i-1]} "sed -i 's/ZOOKEEPER_IPS/$zKArray/g' /opt/$realTimePlatformDir/storm/storm-0.9.0-wip4/conf/storm.yaml;
sed -i 's/,/\n/g' /opt/$realTimePlatformDir/storm/storm-0.9.0-wip4/conf/storm.yaml;
sed -i 's/NIMBUS_IP/$stormNimbus/g' /opt/$realTimePlatformDir/storm/storm-0.9.0-wip4/conf/storm.yaml;
sed -i 's/LOCAL_DIR/$rtpLocalDir/g' /opt/$realTimePlatformDir/storm/storm-0.9.0-wip4/conf/storm.yaml;
cp /opt/$realTimePlatformDir/storm/storm-0.9.0-wip4/conf/storm.yaml $yamlFilePath;
sudo apt-get install git;
sudo apt-get install uuid-dev;"

ssh -t $username@${array[i-1]} "cd /opt/$realTimePlatformDir/storm;
                        wget http://download.zeromq.org/zeromq-2.1.7.tar.gz;
                        tar -xzf zeromq-2.1.7.tar.gz
                        cd zeromq-2.1.7                    
                        ./configure
                        make
                        sudo make install

                        cd ..
                        export JAVA_HOME=$javaHome;
                        echo $JAVA_HOME;
                        git clone https://github.com/nathanmarz/jzmq.git
                        cd jzmq
                        ./autogen.sh
                        ./configure
                        make
                        sudo make install"

done


Yep Done! Hope it helped. My next post shares the installation script for CentOS and the next to it a small start up script for the installed Storm cluster.

Monday, June 15, 2015

Installing R on Linux

I was a bit skeptical about writing this post due to the scarcity of content, but anyhow you know what I chose.

So, installing R on Ubuntu is all about the following two steps:


sudo apt-get install r-base
sudo apt-get install littler                                                                                                                        

To check you installation, type-in “R” in your terminal, which should open an R shell. Here you can try installing a package for example:


> install.packages("randomForest")
or
> install.packages("stringr")                                                                                                                   

If all worked fine, your machine has got R correctly installed. The same way you can install R on a CentOS machine using the following commands:


sudo yum install r-base
sudo yum install littler                                                                                                                             

Rest follow the same steps.

Hope this small piece of information is helpful to some of you.