Monday, February 28, 2011

Running an S4 Application

As mentioned in my previous post, this post is an attempt to help you run the sample S4 appilcations open sourced by Yahoo.
  • Linux (tried on ubuntu 9.04)
  • Java 1.6
  • Maven
  • S4 Communication Layer
  • S4 Core

Setting up the S4 environment

In order to build an S4 application of your own, you need to setup the S4 core , comm and examples(optional) directories as follows:

1. Create a directory /S4/s4image at some location of your choice (assuming  /home/abc)
  •  mkdir S4;cd S4/;mkdir s4image;cd s4image/
2.   In this s4image directory, we can either download the S4 core tarball or build S4 core ourself. Going by the latter option,
  •     git clone
  •     git clone
   You can check core and comm folders created in the s4image directory.

3. Building the comm and core folder
  •     cd comm
  •      mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=zookeeper -Dversion=3.1.1 -Dpackaging=jar -Dfile=lib/zookeeper-3.1.1.jar
  •     mvn install
  •     cd ../core
  •      mvn install:install-file -DgroupId=com.esotericsoftware -DartifactId=kryo -Dversion=1.01 -Dpackaging=jar -Dfile=lib/kryo-1.01.jar
  •     mvn install:install-file -DgroupId=com.esotericsoftware -DartifactId=reflectasm -Dversion=0.8 -Dpackaging=jar -Dfile=lib/reflectasm-0.8.jar
  •      mvn install:install-file -DgroupId=com.esotericsoftware -DartifactId=minlog -Dversion=1.2 -Dpackaging=jar -Dfile=lib/minlog-1.2.jar
  •     mvn assembly:assembly install
After successfully running these, you can locate the /core/target/s4_core-
  •     cd target/
  •     tar xzf s4_core-*.tar.gz
4. Building the examples folder
  •     cd /home/abc/S4/s4image
  •     git clone
An examples directory should have been created along with comm and core.

Running the examples

To have an understanding of the structure of an S4 app, running a few examples is recommended. For a start, try to run the speech01 example:
  •     cd /home/abc/S4/s4image/examples/speech01
  •     mvn assembly:assembly install
 will create a target folder in speech01
  •     cd /home/abc/S4/s4image/core/target/s4_apps/
  •     tar xzf /home/abc/S4/s4image/examples/speech01/target/speech01-*.tar.gz
  •     cd /home/abc/S4/s4image/core/target/bin
  •     ./ &
  •     head -10 /home/abc/S4/s4image/examples/testinput/speeches.txt | \       ./ -x -r 2 -u /home/abc/S4/s4image/core/target/s4_apps/speech01/lib/speech01- -

In my next post, would be on "how to build your own S4 application". Any suggestions would be highly appreciated.

1 comment: