Showing posts with label S4. Show all posts
Showing posts with label S4. Show all posts

Tuesday, March 8, 2011

Building an S4 Application

Having gathered some knowledge about 'how to run an S4 app' in my previous post, its time we move up to building applications of our own. Before we kick off this task, a precise note on the structure of a basic S4 application is required.

Structure of an S4 application 



 To start, create a similar directory structure in your s4image directory's example folder. Assuming your application name 'appName', we would  take up each segment of the app one by one.
.java files

The most elementary S4 application also requires 2 java classes :
1. Bean class :This class declares variables that represent the event of its type. In your application, you can start with a Login bean having two string variables 'id' and 'pwd' and their getter and setter methods.

2. Receiver class for the Bean :It is significant to point out the action to be taken when an event is fired. For each distinct event, represented by a bean class, there has to be a receiver class that describes how to process the event it is meant for, in its processEvent() function.

public void processEvent(Login data) {
//handle the Login object
}
public String getId() {
return this.getClass().getName();
}

appName_conf.xml

Precisely, this xml defines a set of beans and the stream and the keys for each receiver classes, on which they would operate. Each bean is allocated an id of your choice and the class it represents.
In this example the xml could be as :

<bean id="loginDataCatcher" class="io.s4.example.appName.LoginReceiverPE">
    <property name="keys">
      <list>
        <value>LoginData *</value>
      </list>
    </property> 

</bean>

assembly.xml

This xml is common to all the applications and you only need to replace the 'fileset include' with appName.xml which is indicating the resource directory as follows :

<fileSet>
      <directory>${project.basedir}/src/main/resources</directory>
      <outputDirectory />
      <includes>
    <include>appName_conf.xml</include>
      </includes>
 </fileSet>

pom.xml

Here you only need to replace  <artifactId></artifactId> and <name></name> tag values with appName

<modelVersion>4.0.0</modelVersion>
<groupId>io.s4.examples</groupId>
<artifactId>appName</artifactId>
<packaging>jar</packaging>
<version>0.0.0.1</version>
<name>appName</name>

Once you are done with these files, you can create an input (.txt) file and in case of any doubt you can download the sample application. Though commands for 'Running an S4 app' have been listed in my previous post, here's an apprise listing :
  •     cd /path _to_s4image_dir/examples/appName
  •     mvn assembly:assembly install
 will create a target folder in appName
  •     cd /path _to_s4image_dir/core/target/s4_apps/
  •     tar xzf /path _to_s4image_dir/examples/appName/target/appName-*.tar.gz
  •     cd /path _to_s4image_dir/core/target/bin
  •     ./s4_start.sh &
  •     head -10 /path _input_file_location/xyz.txt | \       ./generate_load.sh -x -r 2 -u/path _to_s4image_dir/core/target/s4_apps/appName/lib/appName-0.0.0.1.jar -
Hope this helped you somehow ! Any suggestions would be highly appreciated.  


Monday, February 28, 2011

Running an S4 Application

As mentioned in my previous post, this post is an attempt to help you run the sample S4 appilcations open sourced by Yahoo.
 
Prerequisites
  • Linux (tried on ubuntu 9.04)
  • Java 1.6
  • Maven
  • S4 Communication Layer
  • S4 Core

Setting up the S4 environment

In order to build an S4 application of your own, you need to setup the S4 core , comm and examples(optional) directories as follows:

1. Create a directory /S4/s4image at some location of your choice (assuming  /home/abc)
  •  mkdir S4;cd S4/;mkdir s4image;cd s4image/
2.   In this s4image directory, we can either download the S4 core tarball or build S4 core ourself. Going by the latter option,
  •     git clone https://github.com/s4/core.git
  •     git clone https://github.com/s4/comm.git
   You can check core and comm folders created in the s4image directory.

3. Building the comm and core folder
  •     cd comm
  •      mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=zookeeper -Dversion=3.1.1 -Dpackaging=jar -Dfile=lib/zookeeper-3.1.1.jar
  •     mvn install
  •     cd ../core
  •      mvn install:install-file -DgroupId=com.esotericsoftware -DartifactId=kryo -Dversion=1.01 -Dpackaging=jar -Dfile=lib/kryo-1.01.jar
  •     mvn install:install-file -DgroupId=com.esotericsoftware -DartifactId=reflectasm -Dversion=0.8 -Dpackaging=jar -Dfile=lib/reflectasm-0.8.jar
  •      mvn install:install-file -DgroupId=com.esotericsoftware -DartifactId=minlog -Dversion=1.2 -Dpackaging=jar -Dfile=lib/minlog-1.2.jar
  •     mvn assembly:assembly install
After successfully running these, you can locate the /core/target/s4_core-0.3.0.0.tar.gz
  •     cd target/
  •     tar xzf s4_core-*.tar.gz
4. Building the examples folder
  •     cd /home/abc/S4/s4image
  •     git clone https://github.com/s4/examples.git
An examples directory should have been created along with comm and core.

Running the examples

To have an understanding of the structure of an S4 app, running a few examples is recommended. For a start, try to run the speech01 example:
  •     cd /home/abc/S4/s4image/examples/speech01
  •     mvn assembly:assembly install
 will create a target folder in speech01
  •     cd /home/abc/S4/s4image/core/target/s4_apps/
  •     tar xzf /home/abc/S4/s4image/examples/speech01/target/speech01-*.tar.gz
  •     cd /home/abc/S4/s4image/core/target/bin
  •     ./s4_start.sh &
  •     head -10 /home/abc/S4/s4image/examples/testinput/speeches.txt | \       ./generate_load.sh -x -r 2 -u /home/abc/S4/s4image/core/target/s4_apps/speech01/lib/speech01-0.0.0.1.jar -

In my next post, would be on "how to build your own S4 application". Any suggestions would be highly appreciated.




Sunday, February 27, 2011

An Introduction to S4

What is S4 ?
S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data, as described by S4.io. 

Need Assessment of S4
The emergence of commercial search engines and new applications such as real-time
search, high frequency trading, and social networks demands a streaming paradigm that can process the unbounded stream of events that flows into the system. Since the Map Reduce systems operate on static data by scheduling batch jobs, the renowned Hadoop platform delivers its best for batch processing systems. Therefore there is a clear need for a highly scalable stream computing solution that can operate at high data rates and process massive amounts of data.
Yahoo's open source product S4 (Simple Scalable Streaming System) is one such solution.

Digging a bit more into S4
S4 affirms to support many features and substantiates itself as :
• Decentralized : All nodes of an S4 cluster are symmetric and there is no single point of failure.
• Scalable : Throughput increases linearly with number of nodes.
• Partially fault-tolerant : Fault tolerant because
- cluster management system reroutes event from failed to other servers 
and Partially because
- state of the Processing Elements running on the failed server might be lost in case persistent
storage has not been used.
• Elastic : computing load automatically gets distributed
• Expandable : a simple API has been provided
•Object Oriented : POJOs used for internode communication.
  
Basic Components of S4 

• Processing Element (PE)
Basic computational unit which can send and receive messages called Events. Upon receiving any
event a PE can :
- emit one or more events which may be consumed by other PEs
- publish results, possibly to an external data store or consume.

• Processing Node (PN)
The logical hosts to PEs which can :
- Listen to events
- Dispatch events with the help of the communication layer
- Emit output events
- and execute operations on the incoming events.

• Adapter
The events being fed to the S4 cluster for processing need to be translated into S4
compatible events and similarly the events received from an S4 cluster have to be made
understandable to the client. The Client I/O Stub solves this purpose whereas the
Adapter injects events into the S4 cluster and receives from it via the Communication
Layer.

Existing solutions and S4
The S4 design is not new in the industry as it implements the Actor framework. Erlang
and Scala already have a similar implementation. But the power of mixing in Zookeeper
and a pluggeable architecture can set S4 apart from previous frameworks.
At the moment Yahoo has open sources a handful set of applications. My next post would be an attempt to help you out in creating an S4 application.