Mahout’s Naïve Bayes: Test Phase

This post is in continuation to my previous post where Mahout Naive Bayes "trainnb" command has been explained. This one would describe the internal execution steps of the "testnb" command, which is used to test the model we build using 'trainnb".

Command Line Options for "testnb"

Generic Options

-archives <paths>: comma separated archives to be unarchived on the compute machines.                                                              -conf <configuration file>:  specify an application configuration file  -D <property=value>: use value for given property  -files <paths>: comma separated files to be copied to the map reduce cluster  -fs<local|namenode:port>: specify a namenode  -jt<local|jobtracker:port>: specify a job tracker  -libjars<paths>: comma separated jar files to include in the classpath.  -tokenCacheFile<tokensFile>:  name of the file with the tokens

Job-Specific Options
                                                       

--input (-i) input: Path to job input directory.                                                                                                                                                       --output (-o) output: The directory pathname for output.           --testComplementary (-c): test complementary?   --labelIndex (-li) labelIndex: The path to the location of the label index         --overwrite (-ow): If present, overwrite the output directory before running job                                 --help (-h): Print out help                               --tempDirtempDir: Intermediate output directory                 --startPhasestartPhase: First phase to run                           --endPhaseendPhase: Last phase to run   --runSequential (-seq):  run sequential?   --model (-m) model: The path to the model built during training

Flow of execution of the "testnb" command

Hope it helped!

Mahout’s Naïve Bayes: Train Phase

Mahout’s Naïve Bayes Classification algorithm executes in two phases:

1. Train Phase: Trains a model using pre-processed train data
2. Test Phase: Classify documents (pre-processed) with the help of the model

 This blog post provides code-level understanding of the training process in the algorithm. And testing phase is covered in the next blog. The Mahout command "trainnb" is used to train a Naive Bayes model in Mahout.

"trainnb" command in Mahout

For reference, above is the structure of the train data directory specified as an input (similar to the one used in data pre-processing).

Command Line options for “trainnb”

Generic Options

-archives <paths>: comma separated archives to be unarchived on the compute machines.                  -conf <configuration file>:  specify an application configuration file  -D <property=value>: use value for given property  -files <paths>: comma separated files to be copied to the map reduce cluster  -fs<local|namenode:port>: specify a namenode  -jt<local|jobtracker:port>: specify a job tracker  -libjars<paths>: comma separated jar files to include inthe classpath.  -tokenCacheFile<tokensFile>:  name of the file with the tokens

Job-Specific Options

                                                          

--input (-i) input: Path to job input directory.                    --output (-o) output: The directory pathname for output.                                                                        --labels (-l) labels:comma-separated list of labels to include in training                                        --extractLabels (-el):Extract the labels from the input               --alphaI (-a) alphaI: smoothing parameter                             --trainComplementary (-c):train complementary?                            --labelIndex (-li) labelIndex: The path to store the label index in            --overwrite (-ow): If present, overwrite the output directory before running job                              --help (-h):Print out help                                  --tempDirtempDir: Intermediate output directory                   --startPhasestartPhase:First phase to run                              --endPhaseendPhase: Last phase to run

Flow of execution of the "trainnb" command

Hope it helped!

Understanding Data Pre-processing in Mahout–Part II

In continuation to my previous post where first one of the two commonly used commands for data pre-processing in Mahout is described, we shall continue with the second one i.e. “seq2sparse” in this post.

The command expects sequence files as an input, which have been formed using the “seqdirectory” command. During its processing, this command creates a couple of sub-directories in the output directory, like tokenized-documents, tf-vectors etc. The flow described below explains the execution of the command on the basis of the sequence of formation of these sub-directories. The input directory would be assumed to contain two sub-directories each of which has two files:

                                                           

Command options

Usage:                                                   
                       

[--minSupport <minSupport> --analyzerName <analyzerName> --chunkSize         <chunkSize> --output <output> --input <input> --minDF <minDF> --maxDFSigma     <maxDFSigma> --maxDFPercent <maxDFPercent> --weight <weight> --norm <norm>     --minLLR <minLLR> --numReducers <numReducers> --maxNGramSize <ngramSize>       --overwrite --help --sequentialAccessVector --namedVector --logNormalize]

Options:                                                                        

--minSupport (-s) minSupport : (Optional) Minimum Support. Default Value: 2                               --analyzerName (-a)  analyzerName: The class name of the analyzer           --chunkSize (-chunk) chunkSize: The chunkSize in MegaBytes. 100-10000 MB --output (-o) output: The directory pathname for output.       --input (-i) input:  Path to job input directory.             --minDF (-md) minDF: The minimum document frequency.  Default is 1                                     --maxDFSigma (-xs) maxDFSigma:  What portion of the tf (tf-idf) vectors to be used, expressed in times the standard deviation (sigma) of the document frequencies of these vectors.             Can be used to remove really high frequency terms. Expressed as a double value. Good value to be specified is 3.0. In case the value is less then 0 no vectors will be filtered out. Default is   -1.0.  Overrides maxDFPercent           --maxDFPercent (-x) maxDFPercent: The max percentage of docs for the DF. Can be used to remove really high frequency terms. Expressed as an integer between 0 and 100. Default is 99.  If  maxDFSigma is also set, it will override this value. --weight (-wt) weight:  The kind of weight to use. Currently TF or TFIDF --norm (-n) norm: The norm to use, expressed as either a float or "INF" if you want to use the  Infinite norm.  Must be greater or equal to 0.  The default is not to normalize   --minLLR (-ml) minLLR: (Optional)The minimum Log Likelihood Ratio(Float)  Default is 1.0 --numReducers (-nr) numReducers: (Optional) Number of reduce tasks. Default Value: 1                 --maxNGramSize (-ng) ngramSize: (Optional) The maximum size of ngrams create (2 = bigrams, 3 = trigrams, etc) Default Value:1                         --overwrite (-ow): If set, overwrite the output directory   --help (-h): Print out help --sequentialAccessVector(-seq):(Optional)Whether output vectors should be SequentialAccessVectors. If set else false --namedVector (-nv): (Optional) Whether output vectors should be NamedVectors. If set true else false   --logNormalize (-lnorm): (Optional) Whether output vectors should be logNormalize. If set true else false

Code-level Explanation

Code-level understanding of "seq2sparse" command

Hope this helped!

Understanding Data Pre-processing in Mahout – Part I

Two most common commands used for pre-processing of train or test data when running Mahout algorithms are:

• seqdirectory:  Turns raw text in a directory into mahout sequence file.
• seq2sparse: Creates tfidf weighted vector from the sequence files created in “seqdirectory”.

This blog post describes the first command “seqdirectory” to the code level and the next blog post shall focus on the second one.

Let’s start then!

Command Options

Firstly have a look at the Command-line options that can be used with “seqdirectory”.

-archives <paths>: comma separated archives to be unarchived on the compute machines.  -conf <configuration file>:  specify an application configuration file  -D <property=value>: use value for given property  -files <paths>: comma separated files to be copied to the map reduce cluster  -fs <local|namenode:port>: specify a namenode  -jt <local|jobtracker:port>: specify a job tracker  -libjars <paths>: comma separated jar files to include in the classpath.  -tokenCacheFile <tokensFile>: name of the file with the tokens

Job-specific Options

--input (-i) input: Path to job input directory.   --output (-o) output:  The directory pathname for output.                         --overwrite (-ow): If present, overwrite the output directory before running job                     --chunkSize (-chunk) chunkSize: The chunkSize in MegaBytes. Defaults to 64                 --fileFilterClass (-filter) fileFilterClass: The name of the class to use for file parsing.   --keyPrefix (-prefix) keyPrefix: The prefix to be prepended to the key                         --charset (-c) charset: The name of the character encoding of the input files.   --tempDir tempDir: Intermediate output directory   --startPhase startPhase: First phase to run             --endPhase endPhase: Last phase to run

Code-level Explanation

The following image segregates the internal execution of the command into steps, and then further breaks down each step into tasks to be able to give a clear picture of what happens behind the scenes.

Code-level understanding of "seqdirectory" command

The next post describes the "seq2sparse" command in a similar fashion. Hope this helped.

Running Mahout's Logistic Regression

Logistic Regression(SGD) is one the algorithms available in Mahout. This blog post lists and provides all that is required for the same. To install Mahout on your machine, you can refer to my previous post.

Logistic Regression executes in two major phases:

• Train the model: This step is about creating a model using some train data, that can further be used for the classification of any input data rather I would say test data.

• Test the model: This step tests the generated model in step 1 by evaluating the results of classification of test data, and measuring the accuracy, scores and confusion matrix.

Steps for running Mahout’s LR

Step-I: Get the input data file called donut.csv, which is present in the mahout setup. But for your ready reference I have also shared it. You can download it from here.

Step-II: Next cd to the MAHOUT_HOME. Here we would be running the “org.apache.mahout.classifier.sgd.TrainLogistic” class that would train the model for us using the “donut.csv” file what we would be providing as train data. Here’s the command to be run from within MAHOUT_HOME:

bin/mahout org.apache.mahout.classifier.sgd.TrainLogistic --passes 1 --rate 1 --lambda 0.5 --input loc_of_file/donut.csv --features 21 --output any_loc_on_your_machine/donut.model --target color --categories 2 --predictors x y xx xy yy a b c --types n n

If the Mahout version is 0.7 you are likely to face the error below:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/ProgramDriver     at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:96) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.ProgramDriver     at java.net.URLClassLoader$1.run(URLClassLoader.java:366)     at java.net.URLClassLoader$1.run(URLClassLoader.java:355)     at java.security.AccessController.doPrivileged(Native Method)     at java.net.URLClassLoader.findClass(URLClassLoader.java:354)     at java.lang.ClassLoader.loadClass(ClassLoader.java:423)     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)     at java.lang.ClassLoader.loadClass(ClassLoader.java:356)     ... 1 more

Don’t worry, all you need to do is:

export CLASSPATH=${CLASSPATH}:your_MAHOUT_HOME/mahout-distribution-0.7/lib/hadoop/hadoop-core-0.20.204.0.jar

After editing the CLASSPATH as mentioned above the command should run successfully and print something like:

color ~ -0.016*Intercept Term + -0.016*xy + -0.016*yy       Intercept Term -0.01559                   xy -0.01559                   yy -0.01559     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000    -0.015590929     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000  13/05/26 02:14:02 INFO driver.MahoutDriver: Program took 588 ms (Minutes: 0.0098)

The most important parameters influencing the execution of the Training process are:

"--passes": the number of times to pass over the input data
"--lambda": the amount of coefficient decay to use
"--rate": the learning rate

You can vary the values of these 3 variables and see the change in performance of the algorithm. Also, you can now see that the model would have been created at the location which you had specified in the command.

Step-III: Now it’s time to run the classifier using the model that has been trained in Step-II. As the test data we would be using the same donut.csv file that we used for training or you can even split the file in some ratio for eg. 70-30 and use the 70% file for training of the model and 30% file for testing. Here’s the command for testing the model and running the classifier:

bin/mahout org.apache.mahout.classifier.sgd.RunLogistic --input loc_of_file/donut.csv  --model loc_of_model/donut.model --auc --scores --confusion

which should print an output something like:

"target","model-output","log-likelihood" 0,0.496,-0.685284 0,0.490,-0.674055 0,0.491,-0.675162 1,0.495,-0.703361 1,0.493,-0.706289 0,0.495,-0.683275 0,0.496,-0.685282 0,0.492,-0.677191 1,0.494,-0.704222 0,0.495,-0.684107 0,0.496,-0.684765 1,0.494,-0.705209 0,0.491,-0.675272 1,0.495,-0.703438 0,0.496,-0.685121 0,0.496,-0.684886 0,0.490,-0.672500 0,0.495,-0.682445 0,0.496,-0.684872 1,0.495,-0.703070 0,0.490,-0.672511 0,0.495,-0.683643 0,0.492,-0.677610 1,0.492,-0.708915 0,0.496,-0.684744 1,0.494,-0.704766 0,0.492,-0.677496 1,0.492,-0.708679 0,0.496,-0.685222 1,0.495,-0.703604 0,0.492,-0.677846 0,0.490,-0.672702 0,0.492,-0.676980 0,0.494,-0.681450 1,0.495,-0.702845 0,0.493,-0.679049 0,0.496,-0.684262 1,0.493,-0.706564 1,0.495,-0.704016 0,0.490,-0.672624 AUC = 0.52 confusion: [[27.0, 13.0], [0.0, 0.0]] entropy: [[-0.7, -0.4], [-0.7, -0.5]] 13/05/26 02:16:19 INFO driver.MahoutDriver: Program took 474 ms (Minutes: 0.0079)

Similarly, you can try on a variety of data sets that you might have. I have seen upto 93% accuracy of results of classification on a different data set.

All the best !!!

Installing Mahout on Linux

Mahout is an acquisition of highly scalable machine learning algorithms over very large data sets. Although the real power of Mahout can be vouched for only on large HDFS data, but Mahout also supports running algorithm on local filesystem data, that can help you get a feel of how to run Mahout algorithms.

Installing Mahout on Linux

Before you can run any Mahout algorithm you need a Mahout installation ready on your Linux machine which can be carried out in two ways as described below:

Method I- Extracting the tarball

Yes, it is that simple. Just download the latest Mahout release of from

http://www.apache.org/dyn/closer.cgi/mahout/

Extract the downloaded tarball using:

tar –xzvf  /path_to_downloaded_tarball/mahout-distribution-0.x.tar.gz

This should result in a folder with name /path_to_downloaded_tarball/mahout-distribution-0.x

Now, you can run any of the algorithms using the script “bin/mahout” present in the extracted folder. For testing your installation, you can also run 

bin/mahout

without any other arguments.

Method II- Building Mahout

1. Prerequisites for Building Mahout

 -   Java JDK 1.6

 -   Maven 2.2 or higher (http://maven.apache.org/)

Install maven and svn using following commands:

sudo apt-get install maven2                                                                 sudo apt-get install subversion

2. Create a directory where you would want to check out the Mahout code, we’ll call it here MAHOUT_HOME:

mkdir MAHOUT_HOME cd MAHOUT_HOME

3. Use Subversion to check out the code:

svn co http://svn.apache.org/repos/asf/mahout/trunk

4. Compiling

cd MAHOUT_HOME mvn -DskipTests install

5. Setting the environment variables

export HADOOP_CONF_DIR=$HADOOP_HOME/conf export MAHOUT_HOME=/location_of_checked_out_mahout         export PATH=$PATH:$MAHOUT_HOME

After following either of the above methods, you can now run any of the available mahout algorithms with appropriate arguments. Also, note that you can run the algorithm over HDFS data or local file system data. In order to run algorithms over data on your local file system set an environment variable with the name “MAHOUT_LOCAL” to anything other than an empty string. That would force mahout to run locally even if HADOOP_CONF_DIR and HADOOP_HOME are set.

To plunge into Mahout by trying out running an algorithm, you can refer to my next post. Hope this proved to be a good starter for you. 

All the best !!!