Feng Li

Flux Redis

2019-04-14T00:00:00+00:00

It is interesting that we can find flux topology example for HBase and Kafka but it’s hard to find one for Redis.

A HBASE + Storm/Flux example can be found at: https://github.com/apache/storm/blob/master/flux/flux-examples/src/main/resources/simple_hbase.yaml

A Kafka + Storm/Flux example can be found at: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-storm-with-kafka

Let’s see how we can do a write to Redis.

So a flux word count example can be found at https://github.com/Azure-Samples/hdinsight-python-storm-wordcount. On top of that what if we want to save the word count results to Redis?

We just need to add a bolt to receive word count results from count bolt and write them to Redis. We can call this bolt “redis-store-bolt”. Simply it can be defined like following in Flux topology.yaml:

id: “redis-store-bolt” className: “org.apache.storm.redis.bolt.RedisStoreBolt” constructorArgs:
- ref: “poolConfig”
- ref: “storeMapper” parallelism: 1

Storm has ready-to-use class to talk to Redis - org.apache.storm.redis.bolt.RedisStoreBolt. To initialize this class we need two parameters: 1) Redis service configuration in the format of class JedisPoolConfig, 2) a RedisStoreMapper object to map incoming data to be key/value pairs in Redis. (See details: https://storm.apache.org/releases/1.1.2/javadocs/org/apache/storm/redis/bolt/RedisStoreBolt.html).

And in “components” section of the topology.yaml file we can define for the two parameters:

id: “poolConfig” className: “org.apache.storm.redis.common.config.JedisPoolConfig” constructorArgs:
- “localhost”
- 6379
id: “storeMapper” className: “org.apache.storm.redis.common.mapper.RedisStoreMapper” constructorArgs:

Install/Configure Hive 3.1.1 Using MySQL Metastore

2019-01-13T00:00:00+00:00

This task is verified on top of Hadoop 3.1.1. Work as user “hadoop” if not specified.

Download Hive 3.1.1

$ pwd
/localuser/hadoop

$ wget http://mirror.dsrg.utoronto.ca/apache/hive/hive-3.1.1/apache-hive-3.1.1-bin.tar.gz

$ tar zxvf apache-hive-3.1.1-bin.tar.gz

$ ls
apache-hive-3.1.1-bin  apache-hive-3.1.1-bin.tar.gz  hadoop  hadoop-3.1.1.tar.gz  hadoopdata  perl5

$ mv apache-hive-3.1.1-bin hive

$ ls
apache-hive-3.1.1-bin.tar.gz  hadoop  hadoop-3.1.1.tar.gz  hadoopdata  hive  perl5

Set environment variables

Set environment variables for user “hadoop”. With all the Hadoop settings user hadoop’s “.bashrc” file should look like following:

$ cat .bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
export HADOOP_HOME=/localuser/hadoop/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

# hive
export HIVE_HOME=/localuser/hadoop/hive
export PATH=$HIVE_HOME/bin:$PATH

[hadoop@li-9 ~]$
[hadoop@li-9 ~]$ source .bashrc

Create directories on HDFS for Hive

$ hadoop fs -mkdir /tmp
$ hadoop fs -mkdir /user
$ hadoop fs -mkdir /user/hadoop
$ hadoop fs -mkdir /user/hive
# warehouse is where Hive stores actual data if you create a table and insert data later on.
$ hadoop fs -mkdir /user/hive/warehouse
$ hadoop fs -chmod g+w /tmp
$ hadoop fs -chmod g+w /user/hive/warehouse

Note, following jar file is provided by both Hadoop and Hive. We’ll remove it from Hive to avoid errors.

$ rm /localuser/hadoop/hive/lib/log4j-slf4j-impl.jar

Configure hive-site.xml with metastore information to be used

$ pwd
/localuser/hadoop/hive/conf
$ cat hive-site.xml
<configuration>

 <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://<MySQL Server>:3306/hive?createDatabaseIfNotExist=true</value>
    <description>JDBC connect string for a JDBC metastore</description>
</property>

 <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
 </property>

 <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hadoop</value>
    <description>username to use against metastore database</description>
 </property>

 <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hadoop123</value>
    <description>password to use against metastore database</description>
 </property>

 <property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9083</value>
 </property>

</configuration>

Configure MySQL database for hive-site.xml

On MySQL database server as user root, create “hive” database and user “hadoop” based on information in hive-site.xml:

# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
...

mysql> create database hive;
Query OK, 1 row affected (0.00 sec)

mysql> create user 'hadoop'@'%' identified by 'hadoop123';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on hive.* to 'hadoop'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql>

Back on Hive machine, Verify above changes:

$ mysql -h <MySQL server> -u hadoop -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
...

MySQL [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| hive               |
| mysql              |
| test               |
+--------------------+
4 rows in set (0.00 sec)

MySQL [(none)]> use hive;
Database changed
MySQL [hive]>

Put MySQL JDBC jar file in place for Hive

$ pwd
/localuser/hadoop/hive/lib
$ cp /scratch/support/li/pac-package/mysql-connector-java-5.1.22-bin.jar .

Initialize metastore schemas

$ schematool -dbType mysql -initSchema
Metastore connection URL:     jdbc:mysql://li-4:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver :     com.mysql.jdbc.Driver
Metastore connection User:     hadoop
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.mysql.sql
…
Initialization script completed
schemaTool completed

$ mysql -h li-4 -u hadoop -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
...

MySQL [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| hive               |
| mysql              |
| test               |
+--------------------+
4 rows in set (0.00 sec)

MySQL [(none)]> use hive
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MySQL [hive]> show tables;
+-------------------------------+
| Tables_in_hive                |
+-------------------------------+
| AUX_TABLE                     |
| BUCKETING_COLS                |
…...
| WM_TRIGGER                    |
| WRITE_SET                     |
+-------------------------------+
74 rows in set (0.00 sec)

Launch metastore service

Modify/create hive-env.sh in /localuser/hadoop/hive/conf/hive-env.sh to add

        export METASTORE_PORT=9083

This port should be consistent to the thrift URI parameter in hive-site.xml:

<property>
    <name>hive.metastore.uris</name>
    <value>thrift://localhost:9083</value>
 </property>

Now start metastore service

$ hive --service metastore
Start Metastore Service ...

Launch Hive CLI and Do Tests

$ hive
...
hive> show tables;
OK
Time taken: 1.929 seconds
hive>
hive> create table mytable
    > (
    > no int,
    > name string,
    ...
    > )
    > row format delimited
    > fields terminated by ','
    > ;
OK
Time taken: 15.778 seconds
hive> show tables;
OK
mytable
Time taken: 0.085 seconds, Fetched: 1 row(s)
hive>
hive> load data local inpath '/localuser/hadoop/testhive/mytabledata.txt' overwrite into table mytable;
Loading data to table default.mytable
OK
Time taken: 2.205 seconds
hive> select * from mytable;
OK
...

Hive stores the actual data in HDFS Hive warehouse

$ hadoop fs -ls /user/hive/warehouse
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2019-01-11 12:50 /user/hive/warehouse/mytable
$ hadoop fs -ls /user/hive/warehouse/employee
Found 1 items
-rw-r--r--   1 hadoop supergroup        156 2019-01-11 12:50 /user/hive/warehouse/mytable/mytabledata.txt

Pattern Mining using Apriori algorithm

2018-02-19T00:00:00+00:00

Given an input file which is an attributes list of one place. The attributes of the place consists of a number of category instances. Each line represents one place. Examples are as following:

Breakfast & Brunch;American (Traditional);Restaurants
Sandwiches;Restaurants
Local Services;IT Services & Computer Repair
Restaurants;Italian
Food;Coffee & Tea
Fast Food;Restaurants
Mortgage Brokers;Home Services;Real Estate

So the task is to mine frequent category patterns under a given min-support.

The philosophy of Apriori algorithm includes two steps: frequent pattern generating and pruning. It’s an iterative process that starts from Candidates set and ends up with Frequent set. Starts from 1-item to 2-items and so on. For example,
Step 1, Scan data to generate 1-item candidate set. And filter the patterns with min-support to find frequent patterns.
Step 2, Generate 2-item candidate set based on 1-item set and
- Trim the data by removing infrequent 1-item patterns.
- Calculate support for 2-item patterns and prune to with min-support to get 2-item frequent patterns.
Step 3, repeat above steps 2 to generate 3-item frequent patterns and so on. Until there is no x-item patterns exist in data or no patterns satisfy the min-support.

So we’ll work on this “category of places” data to find all frequent patterns. What’s the value of this? To find what services are always together in one place could be used as references to open your new stores for best profits.
Let’s say if we find “Hotels & Travel”, “Hotels” and “Event Planning & Services” are associate most in some places what business chances you want to take there?

So we’ll get started.

It looks familiar to generate 1-item patterns calculating supports as to do a classic “word count”. So just use handy Spark cluster to get this done firstly.

After that we’ll need to adjust format and apply min-support (771) to get 1-item frequent set.

Here is a look at the 1-item frequent set - “support:category” as following:

Next, we’ll work on 2-item frequent set.
Step 1, we’ll generate 2-item candidate set based on 1-item frequent set using itertools.combinations method.

Step 2, we’ll remove infrequent 1-item categories from dataset and then calculate support for 2-item candidate category set.

Now we have 2-item frequent category set.

So we can continue to get 3-items frequent category set like this - only one line:

By now we can also tell this is the end of the frequent patterns - three items cannot make up four or more patterns.

Lastly we can combine all the frequent patterns: 1-item, 2-item and 3-item patterns together to get all the frequent patterns out of this dataset.

Brief Exercises with Pig Latin

2018-01-31T00:00:00+00:00

With Hortonworks Sandbox 2.6 we’ll have a brief exercise using Pig to take a look at movie dataset.

So a movie dataset can be downloaded from http://files.grouplens.org/datasets/movielens/ml-100k.zip. It includes After extracting the zip file, we’ll need u.data and u.item two files.

u.data – There are 100,000 ratings by 943 users on 1682 movies. This is a comma separated list of user id | item id | rating | timestamp. (The time stamps are unix seconds since 1/1/1970 UTC)

Sample data from file u.data: 196,242,3,881250949 # user #196 rated movie #242 with score 3 at time of 881250949 186,302,3,891717742 22,377,1,878887116

u.item – Information about the items (movies); this is a tab separated list of “movie id

movie title

release date

…” The fields after “release date” are not related to our task so we ignore them for now.

Sample data from file u.item: 1|Toy Story (1995)|01-Jan-1995|… 2|GoldenEye (1995)|01-Jan-1995|… 3|Four Rooms (1995)|01-Jan-1995|…

So the question we’re going to attack is “what are the top 20 best rated latest movies?”

Following are Pig Latin script to do so:

ratings = LOAD '/user/maria_dev/ml-100k/u.data' 
          USING PigStorage(',') 
          AS (userID:int, movieID:int, rating:int, ratingTime:int);

metadata = LOAD '/user/maria_dev/ml-100k/u.item' 
           USING PigStorage('|')
           AS (movieID:int, movieTitle:chararray, releaseDate:chararray,
               videoRelease:chararray, imdbLink:chararray);
               
nameLookup = FOREACH metadata GENERATE movieID, movieTitle, 
                 ToUnixTime(ToDate(releaseDate, 'dd-MMM-yyyy')) AS releaseTime;
                 
ratingByMovie = GROUP ratings BY movieID;

avgRatings = FOREACH ratingByMovie GENERATE group as movieID, 
                                           AVG(ratings.rating) AS avgRating;

fiveStarMovies = FILTER avgRatings BY avgRating > 4.5;

fiveStarMoviesWithData = JOIN fiveStarMovies BY movieID, nameLookup BY movieID;

oldestFiveStarMovies = ORDER fiveStarMoviesWithData BY nameLookup::releaseTime DESC;

first20 = LIMIT oldestFiveStarMovies 20;

DUMP first20;

Run this script in Pig View in Ambari we’ll get the latest top 20 movie list that are best rated. The latest movie in this small dataset is dated to April 1998.

Next, based on this script, we can also find what movies are most rated by using COUNT(ratings.rating).
It could be interesting to find most rated movie with lowest ratings.

We can do that with following Pig Latin script:

ratings = LOAD '/user/maria_dev/ml-100k/u.data' 
          USING PigStorage(',') 
          AS (userID:int, movieID:int, rating:int, ratingTime:int);

metadata = LOAD '/user/maria_dev/ml-100k/u.item' 
           USING PigStorage('|')
           AS (movieID:int, movieTitle:chararray, releaseDate:chararray,
               videoRelease:chararray, imdbLink:chararray);
               
nameLookup = FOREACH metadata GENERATE movieID, movieTitle, 
                 ToUnixTime(ToDate(releaseDate, 'dd-MMM-yyyy')) AS releaseTime;
                 
ratingByMovie = GROUP ratings BY movieID;

countRatings = FOREACH ratingByMovie GENERATE group as movieID, 
                       AVG(ratings.rating) AS avgRating, COUNT(ratings.rating) AS numRatings;

lowestRatingMovies = FILTER countRatings BY avgRating < 2.0;

lowestRatingMoviesWithData = JOIN lowestRatingMovies BY movieID, nameLookup BY movieID;

mostRatedLowestRatingMovies = FOREACH lowestRatingMoviesWithData GENERATE nameLookup::movieTitle AS movieName,
                                      lowestRatingMovies::avgRating AS avgRating, lowestRatingMovies::numRatings AS numRatings;

mostRatedLowestRatingMoviesSorted = ORDER mostRatedLowestRatingMovies BY numRatings DESC;

first20 = LIMIT mostRatedLowestRatingMoviesSorted 20;

DUMP first20;

Run above script in Ambari:

This is an exercise in Udemy course “the-ultimate-hands-on-hadoop-tame-your-big-data”.

Central Limit Theorem(CLT) test with R

2017-12-31T00:00:00+00:00

In this simple note, I will investigate the exponential distribution in R and compare it with the Central Limit Theorem(CLT). CLT tells us that sample mean distribution is a normal distribution when sample size is large enough (for example >30). Expected value of sample mean distribution is used to estimate population mean. Sample mean variance is population variance divided by sample size.

In following plot, red line stands for sample mean and green line stands for theoretical mean.

This simple note is from Data Science course by JHU in Coursera. For details check on RPubs

NOAA Storm Data Analysis

2017-12-31T00:00:00+00:00

Storms and other severe weather events can cause both public health and economic problems for communities and municipalities. Many severe events can result in fatalities, injuries, and property damage, and preventing such outcomes to the extent possible is a key concern.

This project involves exploring the U.S. National Oceanic and Atmospheric Administration’s (NOAA) storm database. This database tracks characteristics of major storms and weather events in the United States, including when and where they occur, as well as estimates of any fatalities, injuries, and property damage.

This data analysis will address the following questions:

Across the United States, which types of events (as indicated in the EVTYPE variable) are most harmful with respect to population health?

Across the United States, which types of events have the greatest economic consequences?

This was a homework in Data Science course by JHU in Coursera. For details, check on Rpubs

Airquality Dataset Viz with Vega-lite

2017-12-31T00:00:00+00:00

This project is a final project for Data Viz course by UIUC in Coursera. The purpose of the project is to practice narrative visualization structure using Java script plus Vega/Vega-lite libraries.

This Data Viz project can be seen at link

This project follows the Interactive Slideshow narrative visualization structure. It is a hybrid structure mixing author-lead and reader-lead scenes. This project consists a Home page, three Scene pages and a Final summary page. So user will follow an author directed path through the slideshow. During home page and the three scenes user can drill down a little bit for some details which they are interested. Or user can just continue if they are not interested to certain pages.

In addition, the project provides this About page to describe the design of this project.

The charts were built using Vega-lite to complete this project in limited time. Note, Vega plus D3 will support advanced interaction to those charts which leave the improvement at a later time.

User Interface Events The buttons in the navigation bar (page top) and the Prev, Next buttons (page bottom) enable the User Interface events. The buttons allow user to walk through the author-lead structure.

Annotations The annotation is represented by the legend on top right of the scatter plots and tooltips when mouse is hovered over the bar chart.

Triggers The mouse click acts as a trigger event for navigation on button. The mouse hover event is a trigger for the tooltip in bar chart. Also the mouse can be used to zoom in/out the scatter plots in the three scene pages for details.

Parameters All the pages are hyper linked to each other. The page names are passed as input to the buttons on page top and page bottom.

Source code location is link

Create a map marked with UofT using leaflet

2017-12-31T00:00:00+00:00

The requirements to this assignment in DS course by JHU:

Create a web page using R Markdown that features a map created with Leaflet.
Host your webpage on either GitHub Pages, RPubs, or NeoCities.
Your webpage must contain the date that you created the document, and it must contain a map created with Leaflet.

So R code and interactive map can be found on RPubs

Take a quick look at MPG using box plot, t-test and build a regression model

2017-12-31T00:00:00+00:00

This project assumes that you work for Motor Trend, a magazine about the automobile industry. Looking at a data set (mtcars from datasets library) of a collection of cars, they are interested in exploring the relationship between a set of variables and miles per gallon (MPG) (outcome). They are particularly interested in the following two questions:

“Is an automatic or manual transmission better for MPG”
“Quantify the MPG difference between automatic and manual transmissions”

The purpose of this project report is to address above two questions use the knowledge from course 7 regression models.

Can you draw any conclusion from following plot for the first question?

Statistical analysis (t-test) for the second question and more can be found on RPubs

NLP with R part 2 predict next words using Shiny App

2017-12-31T00:00:00+00:00

This project was capstone project for Coursera Data Science Specialization held by Johns Hopkins University. SwiftKey is corporate partner in this project.

The purpose of this project is to learn and practice text mining and NLP knowledge and skills. We’ll start from text dataset provided by SwiftKey, build up algorithms to predict next most likely word following given text. A Shiny application will be created to accept user input and display predicted word.

Algorithm Description

After random sampling the text data set, a corpus data was created and cleaned. Data cleaning approaches include: change to lower case, remove number, remove punctuation, strip whitespaces, remove English end words.

Based on this, tokenization was conducted to create bi-grams and tri-grams data matrixes.

For example,
Top 5 bi-gram words are: “right now”, “dont know”, “last year”, “new york”, “cant wait”
Top 5 tri-gram words are: “happy mothers day”, “cant wait see”, “new york city”, “let us know”, “world war ii”

Furthermore, frequency data tables are created respectively and thus being used for prediction.

Shiny App Description

This app accept user input text at left sidebar and display the predicted next word on right panel. For example, input “happy” and press Enter, “birthday” will be predicted. Input “let us”, “know” will be predicted.
The app takes input words and match them in frequency data table to find the most likely next word.

App is hosted on Shiny server
Source code on Github