Examples to help you differentiate between Business Intelligence and Data Science problems:

In this post, I’ll list few examples from various industries to help you differentiate between business intelligence and data science problems.

Sometime back, I blogged about “Business Analytics Continuum” and in the post we saw that Every Organization has DATA but they use their business data at different levels because of their maturity level. Excel (or other transactional reporting tools) is usually the starting point for any organization – it helps them see WHAT happened. They advance to the next stage, where they get capabilities to slice and dice their data – To find out WHY – and usually this capability is delivered using Business Intelligence tools & techniques. Once the data culture spreads – Thanks to a successful Business Intelligence project – then they soon start to outgrow their business intelligence capabilities by asking problems that need predictive capabilities. This is advanced analytics and Data Science stage. To that end, here are 5 examples to help you differentiate between business intelligence and data science problems:

	Business Intelligence.(WHAT & WHY)	Data Science & advanced analytics.
Bike Rentals	How many bikes did we rent in Q3 2014? How does that compare to Q3 2013? What is the trend of total bike rentals at week level? Can you break it down by geography?	Can you predict bike rentals on an hourly basis?
Credit Risk	How many customers have a credit risk of ‘C’? Can you rank customers by their payments due amount that have a credit risk ‘C’?	Can you predict the credit risk of the customer during contract negotiations stage?
Customer relationship management	How many account cancellations occurred this year (broken down by month and customer segmentation)? How does percentage of account cancellations this year compare to that previous year?	Can you predict customer churn?
Flight Delays	What is the trend of % of flight delayed this year? Can you break down flight delays this year by their reasons?	Can you predict whether a scheduled flight will be delayed by more than 15 minutes?
Customer feedback	What is the customer satisfaction % trend this year? What is the customer satisfaction % broken down by customer segments and product segments?	Can you classify a customer feedback comment into “positive”, “negative” or “neutral”?

I hope this helps!

The role of Sentiment Analysis in Social Media Monitoring:

I’ve posted tutorial/resources about the Technical Side of Sentiment Analysis on this Blog. Here are the Links, if you need them:

LingPipe (Java Based) | Python | R language resource | Microsoft’s Tool “Social Analytics”

Apart from this, I’ve used other Tools per project requirements and It’s been fun designing and developing projects on “Sentiment Analysis” primarily using Social Media Monitoring. Having worked with clients on projects that use “Sentiment Analysis” – I reflected about the role of Sentiment Analysis in Social Media Monitoring. And in this blog post, I am sharing these reflections:

What is Social Media Monitoring?

Social Media Monitoring is a process of “monitoring” conversations happening on social media channels about your brand/company.

Is it NEW? Not really. The idea of monitoring or gathering data about what is being talked about the brand/company is not new. Earlier, it was newspapers and magazine-articles and now, it’s the social media channels including online news, forums and blogs and thus the name given to this process is “Social Media Monitoring”

What is Sentiment Analysis?

Analyzing data to categorize it under a “sentiment” (emotion).

Example. Is this review saying positive, negative or neutral thing about our product.

side-note: Sentiment analysis is often categorized under “Big Data Analytics”.

What’s the Role of Sentiment Analysis in Social Media Monitoring?

We’ve seen that in social media monitoring, we gather all online conversations about a brand/product/company. Now wouldn’t it be great to take the data that we have and bucket it under “Positive”, “Negative” or “Neutral” categories for further analysis?

So few questions that can be answered after we have results from sentiment analysis:

1) Are people happy or sad about our product?

2) What do they like about our product?

3) What do they hate about our service?

4) Is there a trend or seasonality in sentiment data?

Among other business insights that may be not be easily answerable with just plain text data.

Thus sentiment analysis is one of the step in social media monitoring that assists in analyzing sentiment of all the conversations happening on the social web about a brand/product.

That’s about this for this post. Here’s a related post: Three Data Collection Tips for Social Media Analytics

your comments are very welcome!

Beginner’s Guide: Sentiment Analysis using Python on Windows

This is beginner’s guide to sentiment analysis using Python NLTK on windows. We’ll start w/ installing Python and NLTK and then see how to perform sentiment analysis.

Step 1: Install Python & NLTK

I followed the steps listed on http://nltk.org/install.html

1. Search for python 2.7.3 for windows and install it.

2. Search for Python setup Tools for Windows and install it.

3. Install PIP (for win 64 bit), NLTK and PyYAML.

4. Test installation: Start>All Programs>Python27>IDLE, then type import nltk

Now,

5. Also type:

>>> Import random

6. And also install movie_reviews corpus by typing:

>>>nltk.download()

in the new window that opens, install the movie_reviews corpus.

Step 2: Sentiment Analysis

I followed the code explained in the NLTK book in the section “document classification” in ch 6 learning to classify text. Here is the section: http://nltk.org/book/ch06.html#document-classification

Using the code I was able to run the Naive Bayes Classifier to categorize text:

Conclusion:

In this post, we learned how to perform sentiment analysis using Python on windwos platform. NLTK supports classifiers other than Naive Bayes, and also there are resources that will help you increase the accuracy of the classifier. And I hope that this post acts as a starting guide for you!

Sentiment Analysis using LingPipe on windows 7: (parasdoshi.com)
Three Data Visualizations I liked this week: (parasdoshi.com)
Sentiment Analysis in R w/ Twitter data feeds (parasdoshi.com)
Second Try: Sentiment Analysis in Python : Andy Bromberg (andybromberg.com)
Sentiment Symposium Tutorial (http://sentiment.christopherpotts.net/)
What’s “Naive” about Naive Bayes Machine Learning Algorithm? (parasdoshi.com)

Three Data Visualizations I liked this week:

I have been working on creating Dashboards for one of my projects. As a part of the research, I looked at few Dashboards out their on the inter-webs. Here are three of them that I liked:

1. Social Media & Sentiment Analysis:

What I like about this Dashboard is the creative use of Data via Sentiment Analysis:

2. Microsoft Research’s Viral Search Project:

What a creative way to visualize viral content!

3. Social Media analytic’s Dashboard:

Nice one page social dashbaord!

Do you see the bottom right part of the report that shows you engagement levels by post type, if you want to compute it – here’s my blog post on that: Social Media Analytics. Facebook Page Smackdown: Status updates vs Images?

Sentiment Analysis using LingPipe on windows 7: (parasdoshi.com)

What’s “Naive” about Naive Bayes Machine Learning Algorithm?

In this post, I’ll post what why does the “Naive Bayes machine learning” algo have the word Naive in it?

So here is the short answer:

It “assumes” that the features are independent. (In other words: There’s no relation between the features that are used while building the model)

Let’s go a little deeper:

First up, few basic pointers.

> It’s a machine learning algorithm used for classification

> It’s based on Bayesian Statistics.

> you can read about it here: http://en.wikipedia.org/wiki/Naive_Bayes_classifier

Now, what do you mean when you mean that it is Naive because it assumes that features are independent?

Let’s take an example:

Suppose, you are building a “credit card approval” model based on Income and CreditScore

(SideNote: For those who do not know what is credit score, here you go: http://en.wikipedia.org/wiki/Credit_score_in_the_United_States)

And you have the following columns in the training data (Note: In machine learning, think of this columns as features)

Income	CreditScore	Approved
High	High	Yes
High	Medium	Yes
Low	High	Yes
Low	Low	NO

Here the features are Income & CreditScore and the target of the classification model is Approved.

In real world, there’s some relation between “income” and “creditscore”. Agree? Great! But Naive Bayes doesn’t think so. Let me reiterate the point of this blog post and see if it makes more sense now: it assumes that the features are “independent” and that’s why it is Naive!

I hope this helps. your comments are very welcome!

12 Useful Tips for Machine Learning (datascience101.wordpress.com)
Data Mining: Classification VS Clustering
Back to basics: Data Mining and Knowledge Discovery Process

Sentiment Analysis using LingPipe on windows 7:

In this post, I’ll point you to the resource using which you can perform sentiment analysis using LingPipe on a windows OS. Along with that I’ll share couple of issues that I ran into when I was trying to run this demo on a Windows 7:

So first up, here’s the resource:

http://alias-i.com/lingpipe/demos/tutorial/sentiment/read-me.html

Now here are a couple of issues that I had:

1. Error: could not find or load the main class PolarityBasic

To solve this error, you’ll need to build the files given under the C:lingpipe-4.1.0demostutorialsentiment – we use ANT for this. Let’s see how to do that:

2. Building sentiment.jar using ant jar

After successfully downloading ant on windows and setting the ANT_HOME variable to c:apache-ant-1.8.4 – I was still getting the error that ant is not a recognized command.

So I ran following commands:

C:>set ANT_HOME=C:apache-ant-1.8.1
C:>set JAVA_HOME=C:jdk1.6.0_24
C:>set PATH=%ANT_HOME%bin;%JAVA_HOME%bin
C:>ant -version
// it worked!

Thanks: http://stackoverflow.com/questions/5607664/installing-ant-ant-home-is-set-incorrectly-on-windows-7

Now I ran the following command:

3. In the tutorial they used POLARITY_DIR – I didn’t use that, Instead I just inputted c:review_polarity because that’s where I unzipped the movie review dataset:

Here’s the screenshot about the command that does basic polarity analysis:

And Thanks: http://stackoverflow.com/questions/15010184/lingpipe-and-sentiment-analysis/15011482

Sentiment Analysis in R w/ Twitter data feeds (parasdoshi.com)

Insight Extractor – Blog

Paras Doshi's Blog on Analytics, Data Science & Business Intelligence.

sentiment analysis

Examples to help you differentiate between Business Intelligence and Data Science problems:

Like this: