Data Mining

How Marketable is R programming?

Standard

Someone asked this on Quora: How Marketable is R programming?

Answer:

Let’s step back!

Why do you want to learn R? OR why do people learn R?

To solve problems that R can address. Right?

What problems do you have? OR what problems does your COMPANY have? OR what PROBLEMS your Dream company that you want to join have?

<< LIST THEM DOWN HERE>>

example:

  • I want to predict customers that are going to churn next quarter.
  • I want to identify Marketing channel that drove the revenue growth last quarter.
  • etc..

What’s Next?

NOW, take all of these problems and find ways to solve them.

R may or may not help.

You could just do it in Excel. Then do that.

OR R helps you a little bit in the process but you need something else.

In some case, R is a perfect solution! Like building a model to predict customer churn!

So, What?

you see, learning R is important and you might get a job by showing that you have “R” chops but that will not be enough for career growth. You should be focused on learning to solve business problems using data. use R sometimes. use Excel sometimes. use Python sometimes. use SQL. use Tableau. use << INSERT A TOOL HERE>>. Learn them. Apply them. Figure out their strengths and weakness. BUT learn to use all of these technology platforms to solve problems! Solve problems that are thorny. Solve problems that move the business needle. Solve problems that get your bosses boss promoted.

If you do that, marketing your skills wouldn’t be a concern anymore.

It’s NOT easy. And it WILL take time.

TL;DR: Go for it! Learn R! But more importantly, learn to solve problems with data.

VIEW QUESTION ON QUORA

How do you generally detect a fraud using analytics?

Standard

There are two broad range of algorithms that can help you detect fraud: 1) classification (supervised) 2) clustering (unsupervised)

Fraud Analytics Anomaly Data Science

Now it’s a fair assumption that fraud is pretty rare and it’s an outlier in your data. In other words, it’s a anomaly and the process of identifying them is called Anomaly Detection.

So under classification, there are algorithms out there specialize in “anomaly” detection like one-class SVM and PCA based anomaly detection. Try them out on your dataset and see if it’s able to capture “anomalies” in your dataset. While you are at it, don’t discount traditional classification algorithms either, they may be useful as well. You will have to train these algorithms and that’s why they are called “supervised”.

There an alternate approach. Which is to use unsupervised algorithms called “clustering” techniques. You could try something as simple as K-means or something more sophisticated. I haven’t used clustering much for solving fraud problems and have usually deferred to anomaly detection algorithms for this. But I am throwing this out there for making sure you know all the options! I can see these algorithms being applied to exploratory analysis where you are just exploring your data to find outliers to study them.

Hope that helps!

VIEW QUESTION ON QUORA

What’s a good chart making software that can pull online data?

Standard

So essentially you want to build a *live* chart that pulls data from some online data-store (which changes often).

To do that you can do one of three things:

  1. See if they have an API that you can use — if so, you should be able to use that. If not, continue reading…
  2. Build a web scraper on your own. There are tutorials out there that would help you do so in the language of your choice.
    Chart web scraping data
  3. Use a software service like Import.io | Web Data Platform & Free Web Scraping Tool or Web Scraper — or you could find something else. I have used Import[dot]io and was able to build an API using their service — which i used a data-store for my “charts”

Side note: just make sure you are not violating any terms by scraping the website.

VIEW QUESTION ON QUORA

 

Can I be a data analyst at a tech company without a degree in computer science?

Standard

Yes — it’s not a must have to work as a Data Analyst. In fact, a lot of people come from a non-CS background and succeed in this role!

Let’s look at the pros and cons of having a computer science (CS) degree and this should help you evaluate where you fall:

Data Analyst computer science degree

Pros of having a CS-degree:

  • If the data analyst position requires you to have this degree in CS then you qualify! Fortunately this is not that common and usually it says bachelor’s required in cs, business administration or related field so as long as you have bachelors for positions that require it then you should be fine
  • you might already have the basic tech skills that are needed for data analysis jobs and the CS degree might be used to validate that.
  • you can pick up new tech concepts and tools fast(er) — with the cs background, it’s easier to pick up new concepts & tools — and you need to continuously do that to stay relevant.

Cons of having a CS-degree:

  • Not enough business problem solving experience and/or lack depth in business knowledge — so if you have a degree in business then you come ahead! Especially if your background aligns with the role. For example: if you focused on Marketing in your bachelors and the role is focused around marketing analytics then you might have an edge
  • I have a CS degree and then I followed it up with a masters from a “business school” — so this is just based on my experience but few CS students (without real world experience) are inclined to focus on “automation” and “bleeding-edge” instead of focusing on what the problem needs. Lot of data analysis doesn’t need to be automated or shouldn’t be automated and not every company needs <<insert the latest tech trend here: big data, deep learning>> — but CS students tend to do that. That’s what they feel most comfortable with so while that doesn’t stop from getting the job, this would impede their growth as a data analyst within the org.

Conclusion:

So as you can see even if you don’t have a CS degree, you can still find roles that align with your other skills and in fact, you might be able to come out ahead if you can prove that you have basic quantitative and tech skills needed to get the job done.

Related: Paras Doshi’s answer to How do I prepare myself for a career in Data Analysis?

VIEW QUESTION ON QUORA

How to create a Histogram in Excel?

Standard

Histogram is a powerful data analysis technique — it let’s you quickly see the distribution of the data you have. So in this post, I am going to list the steps to create histogram in Excel.

It’s a two-step process.

  1. Install “Data Analysis Tool Pak” (free Excel add-in)
  2. Format the data and build the histogram

Step 1: Install Data Analysis Tool Pak.

One of the most useful data analysis add-in in excel is not available by default! It’s called “Analysis ToolPak”

To activate it. Go to File > Excel options > Addins > For the manage field, select Excel add-ins

Histogram Manage Excel add-insMake sure that ToolPak is activated and click OK.

Histogram analysis toolpak excel(Also, Solver is a great add-in as well! It’s not in the scope of this article to discuss that add-in but it’s a powerful add-in as well. For instance, it let’s you work on optimization problems)

Step 2: Format Data and build the Histogram

So now let’s format the data.

You need two things to create a Histogram. 1) Data 2) Range

Here’s an example: (I have about 3000 numbers and need to see the distribution)

You could have other fields on the sheet as well but you need at least the data field. Range is optional but I recommend that you specify the Range so that your histogram would have the bins that you specified — otherwise you could have just used a bar chart!

Note that both of them are numerical.

Data Histogram

Now go to Menu Bar > Data > Data Analysis

Data Analysis HistogramOut of the options available, click on Histogram and select the Input Range and Bin Range > after you’re done, click OK.

Data Analysis Histogram ToolpakYou should see a new worksheet with raw data (ready for charting!). Now, create a Bar chart using the raw data and you have your histogram:

Histogram Excel Data AnalysisConclusion:

In this post I listed the steps you can take to create a Histogram in Excel. Note that there are other options as well — like R (hist function) that let’s you build histogram as well so you do have choice of tools but if you want to stick with excel and it’s good enough then you now know how. Cheers!

Related Post: What is the difference between Histogram & Bar Chart?

[Video] AI, Deep Learning and Machine Learning

Video

I watched this video over the weekend and wanted to share this very well done presentation by a Venture Capital (VC) firm with you — that’s why I love following VC’s (especially one’s who invest in Data/Analytics theme) since they tend to share some amazing insights on where the industry is going.

Abstract:
“One person, in a literal garage, building a self-driving car.” That happened in 2015. Now to put that fact in context, compare this to 2004, when DARPA sponsored the very first driverless car Grand Challenge. Of the 20 entries they received then, the winning entry went 7.2 miles; in 2007, in the Urban Challenge, the winning entries went 60 miles under city-like constraints.

Things are clearly progressing rapidly when it comes to machine intelligence. But how did we get here, after not one but multiple “A.I. winters”? What’s the breakthrough? And why is Silicon Valley buzzing about artificial intelligence again?

From types of machine intelligence to a tour of algorithms, a16z Deal and Research team head Frank Chen walks us through the basics (and beyond) of AI and deep learning in this slide presentation.

URL: http://a16z.com/2016/06/10/ai-deep-learning-machines/

What are the differences between big data developer and data analyst?

Standard

It depends on how the Analytics & Data Science team is structured in an org but usually you will see following trend:

  1. “Big Data Developer” usually rolls up under the Engineering org. They are responsible for building the data pipelines that feed data to the “data platform” — they use things like Hadoop, Spark, Custom Code, ETL tools, etc to build data pipelines and are responsible developing and maintaining the data platform. And to succeed in this role you need to have deep technical chops. Other titles for this role: Data engineer, Software engineer, etc.
  2. “Data Analyst” usually rolls up under some “business” team like strategy, operations, growth, product, marketing, sales, etc. Data Analyst are the link between the “data platform” and the “business” — these guys are primary consumer of the “data platform” (sometimes you might see shared ownership of data platform between engineering and analytics). They help solve business problems using data and pull data from the “data platform”. These guys need to have a good balance between business and technical skills to be successful in this role.

View the question on Quora.

What is Descriptive Analysis?

Standard

I classify analytics into four kinds:

  1. Descriptive — Reporting “What” Happened
  2. Diagnostic — “Why” is Happened
  3. Predictive — What’s going to happen next?
  4. Prescriptive — How can I use all these things to take business decisions/actions.

With that overview, let’s look at Descriptive analysis a little bit more. This is usually the first step for any organization to start getting value out of all their data. They should be able to answer questions like: What were my sales last quarter? How about same quarter last year? Then compare them to see if they made progress. They can also report on sales (Actual vs goals) for last n months and see if they are trending in the right direction. Things like this! Once you have a good process and platform to get this right then the organization is ready to advance to next step which is diagnostic and this is where you start analyzing the key drivers and underlying reasons to figure “why” it’s happening. But you need start at Descriptive!

Hope that helps!

view Question on Quora

Does data analysis and machine learning go hand-in-hand or are they mutually exclusive activities?

Standard

Originally published on Quora. Link Here

“Machine Learning” is a subset of “Data Analysis” — it’s just one of the activities that you could apply to solve a data analysis problem, you just need to find a problem that can use machine learning wizardry! What kind of activities?, you say — well, to answer that we will need to step back and categorize what problems could be solved by Data Analysis. There are broadly three kinds of problems:

  1. “What” Problems. Few example: What are my sales number for last quarter? Can we compare it to same quarter last year? Now, can we break it down by Regions and Product Categories? — you see all these questions could be answered by a querying your data stores or by your Business Intelligence platform. Yo do NOT need machine learning for this. Moving on…
  2. “Why” Problems: Few example: Why did the customer cancel their contract? Why is the profit in region A declining Quarter over Quarter? You see this is little bit more challenging than “what” questions — you will need to structure the problem and pull data from multiple sources. Why did customer cancel? You may want to look at internal (e.g. customer complaints) and external (e.g. bankruptcy) data. Usually you won’t need to apply Machine Learning here — you might benefit in some cases where you “cluster” all churned customers and see if you can find some patterns but again Machine learning is not you primary tool here. Moving on…
  3. “What’s next” problems: This what you have been waiting for — this is where Machine learning could be applied. Example: Which customer accounts will cancel their account this fiscal year? — This is where you train a machine learning algorithm to predict which customers will churn this year. Note that the work you did for “why” problems where you identified some characteristics of churned customers will still be applicable here — and that brings me to: Most organizations don’t usually jump from “What” to “What’s next” stage — every organization is at a different stage depending on their maturity and you can’t apply machine learning to every data analysis problem. Also, with more and more companies using “data” to gain competitive edge, if you are not using machine learning then chances are high that your competitor is and they may out-compete you and that’s why it’s important to continuously invest and reach the highest level — more and more companies and executives are realizing this and it’s a great thing for the data community!

To conclude: Depending on the analytics maturity of your organization and the business problem at hand, you might have to use Machine learning to solve a data analyis problem…And it never hurts to pick up Machine learning basics along with other data analysis skills that you might have.

Hope that helps.

Why are there so many analytics startups?

Standard

Originally published on Quora: Why are there so many analytics startups?

Question:

Why are there so many analytics startups in the past 2 years?  With Google Analytics getting better every year (for FREE!), what is the value proposition?  I understand the need to augment with some new perspectives such as Clicktale, but I’m not sure I understand the value prop of KissMetrics, SpringMetrics, etc?

Answer:

There are two main reasons:

  1. Features gap between google analytics (free) and google analytics (premium aka 360 now!) — there are a lot companies (esp. with multi-million customers) that want to use premium features but still cant justify the ROI of GA premium. So there are analytics startups out there that try to cater to these “gaps”. Even though GA is improving, there will always be some feature gap(s).
  2. Access to venture capital for these startups — so these startups found a market and they went for it. They also had access to venture capital (easier two years back then it is today!) and it also helped them that “big data” and “data science” was (and still is!) a highly discussed tech topic.

I believe we will see some consolidation in next few years.