How can I start learning and exploring the field of Big Data Algorithms?


Someone asked this on Quora about how to learn & explore the field of Big Data Algorithms? Also, mentioned having some background in python already and wanted ideas to work on a good project so with that context, here is my reply:

There are two broad roles available in Data/Big-Data world:

  1. Engineering-oriented: Date engineers, Data Warehousing specialists, Big Data engineer, Business Intelligence engineer— all of these roles are focused on building that data pipeline using code/tools to get the data in some centralized location
  2. Business-oriented: Data Analyst, Data scientist — all of these roles involve using data (from those centralized sources) and helping business leaders make better decisions. *

*smaller companies (or startups) tend to have roles where small teams(or just one person) do it all so the distinction is not that apparent.

big data

Now given your background in python and programming, you might be a great fit for “Data engineer” roles and I would recommend learning about Apache spark (since you can use python code) and start building data pipelines. As you work with a little bit more than you can learn about how to build and deploy end-to-end machine learning projects with python & Apache spark. If you acquire these skills and keep learning — then I am sure you will end up with a good project.

Hope that helped and good luck!


Can I be a data analyst at a tech company without a degree in computer science?


Yes — it’s not a must have to work as a Data Analyst. In fact, a lot of people come from a non-CS background and succeed in this role!

Let’s look at the pros and cons of having a computer science (CS) degree and this should help you evaluate where you fall:

Data Analyst computer science degree

Pros of having a CS-degree:

  • If the data analyst position requires you to have this degree in CS then you qualify! Fortunately this is not that common and usually it says bachelor’s required in cs, business administration or related field so as long as you have bachelors for positions that require it then you should be fine
  • you might already have the basic tech skills that are needed for data analysis jobs and the CS degree might be used to validate that.
  • you can pick up new tech concepts and tools fast(er) — with the cs background, it’s easier to pick up new concepts & tools — and you need to continuously do that to stay relevant.

Cons of having a CS-degree:

  • Not enough business problem solving experience and/or lack depth in business knowledge — so if you have a degree in business then you come ahead! Especially if your background aligns with the role. For example: if you focused on Marketing in your bachelors and the role is focused around marketing analytics then you might have an edge
  • I have a CS degree and then I followed it up with a masters from a “business school” — so this is just based on my experience but few CS students (without real world experience) are inclined to focus on “automation” and “bleeding-edge” instead of focusing on what the problem needs. Lot of data analysis doesn’t need to be automated or shouldn’t be automated and not every company needs <<insert the latest tech trend here: big data, deep learning>> — but CS students tend to do that. That’s what they feel most comfortable with so while that doesn’t stop from getting the job, this would impede their growth as a data analyst within the org.


So as you can see even if you don’t have a CS degree, you can still find roles that align with your other skills and in fact, you might be able to come out ahead if you can prove that you have basic quantitative and tech skills needed to get the job done.

Related: Paras Doshi’s answer to How do I prepare myself for a career in Data Analysis?


What data are data scientists at startups actually analyzing? How is it collected?


Question: What data are data scientists at startups actually analyzing? How is it collected?
(Coming from a web analytics background I’m wondering what data are data scientist at IT companies actually analyzing. Is it server-side or client-side? Is it collected internally or using some external tool?)


Part 1: What are startups analyzing?

It depends on the Business Model and the Stage that they are at.

Business Models: Marketplace, Ecom, SaaS, Media, etc.

Stage: Early, Mid, Late

So let’s say you have a SaaS model and you’re in Mid-stage (post product-market fit stage) then you would tend to be focused on things like: Engagement, Churn, etc…and ideally they should be focused on measuring what aligns best with the strategy (instead of capturing everything!)

Let’s take another example. Let’s say you are a Marketplace in late-stage. So you would tend to be focused more on the “money” and so you can measure things like: transactions, commissions, etc…

I recommend reading “lean analytics” book as it goes much deeper and it’s a great starting point for anyone to understand how analytics could help a startup.

Part 2: How is it collected?

Now this also depends on your product. Assuming you’re a tech startup, you would have Web App and/or Desktop app and/or Mobile app. And now depending on your delivery approach plus your measurement needs, the “how” part will be determined. It would invariably be a combination of your transactions data source, web/mobile events stack (like Google analytics/other-Vendor or Custom), finance data source among others.

This post points to 10 other blogs which lists their “data” stack: The Data Infrastructure Meta-Analysis: How Top Engineering Organizations Built Their Big Data Stacks – The Data Point

View Question on Quora

What is the title these days for a person that assures data quality?



What is the title these days for a person that assures data quality?
(I need to hire a person to make sure my data is as good as it can be. They need to inspect the data for issues, create logic for how it can be found and fixed, and finally, court the project through application development for a robust solution to stop it from occurring in the first place.)


Quality of the data shouldnt be a responsibility of just one person — ideally, you want all members of the team (and broader business community) to care and own some part of it. But i like the idea of one person owning the “co-ordination” of how this gets done. It might not be a full time gig in a small org but can see this as a full time role in bigger orgs and enterprises. Some titles:

  1. data co-ordinator
  2. Data quality analyst (or just data analyst)
  3. Data steward
  4. Master data management analyst
  5. Data quality engineer (or just data engineer)
  6. Project manager (data quality)
  7. Manager, data quality and master data management

Read the original question on Quora

What are the differences between big data developer and data analyst?


It depends on how the Analytics & Data Science team is structured in an org but usually you will see following trend:

  1. “Big Data Developer” usually rolls up under the Engineering org. They are responsible for building the data pipelines that feed data to the “data platform” — they use things like Hadoop, Spark, Custom Code, ETL tools, etc to build data pipelines and are responsible developing and maintaining the data platform. And to succeed in this role you need to have deep technical chops. Other titles for this role: Data engineer, Software engineer, etc.
  2. “Data Analyst” usually rolls up under some “business” team like strategy, operations, growth, product, marketing, sales, etc. Data Analyst are the link between the “data platform” and the “business” — these guys are primary consumer of the “data platform” (sometimes you might see shared ownership of data platform between engineering and analytics). They help solve business problems using data and pull data from the “data platform”. These guys need to have a good balance between business and technical skills to be successful in this role.

View the question on Quora.

Why are there so many analytics startups?


Originally published on Quora: Why are there so many analytics startups?


Why are there so many analytics startups in the past 2 years?  With Google Analytics getting better every year (for FREE!), what is the value proposition?  I understand the need to augment with some new perspectives such as Clicktale, but I’m not sure I understand the value prop of KissMetrics, SpringMetrics, etc?


There are two main reasons:

  1. Features gap between google analytics (free) and google analytics (premium aka 360 now!) — there are a lot companies (esp. with multi-million customers) that want to use premium features but still cant justify the ROI of GA premium. So there are analytics startups out there that try to cater to these “gaps”. Even though GA is improving, there will always be some feature gap(s).
  2. Access to venture capital for these startups — so these startups found a market and they went for it. They also had access to venture capital (easier two years back then it is today!) and it also helped them that “big data” and “data science” was (and still is!) a highly discussed tech topic.

I believe we will see some consolidation in next few years.

“4W” framework for assessing your Analytics Maturity:


Most organizations could benefit from Analytics but before you set the Analytics road-map for your organization, it’s important to figure out your current stage and then build the road-map to achieve your vision. So how do we figure out the analytics maturity of an organization? Let me share a framework to think about this:

I have blogged about “Business Analytics Continuum” before — it’s a great framework to think about Analytics maturity in an organization — BUT the issue is that it’s harder for business people to remember the stages: Descriptive -> Diagnostics -> Predictive -> Prescriptive — And so there’s a simpler (but equally effective) framework that I have been using over past few months (What -> Why -> What’s next aka “3W” framework). And recently at a Microsoft Analytics conference, I saw this framework with an extra “W” which makes total sense that I liked a lot! So i thought I will share that with you all. So here you go — 4W framework:

Stage 1: What Happened?

Stage 2: Why did it happen?

Stage 3: What will happen?

Stage 4: What should I do?

Analytics Framework What Why Whats Next HOW

Credit: Microsoft Data Insights Summit

I hope the framework as you think about your organization’s analytics vision/road-map and stages that you need to go through to help your org succeed with data!

Building data driven companies — 3 P’s framework.