According to Gartner, product analytics, “help manufacturers evaluate product defects, identify opportunities for product improvements, detect patterns in usage or capacity of products, and link all these factors to customers". The benefits are clear. But there are barriers – product analytics are expensive in terms of time, people and expertise.
What are the barriers?
Companies take one of two approaches to product analytics. The first requires instrumenting your code to generate the data needed in a usable format. For this, you need to know ahead of time what data is needed and be prepared to invest in code instrumentation. You also have to build and manage infrastructure to ingest and store the data. There are some excellent off-the-shelf tools that consume instrumented data (e.g. APM - Application Performance Management, monitoring tools, web analytics, etc.), but these tend to only focus on specific use-cases. Companies that take this approach usually have a backlog of instrumentation requests for engineering.
The second method doesn’t require any additional instrumentation. It leverages existing machine data – log files and metrics – which contains details of how the product is performing and a record of every interaction between the customer and the product. But existing machine data is mostly unstructured, requiring significant time and expertise to extract value. Each new analysis requires data inspection, custom scripting, regexes and data wrangling. Because data volumes are large (often terabytes a day), efficient pipelining and infrastructure is required. Queries usually go through a specialist team that invariably has a large backlog of requests.
At Zebrium we have met with many companies that spend millions a year on people and infrastructure (cloud or on-prem.) for product analytics. And in every case, there is still a large backlog of product analytics requests. For this reason, most companies leverage product analytics only for well-defined, repeatable and high-ROI dashboards where the expense can be clearly justified.
If there were no barriers...
Some of the most impactful uses for product analytics involve custom, ad-hoc or complex queries. Unfortunately, these requests don’t always make the cut based on time, effort, money, etc. Let’s look at a few:
-
Troubleshooting - Machine data is used extensively for troubleshooting. Many troubleshooting situations also require sophisticated analytics to get to the root cause. For example, tracking down a memory leak by charting memory usage against what the customer was doing. We frequently see scenarios where senior engineers have spent countless hours building scripts and massaging customer log bundles to track down a subtle problem that couldn’t be reproduced in the lab.
-
Evaluating risk - What happens if you uncover a critical bug, but the fix is complicated and the QA cycle lengthy? Without the right information the safest thing might be to drop everything and focus on getting a fix out as quickly as possible. On the other hand, if product analytics allowed you to understand that only a small fraction of customers are susceptible, you could deploy a much simpler point solution for just those customers.
-
Bug prioritization - this is a challenging exercise, requiring engineering teams to make judgment calls around customer impact, time needed to fix, risk, etc. Instead, product analytics could be used to not only understand who has hit the issue, but also exactly who is susceptible to it. This type of accurate insight would otherwise be impossible.
-
Product decisions - understanding exactly how customers are using your product is the ideal lens for these decisions. Rich usage reports can highlight which features require more investment and which can be deprioritized. Knowing how customers use your product helps to understand their likelihood to spend more (or less), and helps you determine how to price and position the product.
Zebrium removes the barriers
The raw data that feeds product analytics is already being created by most hardware, software and SaaS products, but it’s trapped in unstructured machine data. This makes it slow and labor intensive to perform product analytics. Zebrium changes this by using machine learning to perfectly structure and place it into an analytics-optimized, highly-scalable database (see Structure is strategic), together with a table of contents for easy navigation.
Given this structured, well-organized data repository, anyone can answer a product analytics question easily – no more waiting for engineering to instrument the code, no more scripting, data wrangling, etc. The benefits of structuring are many:
-
You don’t need to know ahead of time what data you need – it’s already structured and waiting to be browsed or queried
-
Standard tools (like Tableau or Excel) can be used to visualize the data and create reports; the data can be easily joined with other systems of record (CRM, financials, etc.)
-
Complex queries can be run at high-performance and at scale using standard SQL
-
The structured data can be leveraged for data science and machine learning
By opening up structured access to machine data, we believe the sky is the limit for leveraging product analytics. If you’re a technology vendor looking to build, or have already built, product analytics capabilities, we’d love to hear from you.
