CASE STUDY

Making better product decisions at scale and across multiple platforms

The Client

A growing cross-platform mobile app

Our client was a freemium text-to-speech app available on iOS, Android, Chrome, Mac, and Web, with a user base of about 40,000 users. As of the past 12-months prior to our engagement, the company was experiencing rapid growth in their user base amounting to +100% in their overall user base, and just under +200% for daily active users. 

The Challenge

An unscalable product analytics workflow

The teams' existing process was built piece-meal, and required changes to the client-side code to answer each new question.

Our client faced three challenges with the way in which they managed their product analytics:

1. Scalability: The company was historically reliant on user interviews and small beta test groups of 5-10 self-selecting users to make product decisions. Given the rapidly increasing user base, a new process was necessary to understand user needs and guide product direction at scale.

2. Strategy: The team had been using analytics to answer each new question at a specific point in time, and there was no overarching strategy in place. This meant adding new event-tracking code to the app when they needed to answer a new question, which required significant work and a new release each time.

The team's existing process to answer new questions required them to go through the entire process each time.

3. Granularity: The user base was split into four major groups of Students, Reading Disabilities, Parents/Teachers, Professionals, each with different use cases. The existing system was not designed to distinguish between these user groups, leading to biased conclusions or a product direction that was too optimized for one user group at the expense of the others.

The Solution

Build a scalable way to understand user needs and inform product decisions

To solve this problem and put the team back on track to develop the product in alignment with user needs, we designed and introduced a new process, along with a system to support that process along four main areas of concern:

1. User Behaviour: Understanding the explicit and implicit needs of users
2. Product Fit: Measuring how well the product meets the needs of users
3. Subscription: Understanding the current revenue picture
4. Conversion: Understanding the key revenue drivers to optimize on

Design

First, we worked closely with the client product team to define the objectives for an improved analytics system. This process started by our team developing a strong understanding of the client’s product, its core functionalities, and user groups, then working with the client team to develop a plan to present the new data in a way that minimized disruptions to their existing workflow.

We then defined key indicators for each of the areas of concern, ensuring that each indicator conformed with Google’s HEART method and conformed to the MECE (Mutually Exclusive, Collectively Exhaustive) principle. Doing so ensured that these indicators would only impacted when the direct area of concern was affected and remained as independent as possible from other factors. This made it easy for the product team to immediately identify failure points or drill-down to fix a specific bug, as the reason for any change in any particular indicator could be isolated to a specific component of the application. 

Then, we conducted a review of the existing events defined in the client-side code, to determine how to grandfather in the old information while adding new events that increased visibility. We then consolidated these events into high-level groups, again applying the MECE principle to ensure comprehensive coverage across the functionalities of the product.

Implementation

To implement this new data architecture, we designed a simple pipeline intended to provide flexibility for all downstream applications and usage of the data. To do this, we refactored the team’s existing instance of Firebase and BigQuery to create intermediate tables, which, at each step, represented a set of transformations or reorganization of the data that supported the subsequent set of transformations.

The new analytics process, with intermediate tables in BigQuery to provide maximum flexibility and scalability.

The first set of tables represent “clean” data, with no significant transformations or aggregations performed. This includes things like expanding nested fields into separate columns, removing redundant or unnecessary fields, and standardizing syntax, dates, times, and data types. 

From there, the data flows into a second set of tables, with each representing a different concept such as “Users”, “Subscriptions”, and “Engagement”. Each table combines data from one or more “cleaned” data tables, with aggregations performed to obtain metrics such as the number of words-listened per day, or the total value of a user’s subscription payments over time.

By defining the key indicators in advance and developing a comprehensive strategy that covered the full range of product functionalities, we ensured that no significant changes would be required upstream of these tables to answer the majority of new questions that may arise for the team in the future.

The new process to answer questions, which only requires querying the base tables in BigQuery rather than changes to the app or any code upstream of the BigQuery ontology tables.
Delivery

The client team had already been using Google Sheets extensively to support their data analysis, and we believed that changing this workflow and adopting a new process would require a significant investment of time and effort.

Therefore we focused on training key individuals on how to use Google App Scripts to ingest data from BigQuery and populate their existing spreadsheets, which could easily be refactored into a real-time dashboard at a later date.

The Results

A scalable and flexible product analytics system

The overall result was an 80% reduction in the time it took to answer questions about the product’s performance, user behaviours, and revenue generation. The new process also reduced strain on engineering time, as it eliminated the need for a new release each time the team wanted to answer a new question that had not been previously included as an event in the application. 

Because this system was designed to be platform agnostic, the team was also able to deploy this system to a new web application that completed development shortly following our engagement, further increasing their ability to shape product direction to user needs quickly, and with minimal effort.

Download the deliverable report