Assessing an Organization's Analytics Maturity

Written by Ken Ono

Assessing an Organization's Analytics Maturity image

When delivering mobile and web applications that include analytics, it is important to gauge the organization’s current analytics capabilities and how the project can advance them.

Knowing this, the project can deliver the appropriate analytics features and ensure that the investments being made are appropriate and improve the organization’s data-driven processes.

Level 0 - No Analytics

This is getting to be rare nowadays and every Rangle project is expected to include at least basic measurement.

If the sponsoring organization is not familiar with the basics of tools like Google Analytics or Adobe Analytics, be sure to set aside some time (should be less than an hour or two) to get a Google Analytics account and activate pageview tracking. Also take the time to introduce analytics to the stakeholders.

Level 1 - Measurement

At this stage, the organization is able to generate reports showing key performance indicators (KPI’s) that demonstrate how the product or website is achieving its business purpose. Of course, analytics maturity is a spectrum, not discrete steps, and you will find a fair amount of variability for organizations that can be categorized as stage 1.

For example, by simply sending page navigation events to a web analytics tool, you get an impressive number of reports and metrics:

  • Usage reports (pageviews, # of sessions, # of unique visitors both new and returning), average session duration and pages per session, bounce rates and cohort analysis
  • User characteristics (geographic, device and browser and demographics from third-party cookies)
  • How people are getting to your site (organic search and from which engine), paid search and which keywords, campaigns and which display ads and referrers (who linked to your site)
  • Behavior (page ranking, entry and exit pages, site speed, flow between pages)
  • The ability to segment reports based any of these attributes
  • The ability to benchmark your site against peers in your industry

Wow - that’s a lot of stuff you get by just signing up for a Google Analytics accounts and sending pageviews. However, unless some of the metrics above are a reflection of the key business reason for investing in creating a site or app, it is insufficient.

As a general rule for Business to Consumer (B2C) solutions, a KPI is some form of conversion rate, where you have a call-to-action that you would like users to perform. The term “conversion” originally meant to convert visitors into buyers, but in reality there are many forms of conversion such as signing up for emails or getting an account.

The specific KPI’s will of course vary from project type to project type. eCommerce projects are typically focused on orders while marketing site projects may be more interested in subscriptions, downloads or social media likes. Typical usability metrics include completion rate, time on task, error rate and task satisfaction. Enterprise systems are typically more concerned with adoption and task completion.

To enable the reporting of these KPIs, you may need to expand the data that is collected and perform additional configuration in the analytics tools. For example, if a conversion is providing contact information, viewing a “thank you” page can be configured as a goal so that the conversion rate will be computed. And often there are many steps leading up to a conversion so analysts will want to configure funnel reports that show where people are dropping out of the process. Finer grain analysis (e.g. field level drop offs) can also be configured.

Level 2 - Agile Analytics - Build, Measure, Learn (BML)

At level 2, the team is actively engaged in analyzing data and making changes to increase the business value delivered.

To be successful and efficient at BML, the organization requires capabilities that go beyond just the ability to produce and read reports.

For example, business and technical processes must exist to be able to deploy and measure the impact of a change. In order to reliably and accurately assess a new version, the old version and the new version of the app must be run in parallel (otherwise how would you know changes weren’t caused by other factors?). This is called A/B testing. Level 2 organizations will A/B test almost everything and have it ingrained in their day-to-day processes.

There are a variety of approaches to implementing A/B testing. Tools like Optimizely or Google Optimize allow non developers to tweak the app by changing the website as it is rendered. This won’t work for mobile apps and tends to be best for minor modifications to text, colors or layout.

Where developers are available to accelerate the BML process, and good architectural choices have been made, A/B tests can be bolder. For example, significant changes to a process flow or even significant major features could be experimentally introduced. Teams that master this often use a technique called feature toggles that allows large changes to be integrated into the master branch (thus avoiding destabilizing merges) and the ability to turn on or off features with the flick of a switch to all or some users.

Organizations that are effective at leveraging BML are strong at continuous deployment. This requires a broad set of capabilities including strong DevOps and a high degree of automated testing. Consider how an organization’s activities across many different dimensions need to change depending on where they are in the product lifecycle.

Continyous Delivery Maturity Matrix

However, often the major impediments to achieving an efficient BML process are organizational, not technical. For example, there may be significant compliance processes, outdated IT processes or poorly integrated QA teams that can add weeks and months to the process of making changes to customer-facing properties. This elongation can water down the benefits of BML. To be truly effective, the agile processes must extend well beyond the development organization.

The KPIs the team is concerned with evolve over time as successes lead to new challenges, opportunities for improvement present themselves and questions drive the need for more data. KPI’s may be designed for specific features. For example, how much does the new design impact the task completion rate and time on task?

As the organization gets more confident in data driven processes, it opens the door to further optimizations. For example, rather than searching for “global bests” with A/B testing, rule-based personalization could deliver different user experiences to different user types (e.g. repeat and new visitors get a different user experience).

Level 3 - Machine Intelligence Driving Personalized Experiences

In stage 3, business value is accelerated by the use of machine intelligence and, where data volumes warrant, big-data technology.

One use case for machine learning is to automate the process of developing personalization rules. Rather than a human analyst defining the rules, algorithms crunch the data to find the optimal solutions based on past user behavior. This “predictive model” (which is a model of the historic data) can predict the likelihood of an event.

Prescriptive models, often created through constraint-based optimization, are a set of rules to determine the next best action for a specific customer using outputs from predictive models and other attributes.

A/B testing is still performed, but it is used to test the impact of the machine learning models. For example, if introducing machine learning for the first time, you may measure its impact compared to the prior state of the app.

Predictive and prescriptive models are like super compressed versions of the data they are trained with. As such, they have a shelf life because the world is always changing. One way to manage upgrading and keeping your models fresh is to use a “champion challenger” configuration.

In a champion challenger configuration, you have a favored model that is used for the majority of cases. A challenger is run in parallel. For example, 95% of visitors are randomly assigned to the champion and 5% to the challenger. As they run, you collect data and measure the impact of each. When you have enough observations to decide which is better, you promote the challenger to be the new champion (and bring in a new challenger) or replace the old challenger with a new challenger.

Segmentation models (also known as unsupervised learning) are entirely different type of model that can assist marketers in discovering new categories of customers.

Recommendation engines (think “people who bought X also bought Y”), can also be integrated into the solution. Recommendation engines are typically trained primarily from transactional history and have a benefit of automated learning as they run (via an apriori or collaborative filtering style algorithm). As such, this style of analytics can “jump the queue” and be integrated early in analytics maturation cycle.

Conclusion

There are a wide range of analytics capabilities. Very few organizations have reached level 3 to the fullest degree in all interactions with customers. Those that have are busy paving the way to the next level for the rest of us.

So, in almost all cases, there is an opportunity to advance an organization’s analytics capabilities. Here is a cheat sheet on what can be done for organizations at each level:

Level 0 - Not Using Analytics - If the organization is new to analytics, help them reach level 1 by performing a measurement plan, setting up Google Analytics and basic metrics and offering training on analytics, search engine optimization (SEO) and adwords.

Level 1 - Measurement - If the organization is already comfortable getting basic metrics from analytics tooling, help them move towards level 2. Perform a measurement plan, integrate the app with the existing tools, and it may be time for SEO and tagging audits and enhancements. Help establish a BML processes through a BML readiness assessment, feature toggles and trunk based development, assist with establishing DevOps and continuous deployment. Change management consulting can extend agile processes throughout the organization.

Level 2 - Agile Analytics with BML - If the organization has established short deployment cycles, excellent DevOps and are thoroughly data driven, they may be good candidates to optimize the user experience using machine learning. There will be well established data collection systems that the new app will need to integrate with. There may be ways of further enhancing feature management, continuous deployment and DevOps (and related analytics). To extend beyond level 2, the behavioral data can be integrated into a longitudinal customer view that combines touch points for all channels. This omnichannel view can fuel advanced analytics such as unsupervised learning for the discovery of new customer categories and predictive and prescriptive modeling to optimize user experience and revenue.

Level 3 - Machine Intelligence Driving Personalized Experiences - If the organization has level 2 and 3 characteristics, the project scope will primarily be integration. Behavioral information captured from users of the new app will be sent to the data infrastructure and the app itself can integrate with existing recommendation and modeling services to provide dynamic personalization.