Drawing (Wrong) Conclusions from Analytics

Data-driven design is all the rage these days. Companies everywhere are utilizing big data to make decisions for their products every single day. But, big data does not equal good data. Data without context can drive incorrect, bad, or harmful decisions.

Let’s say you have a mobile app product. You are using an analytics tool like Firebase or UXCam that tells you how people use the app – everything from where they navigate, what buttons they tap, how much money they spend, and how much time they spend in the app. You notice that a certain feature you built & released 6 months ago hasn’t been used a single time. Not. One. Time. Your first instinct could be to remove that feature entirely. No one’s using it, why not? Well, let’s consider a few possibilities. Did it solve the right problem for your users? Did you pinpoint a problem, but your solution wasn’t the right one? Did no one notice that feature was added to the menu? Did the name you chose for the menu item make people think it was something else entirely, but they actually did need the feature you released?

Take this cautionary tale from this InfoWorld blog, for instance.

One of the classic stories of how data out of context can lead to distorted conclusions comes from Harvard University professor Gary King, director of the Institute for Quantitative Social Science. A Big Data project was attempting to use Twitter feeds and other social media posts to predict the U.S. unemployment rate, by monitoring key words like “jobs,” “unemployment,” and “classifieds.”

Using an analytics technique called sentiment analysis, the group collected tweets and other social media posts that included these words to see if there were correlations between an increase or decrease in them and the monthly unemployment rate.

While monitoring them, the researchers noticed a huge spike in the number of tweets containing one of those key words. But, as King noted, they later discovered it had nothing to do with unemployment. “What they hadn’t noticed was Steve Jobs died,” he said.

Infoworld.com article

With data-driven design, it’s important to use a mix of qualitative and quantitative methods. Quantitative data tells you who, what, when, and where something is happening. Qualitative data tells you how and why it’s happening. Non-numerical data is still data.

Consider another seemingly positive metric – time spent on a page. You’ve noticed users in your app are spending a lot of time on one specific screen – GREAT! You’ve delivered something that is valuable to your users – champagne all around. Or is it? Time spent on a page tells you just that – how long someone was there. What it won’t tell you is that, of the potential reasons someone spent an exorbitant amount of time on that page, was because the form they were filling out froze and they had to fill it out three times before it went through successfully, or that they had to read and re-read the page 2-3 times looking for specific information and couldn’t find it. That time spent could have been a positive experience or a negative one – but only one of those will result in better user retention or happier customers.

In order to build user-centered products, you should balance your data with input and feedback from real people. Design decisions should be based on UX research findings and insights from real users. The very definition of user experience is that you should be designing products for and with real people. Data can help improve your product in a number of ways. Data can help:

1. Prove that your product is on track
2. Reveal pain points or opportunities for improvement
3. Discover new trends or patterns
4. Show that your iterative improvements are working

Data can help you find the problem but shouldn’t dictate how you solve the problem. It’s easy to jump to conclusions in an effort to save time. Data is a tool, not a solution. Data should validate your design decisions, not drive them. There’s a difference. Ensure that you’re balancing your quantitative data with your qualitative to build a truly user-centered product.