Posted on
by
Lies My Data Told Me
Thor Muller
Get Satisfaction
@tempo
#datalies
  • 12.2 minute: average time to completion to close out a customer service ticket
    • for thousands of companies, over decades, this has been a key number that companies manage.
    • Companies look at customers as to-do items, and so less time is better.
    • But we could have
  • Data is backwards looking.
    • Imagine if you are driving a car, but you can only look in the rearview mirror.
    • We can project out with trendlines, and pattern match, and extrapolate — people will say this.
    • But you could say the same about driving looking in the rearview mirror. In general, the road ahead will be much like the road behind. But this doesn’t help you when you come to a Y in the road.
  • LTCM – built the algorithm to look at historical data and to tell them what trades to make. It worked great. Until Russia defaulted on obligations, causing a near global collapse.
  • Launchrock
    • Helps companies launch a business… Cowritable has viral growth even before they have launched. However, success getting people to subscribe for a product they have seen does not predict what will happen when the product comes out.
  • [missed a bunch of notes here while I got distracted reading the Launchrock site.]
  • Data Can Be a Real Killjoy
  • Success often requires patience or a patch… rather than a pivot.
  • When to ignore the data
    • When you are in “blue sky” creative mode.
      • don’t think in terms of constraints. don’t limit your imagination.
    • When available data is too general or ill-fitting.
      • e.g. “2/3rds of all startups fail” — that’s not useful for evaluating a specific startup.
    • When you’re unsure of what matters.
      • e.g. do page visits matter?
  • The Optimization Trap
    • “Multivariate testing caused us to design a product that we’ve come to hate. It’s ugly and pointless, but it generates real revenues so we can’t kill it.”
    • Optimizing your product can it is more like it is, but it can’t get you to a new product.
    • Optimization brings you to a local maximum, rather than a global maximum – you might get a better buy button, but you won’t get a whole new design that gets you 100x customers.
  • The squeaky wheels get the grease
    • OpenID vs. Facebook
    • Once had a very vocal part of community would complain about our OpenID community.
    • Whereas Facebook users hardly complained at all.
    • But Facebook accounted for 50% of users, whereas OpenID was a fraction of a percent. The OpenID people may have been vocal, but it wasn’t really the business relevant thing to do.
  • Data is Amoral
    • Data makes us blind to our own values — people making mortgages to people who couldn’t pay them. It just couldn’t make sense. But since it was returning money, the data said to invest in it.
  • The Tyranny of the Obvious
    • If you spend all your time looking at the strong, concrete data points, you are looking more than ever in the rear view mirror, because you are seeing the strongest trends.
    • If instead you look at the oddities in the data, you can see the yet-to-emerge trends. Like Lego, who found a teacher doing robotics, which turned into the Mindstorm’s product line, highly profitable.