What Car Dashboards Teach About Choosing Metrics


Before adopting a metric by which you will be steering your business, you should know three things: the domain of what you want to measure, the range of the metric, and the responsiveness within the range to changes in the domain.

For anyone who napped in class, here are the math terms:

  • A metric is a function that converts real-world raw data into a numeric representation.

  • The domain of a function is all the values that could be inputs to it, so the domain of a metric is all the possible real situations it must represent.

  • The range of a function is all the values that could be outputs from it. The range of a metric is all the numeric values it can have.

  • The responsiveness is the change in the range that happens for a given change in the domain.

Your car's dashboard features many metrics:

  • The fuel gauge's domain is the position of the float inside your gas tank, and its range is a needle position that goes from E to F.

  • The speedometer's domain is the torque on the speedcup or the frequency with which a magnet goes by a detector, and its range is a needle position or digital readout that goes from 0 to around 140mph or 225kph.

  • The oil light's domain is a signal from a pressure gauge inside the engine, and its range is on or off.

  • The tachometer's domain is a frequency count on a magnet attached to the driveshaft, and its range is about 0 to 7,000rpm, with a red zone marked at the top.

The responsiveness is different for each metric:

  • The fuel gauge starts to move down from F only when the float is no longer jammed against the top by the gasoline. It reads E as soon as the float is lying on the bottom (when there's not enough gasoline to hold it up). So F does not mean all the way full, and E does not mean completely empty.

  • Most speedometers are set to read at around 110 percent of the calculated value -- not a big difference when you're doing 10 and it says 11, but significant in terms of avoiding a ticket if you're really doing 65 and it says 71.5.

  • The oil light reduces a very large domain of possible operating pressures to a simple range: OK or not. It is usually set to allow a slight margin, so that when it comes on, you can get somewhere close by and add oil or have the engine checked, but there's not much slack in it. (It comes on briefly as you start the engine because it takes a couple of seconds for the pump to move oil up to where the sensor is.)

  • The tachometer tends to be exactly accurate. People use it to check idle speed to see if they need a tune-up and to train themselves to manual-shift for fuel efficiency.

Notice the real key here: The responsiveness of the range to the domain always has a purpose. Fuel gauge responsiveness is set so that, if you use it right, you don't top up unnecessarily, and you rarely run out of gas. Speedometer responsiveness is set to keep you out of traffic court. The oil light doesn't scare you unnecessarily or require you to think about it much, but it warns you strongly when there's trouble. (If you pay attention to it every time you start your car, you always know it's working.) It responds to a huge domain with a tiny range. And because the tachometer is for slightly more sophisticated drivers who care about the car, its responsiveness is close to 1:1 -- i.e., extreme accuracy.

You see -- four purposes, and four different kinds of responsiveness.

So if your business is going to be data driven, asking "What can our raw data's domain be?" is reasonable. For example, can sales be negative? (Sure, if returns are allowed.) Can net revenue per transaction be infinite? (Yes, if money arrives in a way that you don't classify as a transaction.) What will the range of the metric be? (If it's always the fraction a subset takes up out of a larger set, for example, it has to be between 0 and 1. If you always use up some material in starting the machine, material efficiency can't go up to 100 percent.)

And, most importantly before you roll something out, what responsiveness will cause the most desired behavior in the user?

John Barnes, Freelance Writer

John Barnes has published 30 commercial novels (mostly science fiction,including two collaborations with astronaut Buzz Aldrin), 53 articles in The Oxford Encyclopedia of Theatre and Performance, more magazine articles than he can remember, and around 30 short stories. Tales of the Madman Underground, Barnes's first "officially" young adult novel, received a Printz Honor Prize at the 2010 American Library Association national convention, and his technothriller, Directive 51, was briefly on the New York Times bestseller list in 2011. His 1990 article, "How to Build a Future," about applying social science forecasting to creating backgrounds for science fiction, has been widely reprinted, and he's still getting email about it. In his twenties, John worked in an R&D shop on reliability math applied to the problems of relational databases and testing/validation; in his thirties he consulted on the connection between document systems design and natural language interfaces. He has taught college courses in theatre, communications, literature, writing, mathematics, political science, economics, and philosophy, and written what was probably the most math-heavy theatre dissertation ever (applying statistical semiotics to the problem of defining basic terms in theatre history). Recently he has pioneered applying statistical semiotics to strategic, analytic, and tactical marketing problems, poll analysis, and trendspotting, and consulted for a variety of firms and government agencies. He lives in Denver, Colorado.  His personal blog is Approachably Reclusive.

Nonlinear Problems: When Brute Force Is Best

Analysts would do well to get out of the rut of using linear regressions by default.

Analyze for Accuracy or Precision, or Vice Versa

Sometimes your results require accuracy and sometimes precision. Knowing the difference matters.


Re: Conditional responsiveness
  • 2/24/2012 10:37:18 AM
NO RATINGS

John,

This is one of the best analogies on setting an expectation from a dashboard I have read.  It does make clear aspects from metrics beyond a score that summarizes the results.  And from a former automotive engineer, you've nailed the idiot lights very well!

Thank you for this insight!

Re: Metrics and Analysis
  • 2/7/2012 10:42:26 PM
NO RATINGS
1 saves

John thanks for the clarification. From your clarification it seems that the interpretation of data matters a lot. System analysis and interpreting the data for a fine tuning is crucial.

Re: Metrics and Analysis
  • 2/7/2012 9:54:46 PM
NO RATINGS

Hi John,

I like the concept of analysis as presentation. When you think about the dashboard analogy, the metaphor follows through with a simple presentation of all your automobile's basic functions displayed in easy view and distilled to easily interpreted symbols. All meterics should be considered with this in mind.

Re: Metrics and Analysis
  • 2/7/2012 1:40:50 PM
NO RATINGS

Thanks for some wonderful extensions on the metaphor!

I'm aware of a couple of moderately odd situations where you might want to have a tach on an automatic -- mainly ones where you have to use the "1" and "2" gears that some automatics still have.  But otherwise, yes, they're the equivalent of a manager getting a number every morning because his grandfather needed it.

And yes, there are occasions on which the dashboard (or dashboard design) has usurped management's decision-making powers.

Re: Metrics and Analysis
  • 2/7/2012 1:00:01 PM

The car dashboard is also a perfect illustration of metrics outliving their usefulness.  Will someone tell me why I have a tachometer on my car with the automatic transmission?  Or maybe it's a matter of the management (me) not being educated about what it can tell me.  Right now it's just taking up space on my dashboard because I ignore it.

Also inadequate metrics.  I have two lights come on - check engine and maintenence required.  It warns me of something but I don't know what!  It tells me that whatever it is is best left to the experts.  I know it's true but I can't help feeling I'm being manipulated into taking my car to the expert.  Do you have metrics that give management the impression that you're just trying to maintain your expert status?  You know it happens.

Re: Metrics and Analysis
  • 2/7/2012 8:36:48 AM
NO RATINGS

But analysis is presentation; in any math problem, the answer is already there (or the fact that the answer can't be reached by mathematical means, which is an answer of a different kind).  The manipulations we do only "solve for the result" in the sense that we pull that one answer out of the tangle of the other numbers, because it's the answer we want to look at.

And the car dashboard analytics are predictive -- it's just that the driver has to interpret them into predictions. 75 km/h means that in one hour, we will be 75 km further along than we are now ("if nothing changes," or "ceteris paribus" -- the magic get out of responsibility free phrases in prediction). The fuel gauge predicts whether you will or won't run out of fuel in the relevant timeframe, the oil gauge predicts, very strongly, that if you don't do something right away your engine will irreparabily seize up, and the tachometer predicts that if you shift now, it will be smooth and fuel-conserving.

Your comment, though, very strongly highlights an important principle -- although analysis may specify a prediction or describe a range of conditions, it is up to management (i.e. the driver) to decide what to do about it.  The speedometer predicts that "we'll have to speed to get there on time" but management then chooses between being late and speeding (with the attendant risks).  And the oil light cannot save management that decides it's just one little light and there's only a hundred miles to go ... as happened with a certain ex spouse of mine ... it has already done its job when it predicts "big trouble real soon if you don't get to someplace where you can turn the engine off."

We measure, analyze, and predict, but for the most part we don't drive.  It's a good idea for the driver to understand what we tell him/her, but we can't force them to understand; all we can do is report clearly, about the things that are likely to matter to them.  (Which is what the car dashboard is a great example of).

Re: Conditional responsiveness
  • 2/7/2012 8:00:29 AM
NO RATINGS

The rhetoricians refer to it as the difference between data and construct (data is the domain, construct is the range) and observe that the construct that is the point of one part of the process is the data at the next.  The highway cop's radar gun takes a complicated relationship of microwave pulses (data or domain) and turns it into a number representing your speed (construct or range), which becomes the data he turns into the  construct of deciding  to stop and ticket you, the ticket becomes the data for the judge which he turns into the construct of the fine, and so forth. This is the problem with the "everyone is entitled to their own opinion but not their own facts" declaration that people make so confidently; as the philosopher Richard Rorty said, facts don't speak for themselves, they don't speak at all.  People do, and they find it useful to categorize some of what they say as facts.  (They also find it useful to make up rules about what goes into the domain/data and range/construct and to enforce those rules; "you can say what you want" does NOT equal "and it will work equally well.")

Metrics and Analysis
  • 2/7/2012 5:13:37 AM
NO RATINGS
1 saves

John, it's very interesting to convert all the basic metric aspects to a car dash board. Metrics can compare and correlate with any factors but is it right to say the car dashboard is analyzing the data for a best prediction. I think NO, because there is no possibility of analysis and predictions. It's only converting the available data to a suitable format for display.

Re: Conditional responsiveness
  • 2/6/2012 11:28:51 PM
NO RATINGS

Hi John,

I like the idea of being able to customize the view of data you want to see. It makes it easier to measure the information of most importance based on your business model. It's also a good reminder of the importance of customizing analytics to meet your specific business goals instead of taking a one-size-fits-all approach.

Data Overload
  • 2/6/2012 7:56:00 PM
NO RATINGS

In the analytics world, people sometimes have a tendency to gather and present too much data.  Data overload can defeat the purpose of what the data was originally intended for.  John makes a great point in using the car dashboard example as putting data analytics back into perspective.         

Page 1 / 2   >   >>
INFORMATION RESOURCES
ANALYTICS IN ACTION
CARTERTOONS
VIEW ALL +
QUICK POLL
VIEW ALL +