HBR has a nice little piece on using statitics that is very applicable to anyone undertaking measurement for communications or marketing. The central tenents are:
-
Know what you know-and what you’re only asserting
-
Be clear about what you want to discover
-
Don’t take causality for granted
-
With statistics, you can’t prove things with 100 percent certainty
-
With statistics, you can’t prove things with 100 percent certainty
The last one is important in communicating upwards to management. Does anyone care you increased coverage 3.2%? Probably not.
Most of what we commonly use and call statistics are meant to describe uncertainty, not certainty. They are not able or were ever supposed to “prove” anything, and definitely not “prove things with 100% certainty.”
Most common statistical techniques are developed from the assumption that a family of data points has a certain distribution and that it is possible to calculate the probability that other data points are part of that distribution. Statistics (generally) provides a probability, and so only demonstrates that something is _not_ certain.
From a mathematical point of view, it is the whole point of statistics that they _don’t_ prove anything. Of course, from a PR measurement point of view, the whole point of statistics is that they _do_ almost prove something.
Interesting post. People often forget that with statistics, there are only two outcomes in any hypothesis test:
1. Reject
2. FAIL to Reject
One cannot “accept” something with statistics. The closest thing to “accepting” a hypothesis is “failing to reject” it. It’s very much the same as not finding a defendant innocent — but rather failing to find the defendant guilty in a jury trial (as in “We the jury, find the defendent ‘not guilty’ on the charge of XYZ.”).
Make sense?