UG7: Probabilities and Uncertainties

“Probability is mostly about the known unknowns. Uncertainty however also recognises the existence of unknown unknowns, and from time to time questions our known knowns.”


Probabilities are for the most part simple arithmetic which people mistake for complex mathematics. As a result most people turn off thinking about them. The lack of an understanding of basic probabilities and uncertainty is a significant factor in arriving at poor decisions, and decreases the likelihood of many things working out as you expect or would like.

Most people fail to adopt an optimal approach in response to the existence of probabilities, ie. chance. The more common examples of faulty thinking relate to the following.
'Even-Chance Bias'
Many people believe that chance events even themselves out in the short or medium term, such that if, for example, a sequence of 13 heads occurred on a coin toss, they will believe that for a while afterwards it is more likely that tails will occur, so as to even out the run of heads.

The concept of odds evening themselves out in this way is, with a bit of thought, clearly absurd. It implies that somehow all the previous coin tosses are somehow ‘remembered’ such that the next coin toss somehow ‘knows’ it now needs to have a higher probability of tails after a number of heads. A mind blowing idea with zero evidence to support it. It makes far more sense to simply recognize the odds of a given coin toss remains 50-50 irrespective of what has gone on before, unless of course there is evidence of a biased coin or deliberate manipulation of the coin toss.
'The Gambler’s Fallacy'
The Gambler’s Fallacy arises from a failure either to understand or to believe that random events are occurring truly randomly. It is wishful thinking on the part of the Gambler that they can beat the odds and come out on top.

After a run of luck the Gambler will convince themselves that it their skill that has given rise to their good results. Even if they then start to lose, they will attribute that to bad luck and continue to believe that their ‘skill’ will eventually win through.
‘Possible is not Probable’
People sometimes convince themselves that because something is possible that it is somehow probable. They become obsessed with something because it is possible without making a realistic estimate of its probability and acting accordingly. In particular they become obsessed with one particular possibility, whilst ignoring many other possibilities that may be equally or more probable.

When faced with a possibility think about what other possibilities might also exist. Assess the likelihood of a given possibility in the light of the all the other possibilities that also exist. With a bit of brainstorming you will often find many more possibilities than might otherwise have immediately come to mind.
‘It is highly likely that unlikely things will happen’
People are often surprised when unlikely things happen. If something is unlikely then people largely assume it won’t happen, and don’t prepare for it. And it probably won’t. However there are many unlikely things, and it is thus likely some of them will happen, it is just that we don’t know in advance which ones or when.

In the real world there are many many things that could happen but are unlikely to do so. We don’t give much if any thought to these. However there are a lot of opportunities for these things happening, and occasionally they will. Thus the general adage ‘expect the unexpected’.

Whilst we clearly cannot prepare for every possible thing that could happen, we can build up reserves which enable us to better handle the unexpected should it occur. We don’t of course know which of the unexpected events will occur and thus can’t prepare in a specific detailed way, but we can prepare in a more general way.
‘Appreciate Regression to the Mean’
Regression to the Mean is whereby after getting by chance a relatively extreme result, the next result is clearly very likely to be less extreme.

Failure to appreciate regression to the mean can lead to causation errors since if we took some corrective action as a result of an extreme occurrence we are likely to attribute the improved performance to our corrective action whereas in fact it may be, and very often is, an instance of regression to the mean. The more extreme the result the more likely the regression to the mean.
‘Not everything is ‘Normal’’
A very common ‘model’ of probability distributions is what is termed the ‘normal distribution’, also called the ‘bell curve’. This is the distribution that gives rise to the concept of ‘standard deviations’ which enable ready calculation of the likelihood of events falling in certain ranges. And indeed if the world behaved in a ‘perfect’ fashion, whereby all the ‘events’ relevant to a given distribution are completely independent, it would be representative of many real world phenomenon. However the world is not perfect, and this means that the assumption that distributions are normal is sometimes not quite right, and as regards more extreme results, sometimes blatantly wrong.

It is very common for a ‘normal distribution’ to predict extreme events as far less likely than in fact they do occur in the real world.

Once you move away from the use of the ‘normal distribution’ it becomes possible for more than 50% of a population could be above (or below) average.
‘Selective Observation Bias’
Probabilities can be and often are widely distorted through selective observation. If you run 20 sampling tests then you will get a spread of results. If you then only present the results from one test at the extreme you will get a heavily biased viewpoint.

Thus, for example, if you have large numbers of tests into a phenomena such as extra-sensory perception, then by chance some of these will not only show a positive correlation, but will do so in a statistically significant manner. Such a study can then be presented as ‘proof’ of the existence of the phenomena. In practice of course this is a just the outlier selection of test results, and what is being ignored are the very much larger number of test results which, when taken with the supposedly outlier results, show no such correlation.

Always be wary of statistical results which are based on relatively small samples. Not only are they more likely to give extreme results, but it is likely they are being presented because they are significant. How many tests were undertaken which did not show the desired results and were thus discarded?
‘Lucky charms and superstitions’
A belief in a lucky charm or superstitious behavior arises as a result of chance being mistaken for causation. We notice that we have our soon to be lucky charm with us and something goes well. It happens again. We only need notice it a few times and the thought is planted that the charm works. Basic cognitive biases then kick in such as confirmation bias, together with the placebo affect and the benefits of positive thinking, and we are away.

Use of a lucky charm may well improve our performance, but it doesn’t do it by magic, it does it because we think it does and we get the benefits of our positive thinking. Recognize this and thus recognize that there are limits to the benefits you might gain. Your lucky charm will not make you immortal.
'What are the odds?'
Our estimate of probabilities in the real world are heavily influenced by our own experiences and by anecdotal stories. Thus for example our estimates of the various dangers to life, cancer, road accidents, etc. will be significantly skewed depending on whether we have known someone to have died that way or the extent to which we remember stories about those who have died in particular ways.

Be very wary of your own, or other people’s estimates of odds. Always look for available statistical data where it is possible to do so.
'The Many Abuses of Statistics'
Where statistics are gathered in an honest manner, by those without a vested interest in the results, then they are mostly reliable, albeit subject to some uncertainty. However statistics can be falsified, and they can be presented in a way that misleads, and when used in support of some argument by those with strong vested interests, particular those that have showed a willingness to use such tactics in the past, they often are.

Some of the many means by which statistics are abused include being selective about what statistics are presented, comparing apples with pears, not stating underlying assumptions or circumstances, use of biased selection criteria, failure to present levels of uncertainty or margins of error, use of inappropriate averages, and using axis offset and scaling to create misleading impressions.

It is so easy to mislead with statistics either deliberately or accidentally, that whenever statistical information is given you need to ask yourself about the motivation of whoever is presenting the information. If they have a strong vested interest in drawing a particular conclusion from the statistics then you need to be wary of it, and look out for the potential for any of the specific misleading tactics. If possible, you should see if there is collaborative information from other parties without the vested interest.
'Appreciate the implication of ‘base rate’'
People generally fail to understand how a ‘base rate’ impacts a probability. Take the example of a test for a rare disease which is described as 99.9% accurate. Many people who test positive for the disease are likely to then believe they have the disease. However a 99.9% accuracy means that the test will misdiagnose 1 time out of every 1,000. If, say, on average, 1 in a million people have the disease, then for every 1 million people tested 1,000 will test positive even though, on average, only 1 of them will have the disease. Thus being tested positive still only means there is 1 in a 1,000 chance of having the disease.

When given a value for the accuracy of a test recognize that this on its own is insufficient for drawing conclusions about likely occurrence. You also need to ask for the ‘base rate’ associated with whatever it is that is being tested in order to calculate the implications of a measured result.

There is a general concept in probability known as Bayes Theorem which ensures that ‘prior knowledge’ is used when estimating probabilities. The use of ‘base rate’ is an example of prior knowledge.
‘Attempt to understand or calculate the expected value’
The expected value is the gain or loss that might be expected ‘on average’ from a given choice. Thus if we have an 80% change of gaining £10 and a 20% chance of losing £30 then the expected value is a gain of £2 ( 0.8x10 - 0.2x30 = 2 ). If offered such a choice it is thus generally a good bet to take, assuming we can afford the risk of losing £30. And if we offered such a bet multiple times then we are highly likely to come out on top.

There are many circumstances where a calculation of expected value will help us determine which of a number of options is potentially the best. Generally we should be looking for options with the highest expected value, and we should avoid any options where the expected value is negative. In the real world however there are other considerations, such as whether we can ‘afford’ an instance of the ‘loss’.
‘Extreme estimation bias’
People have a strong tendency to underestimate very high values and very high probabilities and overestimate very low ones. Whilst we do not come across such values and probabilities often in our everyday lives, where we do so, we need to recognize the bias and account for it accordingly.
‘Type 1 and Type 2 errors’
In a population where some are X and some are not-X, then in determining whether an individual is or is not X two types of errors can occur. A Type 1 error is where the individual is judged to be X when in fact they are not-X: this is what is described as being a false-positive. A Type 2 error is where the individual is judged to be not-X, when in fact they are X: described as a false-negative.

In terms of detector accuracy there is often a trade off between the size of Type 1 errors and the size of Type 2 errors. Ie. the number of false-negatives can be increased or decreased whilst the number of false-positives would be decreased or increased. For example criminal sentencing rules which increase the likelihood of the guilty being found guilty will also increase the likelihood of innocents being found guilty. A far alarm system which more accurately detects a fire will also falsely detect a fire when there is isn’t one.
‘Not all likely’s are equally likely’
People’s estimate of a probability bound when something is expressed as likely, or unlikely, may be very different to someone else’s, and may also be very different when they are thinking about different things.
'Updating probabilities after the event’
Given a belief about the probability of a given event, there is a mathematical formula, known as Bayes Theorem, which provides us with a way of revising the probability in the light of given outcomes.

What is not the case however, as is often claimed by those purporting to be experts in hindsight, is that the non-occurrence of an event does not ‘prove’ that it was always highly unlikely, or that the occurrence of an event ‘proves’ that it was highly likely.
'Only one of two outcomes does not imply 50/50’
Whilst it is blatantly obvious that having two outcomes does not imply a 50/50 chance, there are many circumstances in which people behave and maybe implicitly think it does. ‘We’ll either win or we won’t, 50/50 chance.’ ‘I’ve applied now, they’ll accept me or they won’t, 50/50.’ This will often occur if people don’t have explicit information about what the likely ‘base rate’ odds really are. However just because you don’t have base rate odds to hand it doesn’t mean they don’t exist.
'General good thinking guidelines with respect to probabilities'
For your everyday encounters with probabilistic events, seek as best you can an accurate view of the accepted probability distribution and make your decisions accordingly. This generally involves a calculation of ‘expected value’ and decisions around how to maximize your likelihood of gains, albeit being mindful of the potential for losses.

Be aware of the potential for being misled or scammed. Probabilities and statistics can sometimes be skewed, or lied about, or presented in a misleading way, by those who have a motivation to do so.

Be aware that given probability distributions often have ‘assumptions’ or particular factors that underpin them. These assumptions or factors may not be applicable in a given attempt to apply the distribution.

Be aware that probability distributions themselves are not necessarily fully representative, particularly at the extremes.