We've all had the experience---over and over all the time. You go down to the street to wait for the bus (the train, the subway, the boat); you know that buses come roughly every 10 minutes, so you expect to wait about 5 minutes (arriving, on average, in the middle of the between-buses interval). But in fact, we all know that almost always you have to wait longer than that! Is this an illusion we’ve developed over the centuries because we believe in the “persistence of bad luck,” or is it, perhaps, something real? It is, in fact, a real phenomenon, and this result can even be proved mathematically. Because you arrived after the last bus has left, your overall waiting time is, on average, longer than half the average interval of 10 minutes. An intuitive way of seeing this is to draw the timeline, with short and long intervals---their average is indeed 10 minutes long, but by randomness some of them will be longer and some will be shorter than the stated average. Your appearance at the bus stop is also a random event, and this event is more likely to take place during a long interval between two buses than during a short one! This is probability theory at play: The moment of your arrival at the bus stop is like a dart thrown at the timeline; this dart is more likely to "hit" the number line during a long between-buses period than during a short between-buses stretch (bus arrival times are shown by an "X" on the timeline below): ----X------------X-----X---------------------X---------X-----------X-------- More likely arrival here ^ than here ^ (on the line just above).
The Inspection Paradox
The idea described here is called the "Inspection Paradox," and it says: When you "inspect" a process, you are likely to find that things take longer than their (uninspected) average. The concept seems to contradict what "average" means. But it doesn't. You "inspect" only one interval between buses, not the entire distribution. The actual interval you "inspect" by going down to the bus stop is a special one: it is one that has a higher probability of being chosen the larger it is. So the selection of this interval has a built-in bias toward longer intervals, and hence a more extended waiting time than the average of all such intervals. The inspection paradox has been studied extensively by Sheldon Ross of the University of California at Berkeley, who has given several different rigorous proofs of this result in renewal theory
.* This apparent paradox appears in many different guises in everyday life---some of them are associated with good luck rather than bad. For example, the battery in your flashlight right now will last longer, on average, than the “average” battery (hence the “paradox”). And someone who is 60 years old will likely live longer than the “average” person. This is because that person has already lived 60 years and cannot die at an age younger than 60---in the same way that if a bus hasn't arrived for 20 minutes before you come to the bus stop, the interval between the two buses can't be smaller than 20 minutes.
The Puzzling Case of Longevity
This last guise of the inspection paradox affects longevity estimates in unexpected ways. The 2013 life expectancy figures for various countries in the world
show that Monaco is number 1 with an overall longevity of 89.63 years. Other sites show Israel ranked as number 4 in the world
with an overall life expectancy of 82 years. I want to concentrate on these two countries as they represent a strong distorting effect of the inspection paradox. The case of Israel would appear to be the most puzzling. Israel is in a region that has some tropical illnesses such as the West Nile virus, it suffers from wars and acts of terror that have claimed many victims, road deaths are relatively high, smoking is more prevalent than in the United States (whose life expectancy is ranked 33 on the list on which Israel is number 4!), and stress and stress-related illnesses are prevalent in the country as well. So why would Israel be ranked as fourth in the world for longevity? The answer is the inspection paradox. Israel is a country of immigration. Between 1989 and 2002, 1.1 million Russian Jews immigrated to Israel
---constituting more than 1/7th of the entire population. And this statistic doesn't include the 600,000 immigrants from Arab and Muslim countries who reached Israel by 1972
, as well as other immigration waves over the decades. Every immigrant to a new country raises that country's average longevity by a small amount for the simple reason stated above and derived from the mathematics of the inspection paradox: If a 40-year-old man immigrates to a country, he can no longer die at an age lower than 40 and therefore ultimately contributes to his adoptive country's upper end of the age-at-death distribution. If a 98-year-old woman immigrates to some country, she immediately raises the country's age-at-death statistic because she is already older than most people in that country who've already died. The contribution of a single immigrant is tiny, of course, but when you have millions of people immigrating to a country such as Israel, they artificially inflate that country's "overall life expectancy" figures. It's an (unintended, or uncorrected) example of lying with statistics. What about Monaco, ranked as number 1 on some worldwide longevity lists? Monaco is a kind of "artificial country." Being the most famous (or infamous) tax haven in the world, it has attracted the very rich, who claim citizenship in this tiny principality on the Mediterranean, thus "immigrating" there, at least on paper. (Many wealthy Europeans claim Monégasque residence so as to avoid paying taxes in their home countries.) So Monaco is, at least nominally, an immigration country, and given its small population, the volume of immigration affects its longevity statistics perhaps as much as it does Israel. In addition, the very wealthy Monégasque have access to some of the best healthcare facilities in the world, and tend to live longer---further raising the principality's longevity statistics.
The Math of Luck
The inspection paradox is a little-known result in renewal theory (part of an area of mathematics called stochastic processes), which subtly affects many kinds of processes we encounter in everyday life as well as in economics, science, and pure mathematics. In fact, using this result, one could prove that a search for prime numbers (the subject of a previous post
) would be more efficient if done systematically rather than by choosing intervals of numbers at random (because that would make these "inspected" intervals artificially longer, given that prime numbers seem to be randomly scattered among the integers). *More details and applications of this little-known theorem in stochastic processes (and another proof) can be found in Sheldon Ross, Introduction to Probability Models, tenth edition, Academic Press, 2009. The most elegant and simplest proof is the following. If F(x) is the (continuous) probability distribution of the inter-arrival times, then by definition 1 - F(x) is the probability that the waiting is longer than x. Now, when you "inspect" an interarrival time, it has already lasted a period of time, s (say, 5 minutes since the last bus left at the moment you arrive at the bus stop). We want the probability that the interarrival time for this particular interval will be greater than x, given that it is already known to be greater than s. Let the length of an interarrival time be the random variable X. Then this probability is given by: P(X>x|X>s)= [1 - F(x)]/[1 - F(s)], which is greater than 1 - F (x), since the denominator is less than 1. Image by Robert Hoetink / Shutterstock
