I’m sure you’ve heard of this popular saying. It’s the optimist’s mantra, “everything happens for the best”, so don’t be sad when life gives you lemons, as they will soon somehow turn into lemonade. The religious ones even have an explanation for it: It’s God making it so, because he loves us! Realists and skeptics of course dismiss this as complete tosh but in practice, they still match the same pattern.
What pattern, you might ask? Think about it: When someone is narrating their life story, they will go into detail about the hurdles, the obstacles, the failures, the mistakes and misfortunes, but will usually conclude that they were actually all for the best and if they could go back they wouldn’t change a thing. It almost seems that there is empirical evidence everywhere to support the naïve cliché of the title. What’s going on?
It turns out that science actually supports this saying, but not in the way you probably believe it. Yes, everything does happen for the best, but only because we end up thinking so. It’s a psychological immune system of sorts, where when we’re stuck with something, our brains will gradually make us think it’s for the best.
I’ve observed this phenomenon in my myself and others several years ago, and suspected it must have something to do with self-delusion. Clearly, not all our choices are optimal, yet we almost always end up thinking they were. Consider the common example of teenage pregnancy. Despite conventional wisdom that having a baby in your teens is not the smartest thing to do, those who had an unplanned pregnancy and kept the baby usually end up proclaiming it was the best thing that happened to their life, even when it leads to all kinds of adversity.
However, it was only recently when I became aware that my conjectures were actually backed up by psychology research. I was reading David McRaney’s excellent new book “You are now less dumb” which devoted an entire chapter (Chapter 8: The Illusion of External Agency) to “Subjective Optimization”, as this phenomenon is called.
The thing that sets McRaney’s books apart from other pop psychology books is the extensive list of references at the end which point to the actual research papers his knowledge is derived from. I looked them up and some were even more fascinating than the book itself. If you’re interested, here are the two most significant:
- Lottery winners and accident victims: Is happiness relative? (unfortunately couldn’t find this for free anywhere) - Brickman, Philip; Coates, Dan; Janoff-Bulman, Ronnie 
- The Illusion of External Agency - Gilbert, Daniel; Brown, Ryan; Pinel, Elizabeth; Wilson, Timothy 
In a nutshell, our psychological immune system makes us see the bright side of all the choices we’re stuck with and over time they start seeming optimal. Basically, when life gives you lemons, the lemonade doesn’t magically appear due to some sort of divine intervention: Your brain makes it.
This only applies to choices we can’t change. For those that are not inherently permanent, we are often eternally tormented, worrying whether we made the right choice. This might explain why people are on average very happy about their kids, but not as much about their marriages: They can’t change the former, but can change the latter. Same applies to jobs and a lot of other things.
You might not like the above. Nobody likes the idea of self-delusion, yet our brains constantly delude us, to keep us sane. If you’re about to argue against it in the comments, you might want to read up on Confirmation Bias first: Yet another example of self-delusion, where we reject and even avoid information that doesn’t match our worldview, to avoid cognitive dissonance. And I would certainly recommend both of David McRaney’s books, which present tons of psychology research on self-delusion in short, simple, funny and easy to digest chapters (and no, I’m not being paid to write this, I just really love them).