Big Board Conversation #1 - Probability
For one week Charlie Mylie and Jori Sackin had The Big Board at Missouri Bank asking people to make predictions. They greeted the customers as they entered, prompted them to make predictions about the future and then walked them through plotting it on the board. This is the third Big Board project and this particular iteration was inspired by the book "Superforcasters" by Phillip E. Tetlock and Dan Gardner. If you like prediction you might also check out The Good Judgement Project, an online community that predicts real world events. The conversation that follows is an amalgamated version of the many conversations that the project produced. It is the first in a series of 5 that will be published on Ten Millimeters.
D: Hi. (Looks over at the board) What’s going on here?
J: We're taking people's future predictions. Do you have anything you'd like to predict?
D: I don't know. How does this work?
J: Well, this is The Big Board and all of the white circles you see are the predictions people have made so far. The X-axis is time. Each of these blue lines represent one month, so the furthest to the left is right now and the furtherest to the right is 2 years. The Y-axis is probability so the more probable you imagine your prediction to be, the higher it goes. The predictions you see plotted on the board have been answered many times and they're placed by taking the average of all the answers given. So for instance we have a prediction that Donald Trump is going to be impeached in a year. It's been answered 20 times and the average of all 20 answers is 30%. What do you think? Does that sound right to you?
D: I sure hope he's impeached.
J: Yes...but what do you think the probability is?
D: I'd like to think he's going to be out of office soon.
J: Sure, but do you think it's likely? You might consider that there's a Republican controlled congress.
D: (Laugh) I'm trying to stay positive.
J: Yes, but if you had to say a number what would it be?
D: I mean, it's anyone's guess. 50/50. No one knows the future.
J: Sure. The future is unknown but not all possible futures are 50/50. For instance, I could predict that tonight I'm going to go home and feed my cat. I'm going to predict a 99% probability that it will happen. I say 99% and not 100% because he could die...so there's a small chance that I won't be feeding my cat tonight. He hasn't been sick lately and he’s an indoor cat, so I'm fairly certain it will happen. Sure, that future is unknown, but it is much more likely than the future where my cat learns how to speak and becomes fluent in three languages by the time I get home.
D: I don't think you should think about your cat dying. What if you come home tonight and your cat is dead? You would feel awful.
J: Why would I feel awful?
D: Because it just seems like you'd be a little responsible.
J: So are you saying that the thought of my cat dying influences the likelihood that he will actually die?
D: I don't know. Maybe. It seems like such a depressing thought to dwell on. I'd rather stay positive than focus on a bunch of possible futures where things don't work out. It seems like that would just lead to depression.
J: But look....ok....there's two different directions I could go with this. I don't want to lose the thread of what you are saying here, of thoughts influencing outcomes, but lets leave that aside for now and focus on the anxiety of the many possible futures not working out, which you are saying would lead to depression. I mean, look at me. Here I am right now thinking about the possibility of my cat dying. I love my cat. He's one of the most important beings in my life. I can consider a future where he dies and I'm not dwelling on it. I'm not filled with anxiety. I just consider it as a possibility and figure it into my calculations.
D: Yes, but you can imagine a person who can't do that, who wallows in all of those negative future scenarios.
J: Sure, it's easy to imagine that person. I can imagine a person. You can imagine a person. I don't know how useful that kind of information is. I'm just going off of how I see it operating in my own life, that once I get over the mild anxiety that I don't know what's going to happen next and that there are multiple outcomes that are possible, it seems like the next logical step is to start figuring out which one of those outcomes is more likely, and if I'm going to do that with some accuracy, I'm going to have to confront some things that I'm not used to thinking about.
D: Like death.
J: Sure. Like death.
D: But if I think about my own....I mean...I don't have a cat. I have a dog. Sandy. If I think about Sandy, just doing the calculation makes me hurt a little bit. I feel like I'm manifesting her death by even considering it, like my thoughts are somehow pushing the unlikely event to being more likely.
J: How old is Sandy?
D: She's 14.
J: So let's look this up. The average life of a dog is......10 to 13 years. What kind of dog is Sandy?
D: She's a Chow.
J: Great. Chows tend to live a little longer. 9 to 15 years.
D: Oh man. This is exactly why I didn't want to do this.
J: But let's try to think about this another way. If we were going to predict the death of, well, let's not deal with Sandy right now. Let's just say we are predicting the death of another chow you've never met before. It's an abstract chow, so nothing you can think about this dog will effect the likelihood of its death. Let's start with the average lifespan of all chows, which the internet has told us is...well...let's split the difference and say it's 12.
J: So if the average lifespan is 12, every year that goes beyond 12 the likelihood increases, especially once you get outside the normal range, 15 or so, and then it would go up more dramatically.
D: Sure. That seems obvious.
J: Yes, but this also seems like really useful information for prediction, because once the dog gets to a certain age, it might make the time you spend with her seem more precious. You may end up, after doing the calculation, saddened by the passing of your dog, but what also may come is the realization that you only have a limited time with her and that may prompt you to take that more seriously, as opposed to being surprised by her sudden death.
D: Wait a minute. I thought we weren't talking about my dog.
J: OK, but you see what I'm saying? If I had to calculate the probability that this abstract dog we're talking about will die in the next 2 years, I would say that it's fairly likely. Probably 75%. That doesn't mean I'm hoping for it to happen. It just means I can see the numbers aren't working in her favor.
D: Yes, but it does feel like if you make a prediction for something in the future, that it’s a wish to see that thing come true. For instance, I could predict that I have $5 in my bank account in 6 months, and then because of that wish, I could start spending a bunch of money. So in the end, I sabotage my finances in order for my prediction to be correct. Like a self-fulfilling prophecy.
J: But do you really think you would do something like that?
D: Well, that's not a good example, but you know what I mean about self-fulfilling prophecy right? If you have a job interview and you're replaying the negative outcomes over and over, then when you go in for the interview you may be less confident, not your best self and so you'll end up not getting the job, as opposed to staying optimistic and believing that you'll get it, and so you present your happiest most positive self, and that changes the outcome, right?
J: Yes. I think what you're describing is a real thing, but again, I don't think its the same thing as what we were talking about with the dog. In the job interview example your thought processes and disposition are in relationship to another human beings who can sense all of those changes and subtleties in your being, and that has some determination on the outcome, but that isn't the case with your dog.
D: An abstract dog.
J: Right. I guess I'm just trying to get a sense of where you draw the line at how much of an effect you think thought has on outcomes. For instance, what if we predicted this cup of water is going to be knocked off the table in the next five minutes. Do you think your prediction will effect the probability that the cup will be knocked to the ground? I mean, if we both made an agreement that we weren't going to knock the cup over ourselves or tell anyone about our arrangement, do you think making the prediction would change the odds of the cup being knocked over?
D: Possibly. I think about sports teams playing in front of a crowd that's doing nothing but desiring them to win. It’s the collective will of 75,000 people cheering and wanting and hoping this outcome will occur, and isn't it true that teams more often than not win when they have home field advantage?
J: Yes, but...ok...I don't know about home field advantage. Give me a second, and let me look it up. (Sits down on couch and types on computer) Ok.....Here's a Freakonomics article from 2011. I mean, it's not conclusive, but it will give us somewhere to start. It says that home field advantage does exist, but not because of the crowd or travel or familiarity with the stadium. It's because of the refs. Apparently, they are unconsciously biased.
D: By the crowd though. See. It does have to do with the collective will. If a crowd of 75,000 people wasn't there cheering for this one thing to happen then the refs wouldn't be biased. They bend, ever so slightly, to the will of the desire that surrounds them.
J: Yes, but again, I think there's a difference between unconscious bias effecting people's decision making and the glass of water being more likely to be knocked over. Imagine we set the glass of water on a table in front of that same crowd of 75,000 people and...let's say these people want nothing more than to see this glass of water knocked over.
D: This is a weird example.
J: Yeah I know, but let's just go with it for a second. Do you think the collective will of that many people will make it more likely for that glass of water to fall?
D: I do think it will make it more likely. I don't think people's thoughts are going to physically knock the glass off the table, but I just can't help but think that hope and desire have an effect on what will happen in the future.
J: So I can think of my determination and my desires "pushing things around" in my life, in my inner world, but that's because they're made of "meaning" and so they only interact with other "meaningful" things, such as other human beings who can understand and interpret those kinds of things. A glass of water doesn't care about my desires and it doesn't bend to them, so I shouldn't expect my prediction to have any sort of effect on it. I could say the same thing about your dog. Your dog doesn't understand that you've come to the bank today and some stranger has asked you to make a prediction and that you momentarily had a thought about its death. Sure, that this conversation could change in your behavior toward your dog, and then that could possibly alter how long your dog lives, but overall I would think that effect is weak compared to other effects such as the food you are feeding it, the genetic makeup of your dog and the amount of love and affection you give it.
D: I get what you are saying, but what it comes down to is that if I sat here and thought about the possibility of my dog dying, and then my dog died that same day, I would regret it. It would make me feel like I was responsible, whether that was true or not, and that would be something that would be hard for me to live with. So I don't want to take that risk, because I can see that future, where I live in regret and I don't like it. It may not make me complicit in my dog dying, but it would FEEL that way, and I would always think, "What if I didn't do that? Would things be different?" I don't want to live with that regret, so I don't want to make that kind of prediction.
J: Ok. So let me repeat this back to you and see if I'm understanding what you're saying. What I'm hearing is that causation is a complicated process and maybe we don't understand it as well as we think we do. How powerful are our thoughts in this complex equation? You don't know, and so rather than taking the chance, you play it safe because you don't want the responsibility of being complicit in events, such as your dog...I mean...some abstract dog's death.
D: Right. That's part of it. I don't want to be associated with it.
J: So...ok...that's understandable. But does that mean you aren't going to think about any possible negative outcomes ever? In your own life, don't you imagine future scenarios where things don't work out? Such as...I don't know...if you're thinking about moving and you picture yourself in a new city. You can think about all the great new things that city has to offer. But that strategy seems incomplete unless you consider the drawbacks, all the negative stuff, such as the traffic, longer commutes, more expensive rents, things like that.
D: No. I think you're taking it to the extreme. I do that kind of stuff.
J: But you don't see it as manifesting those things for you?
D: No. Those kinds of things don't have the same emotional weight for me. My dog is someone I love. Rent is something else. I don't think about those two things the same way.
J: Ok, so there is a class of things such as, "things that I love" that is treated differently, at least in terms of causation, than things not in that category? Essentially, you're more careful with those things, not just in action, but in thought. You love those things to such an extent that you are willing to restrict your thinking in certain ways, because it may or may not have an affect, and if you don't understand how this all works, than why risk it? Right?
J: So I'm not trying to push you here or tell you you're wrong. I'm just trying to figure out how you're thinking about this particular issue, and I guess I'm also offering up another way of thinking that I think is helpful. So let's get back to the question, "What do you think the probability is that Trump will be impeached in a year?" At first you gave me an answer that was hopeful, which to me means, "I want to say a high number because that's what I hope it will be,". Now I realize saying a high number may make you feel good, and it may play along with the idea that the world will somehow slightly bend to what you hope it will be, but let's put that aside for a second. Let's pretend, just for this moment, that your prediction has no effect on this particular outcome. I want you to think about the multiplicity of futures that extend out from this moment and tell me what is the probability that we will live through the one where Trump gets impeached in his first year of office.
D: But something like Trump does affect the people I love, so I don't think I can separate the two.
J: Yes, but couldn't you say that about almost anything? If you decide to go to the grocery store today, that is going to affect someone you love. So if that's the standard you use to not think about negative outcomes then everything becomes off limits.
D: Ok. If you need an answer I would say 50% because I just don't know very much about impeachment or how any of that works.
J: So what made you choose 50%?
D: Because it's arbitrary. It's a coin toss.
J: How many presidents have been impeached?
J: OK, that's 2 in about 240 years of U.S. history.
J: So, that's a yearly average of...let me do this on a calculator...um...that's an average of .008333. So that's what we would call, "the base rate" and as you can see it's a pretty rare event. Actually, it would be even rarer than that because both of those impeachments happened with an opposing party controlling congress.
D: Yeah, but we're talking about Trump here.
J: Right, so we'are talking about Trump. So now we can bring in all the specifics of our current situation. Trump does make that number go up because there is some antagonism between him and the Republican party...
D: ....and that dossier that just came out. All that stuff with Russia.
J: Right, so we have to weigh the evidence carefully, because how much should something like the thing you just mentioned move our number up or down?
D: I mean, if just one of those things ends up being true in that dossier I would say it would move it up significantly. Like 20%?
J: I would be more conservative and say it would move it up to 1%. I mean, the information in the dossier is unverified. We don't know whether it's true or not. The fact that you may want it to be true is likely to push you to believe that it is true, a little bit like how the crowd pushes the ref in their favor. 1% actually might be stretching it.
D: Yeah, but we're talking about Trump here.
J: Right, so that is a fairly large unknown. The man does seem reckless. He's lived an entitled life where he may not be used to other people holding him accountable for his actions. So I could imagine that he's done something or will do something that could get him in trouble, but here we are back imagining again, creating whatever person we want to fulfill our desire of impeachment and I guess I'm not sure about the quality of that information. I trust our base rate and I trust bumping it up a little because of Trump, but I don't trust our theory of Trump, of what he will or will not do, and how people may or may not respond, so I'm going to give that a lot less weight.
D: Ok. I'll say 10%.
J: Yes, but that's 1 in 10. 1 in 10 is a big jump from .008%. Imagine playing Russian roulette with a gun that holds ten bullets. Would you still think a 1 in 10 chance is remote if you had to pull the trigger?
D: So why are you asking me? If you think it's 1%, just put it at 1%.
J: I don't know if its 1%. It could be higher. Certainly as events unfold in the news that number will fluctuate.
D: So what would have to happen to get you away from that 1% number?
J: Well, the first thing I would have to see is his approval ratings significantly decline. I know they are low already, but they would need to dip much lower. Then I think you would need to see Republicans openly voicing their disgust with Trump, and finally there would have to be some event, some calamity that pushes everyone over the edge, something he does that is particularly stupid and unjustifiable. That's one path to impeachment, and seeing each one of those things happen would raise that number significantly.
D: What about all of this conflict with the intelligence community? It can't be good to piss off the people who are good at digging up people's secrets.
J Yes, so in order to get a better percentage, we would have to pay attention to whether the public feuding between Trump and the intelligence community dies down or heats up. But what keeps my percentage really low right now is the time frame we are talking about which is one year. That just doesn't seem like enough time for all of these things to happen, though who knows with Trump.
D: So where would you put the prediction?
J: Where I put it isn't as important as the conversation that we're having. We’re collecting data, but more than that, we are talking to people about prediction, how it works and how to get better at it. The valuable part of this conversation is not the final number that we decide on, but the way of thinking that gets us there.
D: Speaking of "getting there" I actually gotta go. I’m late for work. But I guess what this whole thing makes me think about is that I really don't know what is involved with the process of impeachment. I vaguely remember when Clinton was impeached. But I guess if I really wanted to give you a good answer I would have to do some research.
J: Yes! That is exactly right! I know I might be acting like I have all the answers, but I don't really know that much about impeachment either. We are working with just a few facts and a whole lot of assumptions, but even with that limited set of information, we have some tools to do a better job, such as starting with a base rate, and carefully pushing our number up or down.
J: Hey, it was great talking with you.
J: So before you go, do you want to give me a number on where you think I should put this "Trump Impeachment"?
D: Put it at 20%. I'm optimistic.