����vlog����

(The following transcript has been lightly edited.)

Thoko Moyo: Hello and welcome to the Harvard Kennedy School Policy Cast. I’m your host Thoko Moyo. Today I’m very pleased to welcome Iris Bohnet, who is the co-director of the Women and Public Policy Program here at the Kennedy School. She’s also our academic dean.

Iris is a behavioral economist and a leading researcher into gender bias and she’s been giving out some tough advice recently to governments and big corporations: Stop wasting your money on traditional diversity training programs, because they don’t work. It turns out that no matter how enlightened we think we are, the research shows that regardless of race and gender, we are all pretty much affected by unconscious bias. However, there’s some good news too. Using behavioral design, organizations can create better processes that help prevent us imperfect humans from making biased decisions.

Welcome to Policy Cast.

In your work, you talk a lot about unconscious bias being one of the drivers or the reasons that we don’t have as many women in leadership positions as we’d want. Let’s just start at the very beginning. What is unconscious bias and how is it affecting women?

Iris Bohnet: Well, first of all, thanks very much for having me. It’s a great pleasure to talk a bit more about our work on how we can debias how we live, how we learn, and how we work. So unconscious bias is very much part of the human mindsets and that’s in many ways, good and bad news in that the good news is that this is about all of us and the bad news is that it is about all of us. But I am saying that in a sense that unconscious bias is shared by people. This is not about pointing fingers at particular people.

Thoko Moyo: It’s not exclusive to one group.

Iris Bohnet: No, not at all. Not at all. And it has to do with these images that we have of certain individuals. Do they fit the category that I have in mind when I want to hire a conductor or when I want to hire an assistant or when I want to hire a physician? So seeing very much is believing. And if we don’t see people kind of fitting into certain categories, we don’t imagine that this is right for them.

Thoko Moyo: So let’s make it more concrete. So when we talk about women in the workplace, give me an example of what the research or the evidence shows in terms of unconscious bias and how it affects women.

Iris Bohnet: Yeah. So in our universities, we now use a very simple exercise to help our students understand what unconscious bias is in a matter of minutes really. We do a case study with them about Heidi Roizen. Heidi Roizen is a venture capitalist. She’s a real person …

Thoko Moyo: Real person.

Iris Bohnet: … a venture capitalist in Silicon Valley. And this is a case study that most of our listeners would have seen before that describes what she did, how she built her enterprises, how she networked in the valley, et cetera. And then a few colleagues of ours took this case study, which was written by Kathleen McGinn of the Harvard Business School originally and replaced Heidi’s name with Howard. And now we give half of our students to case study with the protagonist being called Heidi and the other half with the protagonist being called Howard. And students don’t just prepare the case, but also evaluate how well Heidi and Howard did. And students agree that both Heidi and Howard did a great job in fact-

Thoko Moyo: Because they are the same person.

Iris Bohnet: They are the same person. It’s exactly right. There’s nothing different really other than their names. But men and women and that’s actually important to know, men and women agree that Heidi is just not quite as likable as Howard. We are less likely to want to hire her or want to work with her.

Thoko Moyo: Well and this is based on the same information?

Iris Bohnet: That’s based on the exact same information. And the reason is that Heidi doesn’t quite conform with our stereotypes of what the typical venture capitalist looks like but also with our stereotypes of what a good woman does.

Thoko Moyo: That is so interesting. And this was both men and women because you’d expect that maybe men would say that. But women held the same view about what a woman, what a venture capitalists could be or what a good woman is?

Iris Bohnet: Yes, exactly. So there are nuances. So there are some gender differences in terms of the sex or the gender of the observer, but they’re much, much smaller than most people think. So most people in fact would think that men are more likely to associate things with men and women are more lenient in judging other women, but we don’t actually find that.

Thoko Moyo: So is it possible … I mean that’s quite something. So is it possible then to change a mindset? Because I’d imagine then you’d want to try and work on changing people’s idea of what a venture capitalist is or what a good woman is. I mean, can you do that?

Iris Bohnet: Sadly, it’s really hard. So the first part of my research focused on research, not my own, on just evaluating the research out there on whether in fact diversity training is possible, whether we can train out bias, so to speak, from our mindsets. And then unfortunately at the time when I wrote the book, my book what works in 2016, I had not found one single study evaluating diversity training that works. Now this could be two explanations. One is it really doesn’t work and I’ll give you some reasons why I think it might actually not work so well. But secondly, an important message here also is that we don’t measure nearly enough. Most organizations just to do diversity training, so to speak, blindly without ever evaluating their impact.

Thoko Moyo: Is it possible that the diversity training that you looked at was just bad? I mean, better diversity training might have results. I mean, is that …

Iris Bohnet: That’s possible. I mean that is totally possible that somebody has come up with diversity training that wasn’t evaluated, that somebody has discovered a secret sauce. But, you know I’m a behavioral economist and in behavioral science we have been trying to, so to speak, fix mindsets, not just in terms of gender bias, but lots of other biases, cognitive biases. So we’ve been trying to fix mindsets for a very long time with relatively little success. And the problem is that we don’t typically have our better selves, our super ego, so to speak, sit in our shoulders and whisper into our ears whenever we are about to fall into one of those traps.

Thoko Moyo: I find that quite scary actually. And I know we’re going to get to good news piece, but you always imagine that there’s a possibility to change sort of someone’s mindset and that you do enough of the right things that you could do it. But if you’re starting from a premise that actually you can’t, so where do we go from there?

Iris Bohnet: So let me maybe offer some nuances. I’m not saying that we cannot at least open some hearts and minds by making people aware of some of these issues. And I think that’s in many ways kind of the good news. I think the more complicated news is that just awareness won’t solve the problem. And think of a totally different example from a different domain. As in healthy eating or exercising, just being aware that we shouldn’t eat more than 2,000 calories or 1,500 calories depending on people’s size a day doesn’t necessarily mean that we won’t have ice cream tonight. And that’s a bit the same problem that this intention, action gap is real for human beings. And that is not always easy for us to live up to our virtuous intentions. So in that sense, there’s nothing very different about gender bias compared to many other biases. It’s just hard then to move beyond awareness. So we have to give people the tools to make those virtuous intentions a reality.

Thoko Moyo: That’s quite an idea. I mean, the estimates say I think about $8 billion is spent in the US on diversity training. So that’s quite a mind shift. What sort of things are you advising when you talk to corporates, NGOs, et cetera, about the money that they’re spending on diversity training? What are the things that do work?

Iris Bohnet: Yes, it’s definitely very common. I mean most organizations do have some sort of diversity training, partly because they’re mandated to do so by law. And certainly there’s huge right in diversity training. From literally checking the box as having a short half hour or one hour online training tool, to longer types of interventions. But yes, so the big question is, how do we move beyond diversity training? What more can we do? And there I would very strongly argue that we have to debias our systems instead of trying to debias our mindsets.

Thoko Moyo: Okay. So let’s talk about that some more. What does that look like?

Iris Bohnet: So maybe it’s easiest if you just go … oh no. In fact, let me start with a concrete example and then maybe we can talk a bit about the workplace, but I want to take a bit of a detour and just give an example of how simple these interventions can be. So one of the very salient examples comes from orchestras. In the 70s, many of our bigger symphony orchestras in this country have introduced curtains and have had musicians audition behind a curtain. So these blind auditions have in fact increased the fraction of female musicians in our major orchestras dramatically. In the 70s, there were about 5 percent women playing music in our orchestras, now it’s close to 40 percent. And research by Claudia Goldin and Cecilia Rouse and Claudia is a professor to Harvard econ department, have shown that the big part of that is in fact the curtain.

Iris Bohnet: And a number of organizations have adopted blind evaluation procedures. So for example, the UK government is the first big country, which is now experimenting with blind evaluations of civil servants. So that’s one behavioral design that you might want to think about as an organization. It’s not the only one, but I think makes this quite clear that one way to overcome unconscious bias is just to disable us basically-

Thoko Moyo: To able to bring that into it.

Iris Bohnet: From seeing difference. Right?

Thoko Moyo: Right. And I guess you’d also, just exploring the idea of sort of blind evaluations in a workplace situation. So you’d be looking to take out anything that identifies the gender and if you want for instance, so maybe the name you blind that and you wouldn’t require people and I think people do anyway, the people to say what gender they are. So that’s basically taking those out, reducing it.

Iris Bohnet: That’s right. At a minimum you want to take off the name. But if you want to be more advanced, you might want to be very carefully go through the CV and also take off sororities and other types of identifiers.

Thoko Moyo: Identifiers.

Iris Bohnet: If you want to push this a bit further, there are now interesting startups, exciting startups, including from a graduate of the Kennedy school in fact, Kate Glazebrook, her company is called Applied. It’s headquartered in the UK and they blind evaluators to the demographic characteristics or job applicants. They don’t just take off the name from the CV, but they in fact they’ve removed the CV completely. Arguing that empirically speaking and they’re right. Empirically speaking, the best predictor of future performance or success in organization is not the CV but a work sample test. So imagine you could test or imagine you want to hire a coder, right? So that’s an easy job to imagine that you would actually ask that person to write some code and that’s probably more predictive of that person’s future success than their CV or anything else they have done in the past.

Thoko Moyo: That’s interesting. But I guess the CV is initially the first filter that you use to get the short list. And I don’t know if you have any more information about how they would even get to the shortlist because you need to get the people in the room to test them. How do you decide who they are?

Iris Bohnet: Yes. So they actually have everyone who applies do this online test. So they don’t filter and an interesting result that they found. So they did exactly this procedure with a tech company and the tech company found some interesting things that nobody really expected. So when they used the CVs beforehand, they thought that somebody to work in this tech firm, had to have a degree in computer science or in engineering to even make the shortlist. But without them being able to filter down to this shortlist, they had all kinds of different peoples with all kinds of different backgrounds kind of do the test. And they ended up hiring more neuroscientists, interestingly enough, because they did brilliantly on the tests, but they would never have made it through the filter beforehand.

Thoko Moyo: That is so interesting. So what about the idea as well that you’ve talked about in terms of attracting more women to apply for position based on the language that you use when you advertising a position? Can you just talk about that a little bit as well.

Iris Bohnet: Yeah, you’re totally right. I mean, all of this of course starts … I mean we could go or we should go probably through the whole life, right, of talent management. Then it starts with our advertising of jobs and it turns out that language can also be biased or gendered in certain ways. So there are words, for example, warm or caring or supportive that in our minds are associated with women. And then there are other words such as assertive and sadly still leadership, that are more likely to be associated with men. And there are now algorithms which can predict how much you either increase or decrease the likelihood that men or women will apply depending on the language that you use.

Iris Bohnet: And they will in fact you know they … you just send them your text, your job ad or also your performance evaluation or your letter of recommendation really anything that you write and they will analyze the text and will give it back to you kind of highlighted with the words that fall into these different gender categories, but also with some recommendations. And what you can do is either do away with these words completely, because you might say, “I don’t actually need supportive. It’s not quite as important to the job description.” That might be one conclusion that you draw or you might say, “Well supportive is really important as I’m looking for a teacher for example, as a caregiver. So I really care about warm and supportive people, but I’m going to counterbalance these gender stereotypically female names with gender stereotypical male names.”

Thoko Moyo: So let me just get my head around that because you’re saying that you would use language that is likely to attract more female applicants because you want them in the pipeline for the recruitment. So just starting off, you want to have women apply for the jobs and then you can do the rest. But at the same time, I guess what I’m wondering is if you do recommend to have words that are associated more with women, are you not in some way entrenching the stereotype of what women are and creating a Heidi problem where people imagine women are in a certain way. Does that make sense?

Iris Bohnet: So first of all, I want to clarify that this is not per se about women, but this is about kind of gender adjectives that in our minds are associated with men or women respectively. And if we use those names, this is just a descriptive statement that I’m making, not a prescriptive statement. Just a descriptive statement that you will be more likely to attract women if you use gender stereotypical female names, or gender stereotypical male names, you will attract more men. Now my argument typically is to broaden the talent pool and have a relatively balanced set of gender stereotypically coded female names or male names.

Iris Bohnet: But I want to clarify, this is not about women per se. It’s typically about the counter stereotypical individual that you want to attract. So for example, let’s assume you are looking for more male teachers. So it’s probably pretty natural for people then to go for caring, warm, supportive

Thoko Moyo: Which might put the men off …

Iris Bohnet: Exactly.

Thoko Moyo: Ah, got it.

Iris Bohnet: But then we know fewer men will apply not just because this is a gender job, which of course we know 85 percent of teachers of primary school teachers are women. So it’s definitely a right observation. But if you on top of that add more gendered language that we associate with women then even fewer men will apply.

Thoko Moyo: So you’re using it as a tool to …

Iris Bohnet: Yes. Exactly.

Thoko Moyo: Okay. Got it. Got it, got it. Okay. So let’s keep going. So you’ve done the recruitment, hopefully you’ve got a nice mix, a nice balance of people that have applied. The next step is the interview. How do you do that in a way that’s smart?

Iris Bohnet: It is hard. Really really hard. Honestly, ideally you just would forego the interview completely. So you need to …

Thoko Moyo: Throw out the CV, throw out the interview …

Iris Bohnet: That’s a hard sell. I know it’s a hard sell. Really hard sell. But you know the interview really is the home of bias because now I am kind of seeing you. I see your demographic characteristics. I see the colors that you wear and you might happen to wear my favorite color or not. And we know all of those irrelevant things, which should be irrelevant, in fact do matter in our evaluations. So my honestly, my best advice is to forego the interview completely and move to work sample tests. But that’s a hard sell. Most people will still want to do an interview.

Iris Bohnet: So then the question is how do you do the interview? And there are a number of recommendations that are pretty well substantiated by the research. The first one is many organizations do panel interviews where they have three individuals, let’s say interview just one person thinking that this would be really great. We have three different people …

Thoko Moyo: Yeah. They will be looking for different things.

Iris Bohnet: Exactly.

Thoko Moyo: And they’ll see different things.

Iris Bohnet: But of course they actually won’t because let’s assume you and I and a third person will be the interviewers. I of course hear the kinds of questions that you ask and also hear the candidates responses to your questions and that will already influence my own assessment. So you could be a lenient interviewer or you could be a tough interviewer. You could ask questions, which in theory I think are irrelevant. But now that I’ve heard you kind of ask the question about the person’s conscientiousness or something, I can’t just disregard that. So we often call this group think, in that the group in fact is not better than kind of an … It’s basically a sample size of one, right? Because this group came up with one assessment of the candidate. And that’s fine if you believe that this is a sample size of one that we have now a group assessment of the one person.

But my recommendation normally is to say if we have to be interviewers, for example on the panel, it’d be much better if the three people run the interview separately. And so that doesn’t mean more time for the interviewer. Every person just spends half an hour kind of talking to the job candidate. It’s more time consuming for the candidate because the candidate now has to spend one and a half hours, half hour with each interviewer, but at least these three interviewers will now come up with different assessments of the candidate. That will be my first recommendation.

Thoko Moyo: Three of sort of influence of other people’s questions and-

Iris Bohnet: Exactly. Exactly. Then my second recommendation is to do a structured interview where we determine the questions beforehand. And that’s actually now thankfully becoming much more common than about 10 or 20 years ago even. In that, people have realized that we often go into tangents when we talk to different people that we might talk about hobby with somebody or their favorite music with somebody else or their kids or their previous vacation. And then some people might share the same hobby that I have, and I just like them because of that hobby. But that’s not necessarily predictive of their future performance in my organization. So the problem of unstructured interview is not that there’s nothing useful happening, but the challenge is that there’s a lot of noise that is hard for my mind kind of to tease apart to two.

Thoko Moyo: But isn’t it human nature as well, and you’re a behavioral scientist, that you want to work with … Someone once gave me the advice that people work well or you get promoted because people like you, not necessarily based on your skills. Like there’s a likability factor. And I think you also … yeah. So maybe let’s explore that. I mean, is that not important? Again, being aware that we are flawed human beings with mindsets that are unlikely to be changed. I mean, part of that interaction so when you will connecting with someone in a conversation, it’s giving you a sense of how will we vibe together when we work together? I mean it’s … what would you say about that?

Iris Bohnet: Absolutely very, very human, but again, home of bias. So we do know that we tend to like people who look like we do often called as affinity bias. That can be in terms of gender, can be in terms of skin color, can be in terms of nationality, sexual orientation.

Thoko Moyo: Yeah. That can be problematic.

Iris Bohnet: You know, lots of things, even accent. So that’s, I think the challenge. Now I’m completely agreeing with you it’s very human that we have this what we sometimes call in-group bias, that we feel more comfortable with people, kind of who share the same kinds of jokes and the same kinds of hobbies as I said before. But a functioning organization and even a functioning team in fact flourishes from diversity, from different backgrounds and kind of comfort and fit and ease of understanding in fact are not good predictors of future team performance.

Iris Bohnet: Now I’m not saying it’s not easier to work in a team where everyone runs into the same direction naturally when we fall prey to in group bias and then group think again where we don’t challenge each other enough. So it’s a real problem that we want to like the person that we work with. And so for people who insist that likability is really important and they spend so much time with their executive assistant or whatever it might be, that they need to have somebody who can share some of the same humor, et cetera. Then I would say A, be aware that this is very likely going to be biased. That you are very likely going to go for people who look like you and we can show that. I mean the research suggests that, we even look for people who graduated from the same college that we did.

Iris Bohnet: So this is real. But secondly, if likability is so important, at least make it explicit. So if you have let’s say five questions in your structured interview, make this question number six, so to speak, or characteristic number six. And you then give the person a rating from one to 10 and it’s literally called likability. So now at least I make what is invisible and we know is affecting many people, we make it visible. And now I can have a more explicit conversation with you, for example, who might have interviewed a person as well and you can push me a little bit and say, “Are you really willing to forego this person’s excellence in these other five questions that you’ve asked her just because you like her?”

Thoko Moyo: Have you actually ever had a real life sort of situation where they did put likability? Because yeah, it’s an interesting idea.

Iris Bohnet: So I myself haven’t. In fact, I did not. So I hired my assistant exactly the way I just described to you.

Thoko Moyo: Okay. The first assistant or the second?

Iris Bohnet: My executive assistant. My executive coordinator. And we in fact used the three step process. So we first blind ourselves demographic characteristics. We looked at Sydney but we blind ourselves demographic characteristics. So we didn’t know gender, race, any other identifiers. We also blinded ourselves to the school the person that had attended. We rated every city from a scale, from one to 10, and then we put the CV in the drawer. And then secondly, we had everyone do a work sample test. In fact, I spent some time thinking about what I look for in an executive coordinator. What are kind of important skills, tasks, competencies that a person has to have. So, for example, I designed a itinerary travel planning kind of problem where-

Thoko Moyo: Because that’s a big part of what the job would entail.

Iris Bohnet: That is a big part of the job. But we had a number of things. This wasn’t the only task, but just to give a concrete example. And then it literally said, she has to go to New York, speak at the UN, then she travels to London and has dah, dah, dah and then to Zurich, et cetera. And of course, people couldn’t know everything. They couldn’t know travel schedules, et cetera, flight schedules. But we wanted to know who asks important questions. Which presentation is she going to give? Is it going to be PowerPoint? Is there going to be an interview involved? Who is she going to meet? What does she need to know about these people? Et cetera, et cetera.

Iris Bohnet: And then we rated all these answers of all candidates from a scale, from one to 10, again, and put them in a drawer. But again, we didn’t know who gave which answer. So these people just had a cold numbers. This was number one, number two, number three. And then finally, thirdly I did interview everyone face to face. And that was of course a little funny because the, I had to tell to people that, “I don’t know who you are. I don’t know what CV you have. And I don’t know how you did on the test. But this is now … but we’re trying to do this based on research evidence out there and how to really evaluate people. And I have prepared five questions. I’m going to ask everyone the same five questions and I start with question one, and I’m then going to compare those questions.”

Iris Bohnet: I don’t know, by the way I said most everything I’m telling you now. The last thing I’m not going to tell you, I probably didn’t say it, but I don’t remember. But then I in fact asked the five questions. But then in the end I compared answers to every question, what I’m going to call horizontally. So I compared every person’s answer to question one across all candidates. And then I looked at all answers to question to two, et cetera. That’s what I call horizontally to calibrate my thinking. Because another problem is that, when I talk to you, your answer to question one might be brilliant and then I’m already sold and I don’t even pay attention to questions two, three, four and five because it’s the halo effect.

Thoko Moyo: You had me at hello.

Iris Bohnet: Right, that’s exactly right. Or you know you wear my favorite color. So you have me already as entered the room. So that’s the thing. So we try to control for the halo effect. In fact, you know, many faculty members when they grade exams, they also grade horizontally. And I did that too meaning you know, for example, I might have three different questions, essay questions-

Thoko Moyo: Essay questions right.

Iris Bohnet: And then I would read all answers to question one first again horizontally, right? And then move on to question two and then blind myself to your answers to question one. So I could give every answer to every question. Get an independent and a new chance and wouldn’t be influenced.

Thoko Moyo: So I have two reactions to this. The first is that sounds like a lot of work, I don’t know how … important work, but I don’t know how many people would want to do that. And then the other question that I had just as I think about it, particularly for the interview, for the person that you were hiring to work with you as an executive assistant was you started off when you did the blind evaluation, looked at the CVs and rank them. What you were saying is that everybody that got to see you could do the job and not … is that where you got to? So by the time you met the people in person, you’d established that they can do the job. So in meeting them face to face, remind me, where are you now testing likeability or was that an integral part of being able to do the job? I don’t know if my question makes sense.

Iris Bohnet: Yeah, I mean I don’t think anyone knows is usually the million dollar question. Whenever I teach on this, people ask me, “So what are the predictive questions? What are the right questions?” Honestly, we don’t know. We don’t know. Of course, most providers will tell you the diet they have figured out, had the secret sauce and the right questions. Most don’t evaluate. Most companies don’t do these longitudinal studies where they then look at the questions asked and how well, whatever the ranking or the rating of the question was. The answer your question was how well that actually predicts future performance.

Iris Bohnet: There are companies like Google, they do exactly that and then they change questions. So they have the structured interviews and then they realize number seven is just not correlated with future performance. So then they replace number seven. But it’s a little bit of trial and error process because not every question is equally as predictive of every job right? You might imagine that conscientiousness is probably pretty important for many jobs. But some other, communication skills, some other skills might be more relevant for some jobs and not for other jobs. So you have to experiment with that a little bit. But I have to answer your first question because that’s an important one.

Iris Bohnet: You know, it does sound complicated and it was because we did everything from hand because I really wanted to experience what this looks like, but it turns out we now have these providers. They do all of this for you. It is just and it doesn’t actually take longer. It often it requires less time to evaluate somebody because you don’t do the CV at all. You move to work samples right away and then the machine does the comparative evaluation for you. So it’s actually, I wouldn’t worry about the time-

Thoko Moyo: Okay. There is technology there.

Iris Bohnet: It’s just different technique and a different software that you need.

Thoko Moyo: And so the person you hired, are they working out?

Iris Bohnet: Yeah, they’re great. They are great.

Thoko Moyo: They’re still here?

Iris Bohnet: I guess. Yes, yes they’re great. They’re here. They’re wonderful. Yes.

Thoko Moyo: Okay, so let’s keep going in this whole sort of process of hiring. Now you’ve got the job. You’re in an … and I’ll stick with the idea of a woman. I’m thinking about women in leadership. Part of what happens once you have the job a year in, a year and a half in, there’s the evaluations. There’s promotions. You want to go up the ladder, what’s happening now?

Iris Bohnet: So I’ll answer that question just a moment, but I have to tell you one more thing about hiring that we haven’t touched upon that I actually feel very strongly about and that is that we should try to hire more in bundles and-

Thoko Moyo: In bundles?

Iris Bohnet: In bundles. Yes.

Thoko Moyo: Okay. What do you mean by that?

Iris Bohnet: In fact, let me maybe give you another, just deviate for a second and give you another illustration and then kind of come back to the hiring context. So about one and half years ago, I was in Stockholm meeting with the Nobel Prize Foundation. And they had invited me because-

Thoko Moyo: Obvious question: They give out the Nobel prize?

Iris Bohnet: Yes.

Thoko Moyo: Okay. Yeah.

Iris Bohnet: And they had realized that 97 percent of all Nobel prices in the sciences had gone to men. And so they were just trying to understand more about kind of what the process that they’re using, what they could be different. I have to reframe that. So they wanted to learn more about the procedures they were using, and whether there might be something that they could improve there. And of course, much of this is secret and should remain secret. But one thing that was written about afterwards, that I can’t talk about now and that I had also suggested without knowing anything about the procedures they’re following. So every year I get a form from the Nobel foundation inviting me to nominate somebody for the Nobel prize in economics.

Iris Bohnet: I’m an economist by training. And that form asks me to nominate one person and one person only. And I told them that will be the first thing I would change. In fact, I suggested them to ask nominators like myself to make three recommendations, not just one. And now, here’s why. So a few years ago people did research, psychologists did research on food choices. And they were trying to understand the variety of foods that people consume. And let me give you a kind of a way to think about this. So imagine you would go into a high school, let’s say, and you were trying to understand what snacks these students consume. Then you would ask half of the high school students to choose a snack for every day of the month for the next month. So today you’re making 30 choices. You have endless snacks available, but today you have to make 30 commitments, so to speak, what to have for the next month.

Iris Bohnet: And then the other half, the high school you would ask to choose a snack every morning of every day of the month. So they also choose 30 snacks, but sequentially. So the first group is kind of simultaneous choice or think of this as the bundle choice. Now I make kind of this whole kind of cluster of choices and then the second group does it sequentially. And lots of studies have been conducted kind of similar to the one that I just described to you, finding that people who choose simultaneously, who choose many snacks at the same time are much more likely to go for variety than people who choose sequentially. Right? And that often-

Thoko Moyo: It’s almost intuitive, isn’t it?

Iris Bohnet: Right? Isn’t it?

Thoko Moyo: You want to make sure you have variety over the 30 days and not eating the same thing over and over.

Iris Bohnet: Exactly. You couldn’t imagine right? Even though you have a favorite snack, you wouldn’t imagine you want to eat it 30 times in a row. I sometimes also talk about our favorite lunch restaurant, right? We go to some restaurant and we always have a Caesar salad with crab meat, but every time, and that’s my own personal example, which is also true. Every time I think I’m going to have something different today. You can’t always have the Caesar salad with crab meat and every time I almost have to Caesar salad and crab meat.

Thoko Moyo: Caesar salad and crab meat.

Iris Bohnet: But if you ask me today, I probably wouldn’t commit for the rest of my life to just have Caesar salad and crab meat. So anyway, so we applied that and a few other people from Wharton as well applied that to people choices. And we’re finding the very same pattern that diversity is much more likely to emerge when you choose a bundle of people at the same time. And that has a number of reasons. I mean the psychology is kind of interesting. One is that think of this more as a portfolio now, that if you buy stocks and you buy more than one, you try to diversify, right? And now you hire five people, you’re trying to diversify, you’re thinking more about collective intelligence rather than just individual intelligence. Now you think about complementarities between the different people, et cetera.

Thoko Moyo: It forces you to do a little bit of thinking about what your decision is.

Iris Bohnet: Yes. As a team, as a group, right? And not just even it forces you, but also enables you to have that kind of thinking because otherwise if you do it sequentially, it doesn’t come naturally to you. The other advantage is based off some by own research where we’re showing that if you do this compared to assessments, if you evaluate people kind of comparatively next to each other, you’re much less likely to rely on stereotypes. And that also has kind of a psychological foundation in that big insight of behavioral decision research, which is kind of some of my background is that it’s very hard for people to make absolute judgements. So whether or not you think it’s cold in here, whether or not you like the water in front of you has something to do with the temperatures you’re used to. Something to do with the drinks-

Thoko Moyo: It’s relative.

Iris Bohnet: You’re having, and if I don’t give you a comparator, your mind is smart enough to find one. And then your mind is going to look at what the do people who kind of have made it in my organization, who will kind of are in those positions, what do they look like? And that’s kind of my reference point. And then I compare the candidate with that reference point. So by giving you various people, I help you make those calibrations with real people and we’re showing that evaluators make assessments much more on the competencies of these people rather than on stereotypes.

Thoko Moyo: And so the Nobel foundation took that recommendation-

Iris Bohnet: Yes, that was a long way right?

Thoko Moyo: Just to come back to that because, yeah.

Iris Bohnet: I almost forgot my initiating story. So yes. So now the new form that I get asks me to make three recommendations and not just one. I’m going to repeat that. So the new form that I’m getting now asks me to make three recommendations, not just one. And of course that’s not research it’s just a story. But last year was the first year we had two women win the Nobel prizes in chemistry and physics simultaneously ever. So anyway, so these are the kinds of design choices which I …

Thoko Moyo: Which are fairly simple.

Iris Bohnet: Are super simple, are basically costless but just are a bit smarter design.

Thoko Moyo: Okay, well we’re almost running out of time, but I want to finish the sort of the whole cycle. So I want to be in the work place …

Iris Bohnet: We have to talk about career advancement. Yes.

Thoko Moyo: We have to.

Iris Bohnet: Yes. I know.

Thoko Moyo: So what happens when a woman’s getting promoted? I mean, the number of times I’ve had conversations with other women about, oh my God, it’s so difficult. I’m not being recognized. What am I doing wrong? Well, what’s happening?

Iris Bohnet: Yeah. So super, super important. In fact, we of course see much more diversity at the entry level than at the top. So this whole question of career advancement is very, very relevant and it’s also where most of my research now is. And I typically think about kind of three potential biases that we should be paying attention to. I mean there’s more than three, but the three that I think are particularly prevalent. And the first one, I actually came across almost by happenstance in that I work them with a law firm and the law firm had called me in because they had already done some data analysis and had observed that they had a gender and race gap and also a US versus non US employees gap in terms of promotions of partners.

Iris Bohnet: And so we saw that. So they knew that and they wanted to fix that. So we looked at the process and we did a few things about the process, but they also kept telling me about something that they called the problem of the thin file. And what they meant with the thin file was that some people literally when they came up for evaluation for promotion to partner in year eight, they had little material kind of to submit, I mean literally a thin file. And we could show that the thin file problem really started in year one when some attorneys, some first year associates were given more opportunities to perform in this company. And we could show that it’s not just that these partners who are primarily of course men, primarily white, primarily Americans, that they replicate those demographic characteristics. But they even show first year associates who had graduated from the same law school as they did. So a lot of ingroup bias going on because partners at that point choose people they want to work with and then people they mentor, nurture, et cetera.

Thoko Moyo: They think it’s a sure thing if I get this person, I know what the result will be because it’s a shortcut …

Iris Bohnet: And it’s unconscious, again, you know …

Thoko Moyo: Yeah. It’s unconscious.

Iris Bohnet: Unconscious bias. This is not conscious as in purposely trying to exclude certain people, but Emilio Kostia of MIT in fact of the Sloan School called this performance support bias, in that some people aren’t given the support to perform. So that’s the first thing every employer should pay attention to. That of course promotion doesn’t start the night before to promotion. It starts much earlier and when some of us are included into cool deals, some of us are given more feedback, are given more stretch assignments, can go to the leadership training and others can’t.

The law firm in fact what they are doing is they have centralized work allocation now that a centralized partner in the little office, they now allocate his first year associates to partners. And then they watch also over time whether some people are more likely to be paired with the star attorneys and others assuming they don’t get an opportunity. So that’s the first our performance support bias …

Thoko Moyo: And I can almost say another nugget from there is that you could advise women to also ask for more opportunities to sort of demand the sort of support more opportunities for stretch assignments more, I mean that is something that you could also demand I guess. I mean that could be part of it …

Iris Bohnet: Yes. I’m giving you a tentative yes here in a sense that, yes and certainly Sheryl Sandberg’s Lean In is all about kind of women can just being slightly more courageous, a bit more risk-taking, a bit more demanding of those opportunities. Now a research by Hannah Riley Bowles here at the Kennedy school, my co-director of the Women and Public Policy Program shows that unfortunately there can be backlash for such counter steer typical behaviors. So what she means of that is that, you’re now running into the Heidi problem …

Thoko Moyo: The Heidi problem.

Iris Bohnet: Where you don’t conform to my image, to my stereotype of what a good woman does, of how a good woman behaves. Right? She is nice and agreeable.

Thoko Moyo: And she’s demanding things.

Iris Bohnet: So it’s tricky. So I don’t want to leave us thinking it’s not possible at all, but I think women have to engage in counter stereotypical actions kind of with their eyes wide open, wide open. But maybe, let me give you kind of two more thoughts quickly. I know we’re close to running out of time. But I do want to leave you with two more quick thoughts. The first one is actually ongoing research right now where we’re looking at performance appraisals. Prevalent in most organizations, surveys suggest that about 95% of all employers handle them in some sort and in practice that many, many employers use is to ask their employees to self evaluate. And then share their self evaluations of their managers before managers make up their minds. Now turns out that …

Thoko Moyo: I’ve done that before.

Iris Bohnet: It doesn’t require rocket science to imagine of course somebody’s self evaluation now is going to impact the manager’s assessment. And if people differ in their willingness to shine the light on themselves or even in the cultural appropriateness of shine the light on yourself …

Thoko Moyo: Yeah, boasting or what’s bad. Yeah.

Iris Bohnet: Some of us and think of the Heidi problem again, some of us will be less likely to rate ourselves very highly because we fear backlash or because it’s just not culture appropriate. So we found both cross cultural differences but also gender differences. And now we lucked out in that we found the company that had exactly this practice but then in one year wasn’t able to share self evaluations of managers before managers made up their mind. So we now can show and really measure what difference does it make. And it turns out that of course the difference is most pronounced for newcomers. For people who did not have a history in this company where the managers couldn’t remember what we have done in previous years, but in fact it just new. And we’re seeing quite different evaluations. Wendy’s managers aren’t anchored on people’s self evaluation.

So what we found was that what you might have expected that men tend to evaluate themselves more highly than women. We also found that whites tend to evaluate themselves more highly than people of color. And then when we don’t share to self evaluations of managers for these newcomers in that given year, what we see is either the gap is closed or for gender even reversed. So sharing self ideations is just not a smart practice. I don’t know of any evidence suggesting that this is actually a good thing. And so here my recommendation would to be, in order to overcome evaluation bias, not to share self evaluations.

Thoko Moyo: But performance evaluations in general are okay?

Iris Bohnet: That’s a very good question there. They’re absolutely controversial. I don’t think I have enough research to say whether they’re overall good or bad. My fear, I’m just going to be very transparent of why I’m reluctant here. My fear is that companies which have done away with them, now might rely even more on their instinct, on their gut in evaluating people. And we know that’s the whole bias. So generally I would always argue to rely on as measurable as objective criteria for evaluations as possible.

Thoko Moyo: We’ve got 30 seconds left. What would you want to sort of end this on? Do you want summarize the sort of big ideas or is there a particular line of thought that you might just want to leave our listeners thinking about?

Iris Bohnet: Yeah. So maybe my biggest piece of advice in many ways is kind of twofold. First of all, measure, measure, measure, measure. Most organizations just don’t measure. And what we know that what doesn’t get measured, doesn’t count and also what doesn’t get measured cannot be fixed. So measurement is really important. Then my second piece of advice is move beyond trying to fix people and fix your systems. You can debias your systems, whether that’s in hiring or promotion in performance appraisals or even in team meetings. Try to think hard about the procedures and practices that you use to create an equal playing field for everyone.

Thoko Moyo: So you’re going to have to come back again because there’s so much there.

Iris Bohnet: Thanks so much for having me.

Thoko Moyo: Thank you so so much. This was wonderful.