Article
How Preeti was born
A beginner’s guide to contextualizing behavioral measures
By Aditya Laumas, Nicholas Owsley, & Pooja Haldea
Are behavioral biases universal? This is a question that Busara has long set out to answer. In 2016, we ran our first large cross-cultural study towards this question and quickly learned that contextualization is key to measuring biases. You can’t ask Kenyans who earn less 100 USD per month about purchasing an Economist magazine subscription for 135 USD — you need to make measures contextually appropriate to run a sensible test.
But how? In this article, we walk you through our contextualization of ‘The Linda Problem’ to show how we contextualized standard ‘Western’ measures for a study with the Centre for Social and Behaviour Change (CSBC) to measure biases in India in 2019.
The original measure, ‘the Linda Problem’ (see above) relied on something behavioral scientists call ‘ignoring base rates’: people ignore important probabilities in making assessments. In this measure, after presenting a profile of a woman called “Linda”, we asked which of the 1–8 possibilities are more likely. The key options were 6 (Linda is a bank teller) and 8 (Linda is a bank teller and is active in the feminist movement); given that ALL feminist bank tellers are still bank tellers, it has to be more likely that Linda is a bank teller than that she is a bank teller AND a feminist. According to this rule, 6 is always more likely than 8. But, people ignore probability; they feel that the moniker ‘feminist’ better ‘represents’ our liberal and agentic Linda and, as such, rank 8 as more likely than 6. Most people make this error, and we consider it a bias — what is called, the ‘Representativeness Heuristic’.
With the shackles of domesticity removed, Preeti was ready to become the measure she was made to be.
When it came to time to use this measure to test bias in India we ran into a couple of problems.
First of all, Linda needs to “represent” a type of person that people recognize for this to work (hence ‘Representativeness Heuristic’). When participants read Linda’s profile they should think she is a ‘feminist’ and ‘not a bank teller’. Second, the list of options should include plausible possibilities: is there a ‘League of Women Voters’ in Haryana? Each item on this list needed to be plausible for our woman in question. The fairly obvious first question then was “Does this measure contain cultural references that might not translate?”
Another question was, “is this measure computationally difficult and likely to produce errors?” Ranking is not uncomplicated — in our 2016 study, roughly 25% of respondents ranked the wrong way around (i.e. gave “8” as the highest rank, not “1”). Anecdotes show that people also kept getting frustrated at remembering all the occupations and what they had ranked — 8 seemed like a lot of ranks.
A final question we asked was “is there some other bias that could be getting in the way here?” For example, bias towards the extreme of a list or menu — the first and last options on the list could receive higher ranks simply because of where they are.
According to these questions, Linda in her original form wasn’t going to cut it in India. We had to determine what core attributes ‘made up’ the measure and allowed us to truly test for the Representativeness Heuristic, so we could reproduce them in a culturally appropriate way. Here they were:
- A profile of a recognizable ‘type’ of person in the Indian context.
- A list of plausible occupations that could associate with that person.
- Variation in the likelihood of different occupations- some options should seem likely, and some not.
- Two occupations that could be paired — one that is likely (ye olde feminist), one that is unlikely (ye olde bank teller), that are unusual but not impossible together.
Some other features that we thought useful to keep consistent are that the woman was also a similar age, relatively liberal and ‘atypical’ but still recognizable, but these are not essential per se.
With all the ingredients in hand we went on to make our measure.
The first step was engaging Indian behavioral experts with understanding of both behavioral science and the local context. CSBC, our partners, had us covered on that front. Then, between the Kenya and India based behavioral scientists, we came up with the below measure — 5 and 7 are the key options:
Contextualized Version 1
Brimming with confidence in our bright and shiny new measure, we took it for prototyping with people closer to the respondents — our field officers. They hail from the same area, are socio-economically similar to our target population, and are conversant with research. We showed them the description of Preeti and the list of possibilities with a focus on ticking the criteria laid out above. What did they think of our new liberal and independent protagonist?
“All [FOs] think this woman is a housewife”. Ouch.
We had created a housewife. Not the more conservative Haryana-village arranged-marriage housewife, but a housewife nonetheless. Unsurprisingly, some of the team also had a problem with Preeti being defined predominantly in her role as a wife.
The list of possibilities was also not perfect. While most of our FOs knew beauticians (our version of the ‘bank teller’ in the Linda Problem) they thought it was nearly impossible for a beautician to be a community activist. For Linda, being a feminist edified her bankteller-ism, whereas for Preeti, being an activist and a beautician seemed absurd. As our colleague said, you need the combination to be unlikely, not impossible: “shopkeeper-community activist seems like a feasible person, but salsa dancing lumberjack is less plausible”.
Back to the drawing board.
We adapted the profile and the list once again, and then took it back to the field team. This time it passed the test. Preeti was no longer a housewife. She was now a politician, NGO worker, political activist, or maybe even a social worker. Our implausible activist-beautician was replaced by a believable garment worker-NGO volunteer.
Contextualized Version 2
With the shackles of domesticity removed, Preeti was ready to become the measure she was made to be.
Before roll out we did one more prototyping session, this time with actual respondents. We asked a group of 7 respondents from our participant pool what they thought of Preeti and her possibilities. Sure enough, our respondents overwhelmingly thought she was a teacher, and some thought she was a form of community leader. When we probed on the possibilities it became clear that the new Preeti was unlikely to be a garment retailer and very likely to work for an NGO (check). However, she could be a garment worker in a certain strange time in her life, and simultaneously volunteer for an NGO (check). Most respondents knew of someone like Preeti (check). We had our measure. If there was a bias, this would uncover it.
And did it? Yes.
58% of the low income Indian sample ranked Preeti being a Garment worker AND a volunteer at an NGO higher (more likely) than her being just a garment worker, applying the Representativeness Heuristic and proving the bias (Figure 1 above — the higher the proportion, the more bias). 57% of the elite Indian student population show the same pattern. These results are remarkably similar to the result for the US student population from the 2016 study — 56% — which did not have clear measurement error. The patterns of measurement error we observed in Kenya in 2016, the reason for embarking on contextualization in the first place, were also not to be found in our new Indian data.
Are biases universal? Thanks to contextualization and the consequent absence of clear measurement error, we can say with more confidence that this bias, the Representativeness Heuristic, just might be. Since we applied this process to 6 other measures in this study and in all cases significantly reduced or eliminated observable measurement error, we will have more to say about this in the near future when we present our full results. For now, we can also say with confidence that most FOs on our team love going to the movies, and that none want to buy a ‘money holder’, which (other than being a terrible example for a measure) is actually not a wallet, but a foreign exchange agent.
