Bias In Social Science
José L. Duarte blogs about some of the socio-political bias he sees in social science:
An example:
4. We've got people measuring a purported fundamental personality trait of Openness to Experience by asking participants "I see myself as someone who..."... is ingenious, a deep thinker.
... values artistic, esthetic experiences.
... is inventive.
... is sophisticated in art, music, and literature.
... likes to reflect, play with ideas
You've got to be kidding. These items are obviously grounded in - and biased in favor of - academia. This core personality trait of "openness" is measuring intellectualism and urban sophistication. These items are invalid on their face, and should not have lasted this long.
How are people in rural communities going to show up on this scale? How about people in developing countries? How would they express their openness to experience? Where do we give them a voice? They don't have opera houses, symphonies, and gallery openings with which to express their "sophistication" in art, music, and literature. They're structurally excluded and marginalized here. The items are not situated at the level of analysis necessary for a valid underlying human personality construct that is commensurable across cultures and backgrounds. We're not even speaking their language. I guarantee that many people in rural communities would be embarrassed to say that they are "ingenious" or "sophisticated". It would be unseemly to them, narcissistic and snobby. They might never use words like esthetic or inventive, not because they're stupid, but because they live in a different world and don't necessarily have use for the same terminology that contemporary intellectuals use.
This is deeply offensive. We're denying these people a voice. I grew up in a rural copper mining town with a population of 2,000. I know these people. I am one of these people. The bias of these items should be obvious to anyone, but it will be most obvious to people who never lived within 100 miles of an opera house. If openness is a real personality trait (I doubt it), commensurable across cultures, then we might ask them:
If they enjoy learning new things from their kids.
If they enjoy looking at the stars at night (FYI, seeing the stars well outside of a city is a radically different, much more powerful experience)
If they enjoy being in the woods.
If they enjoy the tranquility of being on a boat at the lake.
If they enjoy figuring out how something works and repairing it, physical things, like car engines, transmissions, or TVs.
If they enjoy the uplifting experience of church.
If they enjoy reading.
Note also that rural communities are likely to be conservative, and urban sophisticates are likely to be liberal or libertarian, so we've rigged a systematic political bias against conservatives showing up on our "openness" measure. The fact that conservatives score lower on openness is widely reported and savored by politically biased and incurious science writers and sloppy scientists. Take another look at the items. That's what "openness" is. That's what conservatives are scoring lower on. It's urban intellectualism and perhaps narcissism. Given its profound cultural bias, we have no justification for calling it openness.
Duarté's Behavioral and Brain Sciences paper, co-authored with Jarret T. Crawford, Jonathan Haidt, Lee Jussim, and Philip E. Tetlock: Political Diversity Will Improve Social Psychological Science.
via @SteveStuWill







Frankly, I don't see what either of those lists has to do with what I think of as "openness" (except maybe learning new things). When I think of being open to experiences I think of stuff like:
- when confronted with an unfamiliar food at a pot luck, do you skip it, try a bit, or heap it on?
- given the chance to travel to a new place do you just on it, consider your schedule first, or balk?
- do you (or have you) listened to many genres of music? If not, would you like the chance to?
- when waiting in line, do you strike up conversations with people next to you? (this might measure intro/extroversion, but it is also a sensation seeking behavior)
- If you found out you had to relocate to a new area, would you be excited by the change, even if you felt sad about leaving your current situation?
BUT... in the world of occupational therapy (esp pediatric) there is a whole set of stuff on this - sensory seeking/avoiding behavior. Frankly, the idea of "openness to experience" seems highly related (although not completely the same).
Things like being in the woods - what does that have to do with openness? I get what the author is saying about the obvious bias in those questions, but I think questions that really get at what they are trying to measure - rather than what they think correlates with it, would be more effective. I love the woods, but I am not very open to new experiences - and I am urban and have an advanced degree. I just prefer a lot of predictability.
Now, if they want openness to new thoughts (rather than experiences), that's another story. But MUCH harder to measure. A start might be one of those "do you agree with POLITICO X on this?" questions when it really is POLITICO Y's platform and see who is open to it even after they find out it belongs to Y.
Shannon at July 5, 2016 5:07 AM
Openness to Experience is one of the five big personality traits (remembered with OCEAN).
My friend Scott Barry Kaufman writes about it here:
http://blogs.scientificamerican.com/beautiful-minds/openness-to-experience-and-creative-achievement/
He lays them out at the link.
Amy Alkon at July 5, 2016 5:27 AM
The Big Five (personality traits):
Extraversion, agreeableness, openness, conscientiousness and neuroticism.
https://en.wikipedia.org/wiki/Big_Five_personality_traits
Amy Alkon at July 5, 2016 5:29 AM
This is exactly why these are 'soft sciences'. It is a self reported survey for god's sake! What confidence do we have that people are reporting accurately? How about close to nil. So it is also smothered in left wing academic narcissism. What else is new.
Ben at July 5, 2016 6:17 AM
Very few see themselves as they are. Most of us are above average at everything, or so we believe. Academics are not immune.
MarkD at July 5, 2016 6:20 AM
"Things like being in the woods - what does that have to do with openness? "
I think the point was, if the intent is to measure openness to experience with those types of questions, context is important. "Do you enjoy being in the woods" is no less (or more) valid than "Do you enjoy being in the art museum". If you go to art museums all the time, being in another one probably isn't a new experience. And yes, academic and "intellectual" types like to pat themselves on the back for supposedly being more willing to embrace foreign experiences and ideas -- even when all they are doing is trading the downtown loft in New York for the downtown loft in London or Munich or Austin, all of them having the trendy clothing stores, and the singer/songwriter bars with open mic night on Tuesdays, and the ready-made bevy of sophisticated friends and neighbors from academia and government.
It's not clear to me that openness to new ideas is, in of itself, a virtue. Openness to new ideas can include openness to bad ideas, like socialism. Further, I suspect that tools to measure openness to new experience tend to exhibit a bias that favors the young, for the simple reason that older people have experienced more things and have formed their preferences based on that. A survey question might ask about the subject's willingness to see a rock opera. Younger people who have never experienced one would probably answer yes. I went through all that back in the '70s, enough to know that I don't care for rock opera all that much (I might make an exception for Quadraphenia, which I've never seen a production of), so I'm not going to be excited about the opportunity to be subject to another one.
Cousin Dave at July 5, 2016 6:45 AM
It's somewhat related to the socio-political bias in schools, where the 'honor roll' is pretty much restricted to those with good English and math skills, to the exclusion of shop, car repair, home ec, etc.
Kevin at July 5, 2016 8:49 AM
I wonder how such quizzes would score this particular museum?
http://www.visitcopenhagen.com/copenhagen/royal-arsenal-museum-gdk410541
I know that one because I've been there. I've also been to a couple of Kobenhaven's art museums. Push?
Like Duarte, I can traverse between the Cosmopolitan tribe and the Nativists? Doers? Or, to borrow a Dr. Whoism, Sevateem tribe. And I was born and grew up in a copper town.
I prefer the Sevateems: there is very little wordsmithing, and they generally give you their honest opinions about things. The problem I have with the Cosmos is that they'll use a $5 word when a $0.25 word works perfectly well.
Virtue signalling and all that.
I R A Darth Aggie at July 5, 2016 9:36 AM
Not only that, but the $5 word will be one carefully crafted to be malleable and retroactively re-definable.
Cousin Dave at July 5, 2016 10:27 AM
"... is sophisticated in art, music, and literature."
I think Cousin Dave hits it.
Deciding one is sophisticated is definitely as CD says tend(s) to exhibit a bias that favors the young. And I would say young unmarried childless adults working in big "city" environments. Birds of a feather and all that.
Sad that the word "science" is associated w/this stuff.
Bob in Texas at July 5, 2016 12:32 PM
I know a lot of handy people, such as my friend who built his own airplane and the neighbor who has restored a vintage mustang. These things require a strong drive to do new things, but with the hands, not just the eyes. You could ask if the person enjoys tinkering with a new type of motor or shooting a new gun or looks forward to the challenge of building an addition to the house. For ladies it might be if they enjoy trying new recipes (sorry, hard to think of many analogies to building your own airplane). For software it might be the challenge of a new language or new computing environment. This type of person might not read much but they will have "new" experiences and thus skills constantly.
Simple example, in college I wanted to hike cross country so I took an orienteering class then learned how to read USGS topo maps and memorize them (since they were expensive). Not at ALL the same as trying new foods or going to museums--those are passive activities, consuming sensations rather than being active.
Craig Loehle at July 5, 2016 12:34 PM
Amy, I guess I don't get it.
Kinda sounds like "openness" is everything and the kitchen sink. I mean, what about the insanely creative folks with rich imaginations who are are homebodies? Somebody with insane artistic ability, but who doesn't like to try new things?
Perhaps risk tolerance is getting put in there?
Also, I think Cousin Dave had a good point. I don't want to try fish. Any fish. Old fish. New fish. None. Why? Enough experience getting sick after fish and I'm no longer open to it. I WAS, but I don't think my whole personality changed because I no longer wish to try fish.
Sketchy trait, sketchy measures, any statistical findings are likely spurious.
Shannon at July 5, 2016 1:40 PM
"For ladies it might be if they enjoy trying new recipes "
For this lady it would be the new gun..... :-)
Isab at July 5, 2016 6:21 PM
Why would someone consider an art museum superior to the woods? Art appreciation is only mankind congratulating itself for being an artist.
Radwaste at July 6, 2016 7:32 AM
Leave a comment