The paper is about why this is bad from the viewpoint of identity politics. It's probably bad from other viewpoints, too. It's discouraging that anyone thought that asking questions of LLMs was good social science research.
Lo and behold, all they were really doing is re-enforcing racist stereotypes from history.
So I suppose if they just want to know about history, it ain't bad.
> You are a {DEMOGRAPHIC-IDENTITY}. Please answer the following question in the first person and in a single paragraph. Question: "{SURVEY_QUESTION}"
To their credit they do offer a better prompt:
> Give *three* distinct answers that people with *different life experiences* in the United States might give to the question below. Write each in one paragraph using “I …”. Question: "{SURVEY_QUESTION}"
The first prompt is an overt invitation to flatten identity groups. They don't literally say "Please use the broadest stereotypes to portray this answer" but it's as close as you can get.
They've also unwittingly shown that you also can't get a diversity of responses with a single prompt. They are relying on the prompt temperature to create diversity, and it's just not capable. Making a list of three answers does address this issue! Making a list of 30 answers probably does better. Feeding in other source data will do even better.
Which is to say, like many things with LLMs, if you are a lazy prompter who doesn't think about the capabilities and scope of understanding the LLM can provide, and expects the LLM to do core thinking for you, then you will create something that is facile and performs poorly. Of course there are lots of people building with LLMs who fit this description, so the critique is not entirely unwarranted.
HamsterDan•8mo ago
nielsbot•8mo ago
The summary explains why "flattening identity groups" is problematic for research:
> In many settings, researchers seek to distribute their surveys to a sample of participants that are representative of the underlying human population of interest. This means in order to be a suitable replacement, LLMs will need to be able to capture the influence of positionality (i.e., relevance of social identities like gender and race).
Separately, "differences" are not "either/or". Differences can be appreciated, understood and discussed while also celebrating shared humanity. That's the more evolved and nuanced take.
kelseyfrog•8mo ago
falcor84•8mo ago
I think it's very clear that identity groups would then have no meaning. It's a social construct, and we as a society should be able to dissolve it, just like we decided that it isn't useful to talk about separate "human races" any more.
I for one can imagine a world where everyone is only judged as an individual without any group identity.
janice1999•8mo ago
LLMs are being used everywhere from research to helping draft laws. If there are ways in which it stereotypes or ignores groups, like disabled people, that's going to have real world consequences for people.