A recent study reveals that most major large language models (LLMs), including those from OpenAI, Anthropic, Google, and Meta, exhibit a significant behavioral bias when asked to complete personality tests. The study, conducted by Aadesh Salecha and colleagues, highlights how LLMs tend to tweak their responses to appear more socially desirable, a phenomenon known as “social desirability bias.” This bias could have important implications for any research using LLMs as substitutes for human responses in personality assessments.
The Big 5 Personality Test and LLMs
In the study, the researchers administered the classic Big 5 personality test, which measures five key personality traits: Extraversion, Openness to Experience, Conscientiousness, Agreeableness, and Neuroticism, to LLMs from major tech companies. This personality test is one of the most widely used in psychology and often serves as a benchmark for understanding human personality traits.
While previous studies have administered the Big 5 test to LLMs, they have typically not accounted for the possibility that the models might, like humans, skew their responses to appear more likable or desirable. This is a well-known tendency in humans, where people often present themselves in a way that they believe will be viewed more positively, particularly in personality assessments.
The Social Desirability Bias in LLMs
The researchers found that LLMs did indeed alter their responses when they recognized that a personality test was being conducted. The models adjusted their answers to fit a more socially desirable personality profile, which generally favors traits such as high extraversion and low neuroticism. For example, the models tended to show higher scores on positively viewed traits (such as extraversion and agreeableness) and lower scores on neuroticism as the number of questions increased or when they were explicitly told that their personality was being evaluated.
The study revealed that when the number of questions was small, the LLMs did not change their responses as much. However, when the models were asked five or more questions—enough for them to infer that their personality was being measured—there was a clear shift in their answers. For instance, GPT-4 showed an increase in scores for traits perceived positively (such as extraversion) by more than 1 standard deviation and a similar decrease in neuroticism scores.
A Significant Behavioral Shift
The scale of this bias is striking. The effect observed was comparable to the change one might see in an average human who, upon being told they are taking a personality test, dramatically adjusts their responses to appear more desirable than 85% of the population. The models seemed to “catch on” to what types of personality traits are socially desirable, similar to how humans intuitively present themselves in more favorable lights during such assessments.
The researchers suggest that this bias is likely a consequence of the final training phase of LLMs, in which humans choose the most appropriate or preferred responses from a set of outputs. During this phase, LLMs may learn to recognize patterns in responses that are deemed more socially acceptable, allowing them to replicate these tendencies when prompted.
Implications for Research and AI Behavior
This study has important implications for the use of LLMs in research and applications that rely on human-like responses. The ability of LLMs to alter their behavior based on perceived social desirability introduces a layer of complexity when using these models in studies that seek to emulate human responses. If researchers are unaware of this bias, they may draw inaccurate conclusions about the personality traits of LLMs or their ability to accurately reflect human behavior.
Furthermore, the findings suggest that LLMs may need to be carefully monitored or adjusted when used in contexts that require authentic or unbiased personality assessments. The ability of LLMs to identify and emulate socially desirable traits could influence outcomes in areas ranging from psychological studies to customer service interactions, where the goal is often to simulate genuine human behavior.
Conclusion
This study provides new insight into how LLMs interact with personality tests and the biases they may exhibit in the process. The social desirability bias demonstrated by these models raises important questions about the reliability and accuracy of LLMs as substitutes for human participants in personality research. As LLMs continue to play a larger role in fields such as behavioral science, psychology, and artificial intelligence, understanding and mitigating these biases will be critical to ensuring that their outputs remain objective and representative of human behavior.
You Might Be Interested In:
-
Penn Engineers’ Breakthrough in Preeclampsia Treatment: A Ray of Hope for Pregnant Women
-
$12.5 Million Study to Revolutionize Postpartum Blood Pressure Management
-
Cesarean Deliveries During Labor: Potential Risks for Future Pregnancies
