New system quantifies language model personalities through linguistic analysis

Large language models (LLMs) are at the forefront of artificial intelligence (AI) and have been widely used for conversational interactions. However, assessing the personality of a given LLM remains a significant challenge.

A research team at The Hong Kong Polytechnic University (PolyU) has developed an AI-driven assessment system, the Language Model Linguistic Personality Assessment (LMLPA), with capabilities to quantitatively measure the personality traits of LLMs through linguistic analysis.

This innovative interdisciplinary research in AI and computational linguistics has led to the development of robust, data-driven AI tools for evaluating nuanced LLM personality traits and behaviors. The LMLPA system represents a critical step forward in understanding LLMs and developing them to be more aligned with human values and needs.

Led by Prof. Lik-Hang Lee, Assistant Professor of the PolyU Department of Industrial and Systems Engineering, the research has been published in Computational Linguistics.

LMLPA is designed to evaluate and characterize the personalities of LLMs by examining the linguistic patterns, style and other language-related features in their outputs. The system comprises two main components: the Adapted Big Five Inventory (Adapted BFI) and the AI rater. LMLPA first administers the Adapted-BFI, which is derived from previous language-based personality assessment theories, to LLMs.

The AI rater then evaluates the responses, converting the textual answers into quantifiable numerical values representing personality traits.

This innovative technology not only advances human-centered AI and computational linguistics by providing a robust framework for refining AI personality assessments, but also has applications in various fields, including education and manufacturing, as well as in business, such as by helping companies with compliance requirements and Environmental, Social and Governance reporting. It also supports sustainable development goals and the enhancement of legal services.

Prof. Lee said, “In addressing the limitation of LLMs in capturing the cognitive and affective dimensions of human personality, our team has successfully developed novel assessment tools to evaluate LLM personalities in linguistic terms, aligning with their functional capabilities and operational paradigms.”

The research opens up new possibilities for understanding and interacting with AI. By quantifying LLM personalities, their communication styles for specific applications can be tailored, paving the way for more personalized interactions between humans and machines.

Prof. Lee has translated the technological foundations of his research to create an AI-driven business compliance platform. Leveraging natural language processing, the platform can analyze and interpret large volumes of textual data and reports, including those generated by LLMs.

AI technology is used to perform automated data collection, analysis and insight generation, significantly streamlining the business compliance and reporting process. The integration of LMLPA highlights AI’s nuanced capabilities in assessing language-based personality traits, suggesting potential applications in the analysis of qualitative business and human data.