Are LLMs effective negotiators?
Systematic evaluation of LLMs reveals their negotiation strengths and limitations in complex dialogue scenarios.
A successful negotiation requires a range of capabilities, including comprehension of the conversation context, Theory-of-Mind (ToM) skills to infer the partner's motives, strategic reasoning, and effective communication, making it challenging for automated systems. Despite the remarkable performance of LLMs in various NLP tasks, there is no systematic evaluation of their capabilities in negotiation. Such an evaluation is critical for advancing AI negotiation agents and negotiation research, ranging from designing dialogue systems to providing pedagogical feedback and scaling up data collection practices. This work aims to systematically analyze the multifaceted capabilities of LLMs across diverse dialogue scenarios throughout the stages of a typical negotiation interaction. Our analysis highlights GPT-4's superior performance in many tasks while identifying specific challenges, such as making subjective assessments and generating contextually appropriate, strategically advantageous responses.
Latest publications
A comparison of strategies for RAG
Evaluation and comparison of multiple RAG fine-tuning strategies.
EMNLPGRAID: Synthetic data generation with geometric constraints and multi-agentic reflection for harmful content detection
A novel pipeline that leverages Large Language Models (LLMs) for dataset augmentation.
EMNLPConfidence-based response abstinence: LLM trustworthiness
A method for confidence estimation in RAG systems that aligns closely with the correctness of LLM outputs.
EMNLP