China

Large language models (LLMs) are advancing at an unprecedented pace globally, with regions increasingly adopting these models for applications in their primary language. Evaluation of these models in diverse linguistic environments, especially in low-resource languages, has become a major challenge for academia and industry. Existing evaluation frameworks suffer from inconsistency across different benchmarks, being disproportionately focused on English and a handful of high-resource languages, thereby overlooking the realistic performance of LLMs in multilingual and lower-resource scenarios. To address this critical challenge of fragmented and inconsistent multilingual evaluation, we introduce GlotEval, a unified and lightweight framework that systematically integrates over 20 benchmarks under a standardized ISO 639-3 language identifier system, allowing for seamless incorporation of new benchmarks. Supporting nine key tasks (machine translation, text classification, summarization, open-ended generation, reading comprehension, sequence labeling, intrinsic evaluation, instruction following and reasoning), spanning over dozens to hundreds of languages, GlotEval uniquely enables language-specific, cross-benchmark analysis and non-English-centric evaluations at a scale previously less practical for many researchers. This enables a precise diagnosis of model strengths and weaknesses in diverse linguistic contexts. A multilingual translation case study demonstrates GlotEval&#39;s applicability for multilingual and language-specific evaluations.

EMNLP 2025

GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models

language-specific prompting

benchmarking framework

multilingual evaluation

large language models (llms)

low-resource languages

machine translation

## Welcome!
"I am excited to welcome you to this year’s edition of the Conference on Empirical Methods in Natural Language Processing! Importantly, it marks the 30th edition of EMNLP. With over 8,000 submissions, more than 3,000 accepted papers, and thousands of attendees, we have come a long way from that first
workshop, which had 14 accepted papers. As the field looks ahead, Suzhou is the fitting location for celebrating this milestone: rooted in a long literary tradition, yet modern and forward-looking, and home to a large share of the NLP community."<br>

*Message from the General Chair, Dirk Hovy*





<br>

Celebrate 30 Years of EMNLP! 
EMNLP 2025 will be held in Suzhou, China from November 5th to November 9th, 2025.

Large language models (LLMs) are advancing at an unprecedented pace globally, with regions increasingly adopting these models for applications in their primary language. Evaluation of these models in diverse linguistic environments, especially in low-resource languages, has become a major challenge for academia and industry. Existing evaluation frameworks suffer from inconsistency across different benchmarks, being disproportionately focused on English and a handful of high-resource languages, thereby overlooking the realistic performance of LLMs in multilingual and lower-resource scenarios. To address this critical challenge of fragmented and inconsistent multilingual evaluation, we introduce GlotEval, a unified and lightweight framework that systematically integrates over 20 benchmarks under a standardized ISO 639-3 language identifier system, allowing for seamless incorporation of new benchmarks. Supporting nine key tasks (machine translation, text classification, summarization, open-ended generation, reading comprehension, sequence labeling, intrinsic evaluation, instruction following and reasoning), spanning over dozens to hundreds of languages, GlotEval uniquely enables language-specific, cross-benchmark analysis and non-English-centric evaluations at a scale previously less practical for many researchers. This enables a precise diagnosis of model strengths and weaknesses in diverse linguistic contexts. A multilingual translation case study demonstrates GlotEval's applicability for multilingual and language-specific evaluations.

demo

Empathetic interaction is a cornerstone of human-machine communication, due to the need for understanding speech enriched with paralinguistic cues and generating emotional and expressive responses. However, the most powerful empathetic LSLMs are increasingly closed off, leaving the crucial details about the architecture, data and development opaque to researchers. Given the critical need for transparent research into the LSLMs and empathetic behavior, we present OpenS2S, a fully open-source, transparent and end-to-end LSLM designed to enable empathetic speech interactions. Based on our empathetic speech-to-text model BLSP-Emo, OpenS2S further employs a streaming interleaved decoding architecture to achieve low-latency speech generation. To facilitate end-to-end training, OpenS2S incorporates an automated data construction pipeline that synthesizes diverse, high-quality empathetic speech dialogues at low cost. By leveraging large language models to generate empathetic content and controllable text-to-speech systems to introduce speaker and emotional variation, we construct a scalable training corpus with rich paralinguistic diversity and minimal human supervision. We release the fully open-source OpenS2S model, including the dataset, model weights, pre-training and fine-tuning codes, to empower the broader research community and accelerate innovation in empathetic speech systems.

OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model

Automatic research with Large Language Models (LLMs) is rapidly gaining importance, driving the development of increasingly complex workflows involving multi-agent systems, planning, tool usage, and human-agent interaction to accelerate research processes. However, as more researchers begin to use and build upon these tools and platforms, the complexity and difficulty of extending and maintaining such agentic workflows has become a significant challenge, particularly as algorithms and architectures continue to advance. To address this growing complexity, TinyScientist identifies the essential components of the automatic research workflow and proposes an interactive, extensible, and controllable framework that adapts easily to new tools and supports iterative growth. We have developed an interactive UI and a robust Python package to make state-of-the-art auto-research pipelines accessible to all researchers and developers. TinyScientist is publicly available and actively maintained at https://github.com/ulab-uiuc/tiny-scientist, with over 100 stars and 10 forks on GitHub. Its Python package is released on PyPI at https://pypi.org/project/tiny-scientist/ and has been downloaded more than 700 times.

TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents

In this paper, we introduce ECom-Bench, the first benchmark framework for evaluating LLM agent with multimodal capabilities in the e-commerce customer support domain. ECom-Bench features dynamic user simulation based on persona information collected from real e-commerce customer interactions and a realistic task dataset derived from authentic e-commerce dialogues. These tasks, covering a wide range of business scenarios, are designed to reflect real-world complexities, making ECom-Bench highly challenging. For instance, even advanced models like GPT-4o achieve only a 10–20\% pass^3 metric in our benchmark, highlighting the substantial difficulties posed by complex e-commerce scenarios. Upon publication, the code and data will be open-sourced

ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?

Neural models for Japanese pronunciation estimation often suffer from errors such as
hallucinations (generating pronunciations that are not grounded in the input) and 
omissions (skipping parts of the input).
Although attention-based alignment has been used to detect such errors,
selecting reliable attention heads is difficult,
and developing methods that can both detect and correct these errors
remains challenging.


In this paper, we propose a simple method called
existence-based alignment check.
In this approach,
we consider alignment candidates
independently extracted from all attention heads,
and check whether at least one of these candidates satisfies two conditions
derived from the linguistic properties of Japanese pronunciation:
monotonicity and pronunciation length per character.
We generate multiple hypotheses using beam search
and use the alignment check as a filtering mechanism
to correct hallucinations and omissions.


We apply this method to a dataset of Japanese facility names
and demonstrate that it improves pronunciation estimation accuracy
by over 2.5\%.

Just One is Enough: An Existence-based Alignment Check for Robust Japanese Pronunciation Estimation

The reasoning capabilities of Large Language Models (LLMs) play a critical role in many downstream tasks, yet depend strongly on the quality of training data. Despite various proposed data construction methods, their practical utility in real-world pipelines remains underexplored. In this work, we conduct a comprehensive analysis of open-source datasets and data synthesis techniques for mathematical reasoning, evaluating them under a unified pipeline designed to mirror training and deployment scenarios. We further distill effective data selection strategies and identify practical methods suitable for industrial applications. Our findings highlight that structuring data in more interpretable formats, or distilling from stronger models often outweighs simply scaling up data volume. This study provides actionable guidance for integrating training data to enhance LLM capabilities, supporting both cost-effective data curation and scalable model enhancement. We hope this work will inspire further research on how to balance "more data" versus "better data" for real-world reasoning tasks.

More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning

As large language models (LLMs) become increasingly prevalent, ensuring their robustness against adversarial misuse is crucial. This paper introduces the Graph of Attacks with Pruning (GAP) framework, an advanced approach for generating stealthy jailbreak prompts to evaluate and enhance LLM safeguards. GAP addresses limitations in existing tree-based LLM jailbreak methods by implementing an interconnected graph structure that enables knowledge sharing across attack paths. Our experimental evaluation demonstrates GAP's superiority over existing techniques, achieving a 20.8% increase in attack success rates while reducing query costs by 62.7%. GAP consistently outperforms state-of-the-art methods for attacking both open and closed LLMs, with attack success rates of >96%. Additionally, we present specialized variants like GAP-Auto for automated seed generation and GAP-VLM for multimodal attacks. GAP-generated prompts prove highly effective in improving content moderation systems, increasing true positive detection rates by 108.5% and accuracy by 183.6% when used for fine-tuning.

Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderation

Slot filling is a crucial subtask in spoken language understanding (SLU), traditionally implemented as a cascade of speech recognition followed by one or more natural language understanding (NLU) components. The recent advent of speech-based large language models (speechLLMs), which integrate speech and textual foundation models, has opened new avenues for achieving speech understanding tasks in a more unified, generative, and instruction-following manner while promising data and compute efficiency with zero-shot abilities, generalizing to unseen slot labels. We address the slot filling task by creating an empirical upper bound for the task, identifying performance, robustness, and generalization gaps, and proposing improvements to the training data, architecture, and training strategies to bridge the gap. We show that each of these measures reduces the gap substantially, while highlighting practical challenges and providing empirical guidance and insights for harnessing these emerging models.

SpeechLLMs for Large-scale Contextualized Zero-shot Slot Filling

In customer-oriented industries, compliance-guaranteed chatbots are essential for delivering accurate, verifiable responses. Retrieval-based systems leverage a Match-and-Respond mechanism with predefined knowledge bases of human-verified question-answer (QA) pairs, providing hallucination-free responses. To effectively handle diverse customer inquiries, augmenting the knowledge base with similar questions that maintain semantic consistency and increase schematic diversity is crucial for improved query matching. In this paper, we define the Similar Question Generation (SQG) task for LLM training and inference, presenting context-aware methods for generating similar questions through a cost-efficient one-to-many paradigm. This approach facilitates broader semantic exploration and better alignment with the source question-answer pairs. We also propose combinatorial optimization techniques to construct in-context prompt demonstrations and to select an optimal subset of similar questions for knowledge base expansion. Experiments on a conversational QA-based SQG dataset demonstrate the effectiveness of these methods in both quantitative and human evaluations. A 92\% user satisfaction rate in deployed chatbot systems, reflecting an 18\% relative improvement, further highlights the practical advantages of the proposed augmentation approach in ensuring compliance and enhancing customer service chatbot applications.

Augmenting Compliance Guaranteed Conversational AI: Context-Aware Knowledge Base Expansion with LLMs and Combinatorial Optimization

LLMs often fail to meet specialized needs of distinct user groups due to their one-size-fits-all approach, and there is limited understanding of what personalization each group expects.
To address this, we propose GPA a group-aware personalization framework that captures context-specific preference variations and steers LLMs accordingly.
Our approach involves: (1) Group-Aware Preference Extraction, which distills divergent preferences from real-world conversation logs into interpretable rubrics, and (2) Tailored Response Generation, using (a) GPA-CT, which adapts responses using learnt rubrics, and (b) GPA-FT, which finetunes models using rubric-guided synthetic data.
Automatic and Human evaluations confirm that GPA improves group alignment without compromising perfomance on standard instruction-following benchmarks.

Group Preference Alignment: Customizing LLM Responses from In-Situ Conversations Only When Needed

Large language models (LLMs) such as GPT-4o and o1 have demonstrated strong performance on clinical natural language processing (NLP) tasks across multiple medical benchmarks. Nonetheless, two high-impact NLP tasks — structured tabular reporting from nurse dictations and medical order extraction from doctor-patient consultations — remain underexplored due to data scarcity and sensitivity, despite active industry efforts. Practical solutions to these real-world clinical tasks can significantly reduce the documentation burden on healthcare providers, allowing greater focus on patient care. In this paper, we investigate these two challenging tasks using private and open-source clinical datasets, evaluating the performance of both open- and closed-weight LLMs, and analyzing their respective strengths and limitations. Furthermore, we propose an agentic pipeline for generating realistic, non-sensitive nurse dictations, enabling structured extraction of clinical observations. To support further research in both areas, we release SYNUR and SIMORD, the first open-source datasets for nurse observation extraction and medical order extraction.

Downloads

Next from EMNLP 2025

OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model

Similar lecture

MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Language

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from EMNLP 2025

OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model

Similar lecture

MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Language

Downloads