Austria

We introduce SECQUE, a comprehensive benchmark for evaluating large language models (LLMs) in financial analysis tasks. SECQUE comprises 565 expert-written questions covering SEC filings analysis across four key categories: comparison analysis, ratio calculation, risk assessment, and financial insight generation. To assess model performance, we develop SECQUE-Judge, an evaluation mechanism leveraging multiple LLM-based judges, which demonstrates strong alignment with human evaluations. Additionally, we provide an extensive analysis of various models&#39; performance on our benchmark. By making SECQUE publicly available (https://anonymous.4open.science/r/sqbp-F6E6), we aim to facilitate further research and advancements in financial AI.

ACL 2025

SECQUE: A Benchmark for Evaluating Real-World Financial Analysis Capabilities

## Welcome!
"I am excited to welcome you to this year’s edition of the Conference on Empirical Methods in Natural Language Processing! Importantly, it marks the 30th edition of EMNLP. With over 8,000 submissions, more than 3,000 accepted papers, and thousands of attendees, we have come a long way from that first
workshop, which had 14 accepted papers. As the field looks ahead, Suzhou is the fitting location for celebrating this milestone: rooted in a long literary tradition, yet modern and forward-looking, and home to a large share of the NLP community." 

*Message from the General Chair, Dirk Hovy*

EMNLP 2025

Celebrate 30 Years of EMNLP! 
EMNLP 2025 will be held in Suzhou, China from November 5th to November 9th, 2025.

We introduce SECQUE, a comprehensive benchmark for evaluating large language models (LLMs) in financial analysis tasks. SECQUE comprises 565 expert-written questions covering SEC filings analysis across four key categories: comparison analysis, ratio calculation, risk assessment, and financial insight generation. To assess model performance, we develop SECQUE-Judge, an evaluation mechanism leveraging multiple LLM-based judges, which demonstrates strong alignment with human evaluations. Additionally, we provide an extensive analysis of various models' performance on our benchmark. By making SECQUE publicly available (https://anonymous.4open.science/r/sqbp-F6E6), we aim to facilitate further research and advancements in financial AI.

workshop paper

### Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Message from the General Chair: 
*It is my great pleasure and honor to welcome you to the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), held in beautiful Vienna, Austria, from July 27 to August 1, 2025. ACL2025continues our field’s tradition of excellence in scholarship, innovation, and inclusivity, and I am deeply grateful to the many volunteers who have worked tirelessly to bring this event to life.* 
[Read more](https://drive.google.com/file/d/1GI_hvOpjswAuYdUTromfeDiPpCcqidwg/view?usp=sharing)

To access this event page, you need to log in with the **email address you registered with**. Access credentials will be sent to your email from Underline - subject line "Welcome to ACL 2025". Please be sure to check your spam email folder if you do not see an email confirmation right away.

Please log in

To access this event page, you need to log in with the **email address you registered with**. 

Welcome to The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

Chatbots for customer service have been widely studied in many different fields, ranging from Natural Language Processing (NLP) to Communication Science. These fields have developed different evaluation practices to assess chatbot performance (e.g., fluency, task success) and to measure the impact of chatbot usage on the user's perception of the organisation controlling the chatbot (e.g., brand attitude) as well as their willingness to enter a business transaction or to continue to use the chatbot in the future (i.e., purchase intention, reuse intention). While NLP researchers have developed many automatic measures of success, other fields mainly use questionnaires to compare different chatbots. This paper explores the extent to which we can bridge the gap between the two, and proposes a research agenda to further explore this question.

Measure only what is measurable: towards conversation requirements for evaluating task-oriented dialogue systems

While LLMs excel at processing standard linguistic inputs but often underperform when faced with transformed or non-standard texts. This paper introduces a multilingual benchmark and evaluation framework designed to assess LLMs' ability to restore original meanings from heavily modified or playful text in both English and Korean. Our dataset includes visual and phonetic substitutions, abbreviations, and character-level manipulations, commonly found in informal digital communication. We conduct a comprehensive evaluation of state-of-the-art LLMs under three prompting paradigms: Zero-Shot, CoT, and CoT+ICL. Results highlight performance gaps, especially in Korean tasks compared to English tasks, and show how prompt engineering and model architecture influence robustness to linguistic variations. This benchmark emulates real-world linguistic distortions observed in informal and online settings, serving as a complementary resource to existing clean-text benchmarks. This allows us to identify robustness gaps in LLMs when handling realistic noisy inputs. It is accessible at: https://anonymous.4open.science/r/Text-Manipulation-C58B

Evaluating LLMs Beyond Standard Text: A Benchmark on Non-Traditional Text Variations

Referring Expression Generation (REG) is a core task for evaluating the pragmatic competence of vision-language systems, requiring not only accurate semantic grounding but also adherence to principles of cooperative communication~\citep{grice1975logic}. 
However, current evaluations of vision-language models (VLMs) often overlook the pragmatic dimension, reducing REG to a region-based captioning task and neglecting Gricean maxims. 
In this work, we revisit REG from a pragmatic perspective, introducing a new dataset of 1.5k images annotated with both written and spoken referring expressions. 
Through a systematic evaluation of state-of-the-art VLMs, we identify three key failures of pragmatic competence: (1) failure to uniquely identify the referent, (2) inclusion of excessive or irrelevant information, and (3) misalignment with human pragmatic preference, such as the underuse of minimal spatial cues. 
We also show that standard automatic evaluations fail to capture these pragmatic violations, reinforcing superficial cues rather than genuine referential success. 
Our findings call for a renewed focus on pragmatically informed models and evaluation frameworks that align with real human communication.

Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation

The impact of subword tokenization on language model performance is well-documented for perplexity, with finer granularity consistently reducing this intrinsic metric. However, research on how different tokenization schemes affect a model's understanding capabilities remains limited, particularly for non-Latin script languages. Addressing this gap, we conducted a comprehensive evaluation of six distinct tokenization strategies by pretraining transformer-based language models for Nepali and evaluating their performance across multiple downstream tasks. While recent prominent models like GPT, RoBERTa, Claude, LLaMA, Mistral, Falcon, and MPT have adopted byte-level BPE tokenization, our findings demonstrate that for Nepali, SentencePiece tokenization consistently yields superior results on understanding-based tasks. Unlike previous studies that primarily focused on BERT-based architectures, our research specifically examines sequential transformer models, providing valuable insights for language model development in low-resource languages and highlighting the importance of tokenization strategy beyond perplexity reduction.

Can Perplexity Predict Finetuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali

Open-domain Dialogue (OD) exhibits a one-to-many (o2m) property, whereby multiple appropriate responses exist for a single dialogue context. Despite prior research showing that modeling this property boosts response diversity, most modern LLM-based dialogue agents do not explicitly do so. In this work, we model the o2m property of OD in LLMs by decomposing OD generation into two key tasks: Multi-Response Generation (MRG) and Preference-based Selection (PS), which entail generating a set of $n$ semantically and lexically diverse high-quality responses for a given dialogue context, followed by selecting a single response based on human preference, respectively. To facilitate MRG and PS, we introduce o2mDial, a dialogue corpus explicitly designed to capture the o2m property by featuring multiple plausible responses for each context. Leveraging o2mDial, we propose new in-context learning and instruction-tuning strategies, as well as novel evaluation metrics for MRG, alongside a model-based approach for PS. Empirical results demonstrate that applying the proposed two-stage framework to smaller LLMs for OD generation enhances overall response diversity while maintaining contextual coherence, improving response quality by up to 90\%, bringing them closer to the performance of larger models.

Modeling the One-to-Many Property in Open-Domain Dialogue with LLMs

Despite the outstanding performance of large language models (LLMs) across various NLP tasks, hallucinations in LLMs--where LLMs generate inaccurate responses--remains as a critical problem as it can be directly connected to a crisis of building safe and reliable LLMs. Uncertainty estimation is primarily used to measure hallucination levels in LLM responses so that correct and incorrect answers can be distinguished clearly. This study proposes an effective uncertainty estimation approach, Clustering-based semantic consistency (Cleanse). Cleanse quantifies the uncertainty with the proportion of the intra-cluster consistency in the total consistency between LLM hidden embeddings which contain adequate semantic information of generations, by employing clustering. The effectiveness of Cleanse for detecting hallucination is validated using four off-the-shelf models, LLaMA-7B, LLaMA-13B, LLaMA2-7B and Mistral-7B and two question-answering benchmarks, SQuAD and CoQA.

Cleanse: Uncertainty Estimation Approach Using Clustering-based Semantic Consistency in LLMs

We introduce CUS-QA, a manually curated, open-ended question-answering dataset specialized in local common knowledge.
The dataset contains 3,904 question-answer pairs across three regions (Czechia, Slovakia, Ukraine) and two modalities (text, visual).
CUS-QA is augmented with manual English translations to all questions and answers, as well as automatically generated baseline answers by a number of prominent open-weight large language models.
While the results using automatic metrics reveal differences between the tested models and languages,
the correlation between these metrics and human judgment is yet to be assessed, and our preliminary experiments suggest only a negligible match.

CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset

Evaluating the quality and relevance of textual outputs from Large Language Models (LLMs) remains challenging and resource-intensive.
Existing automated metrics often fail to capture the complexity and variability inherent in LLM-generated outputs.
Moreover, these metrics typically rely on explicit reference standards, limiting their use mostly to domains with objective benchmarks.
This work introduces a novel evaluation framework designed to approximate expert-level assessments of LLM-generated content.
The proposed method employs pairwise comparisons of outputs by multiple LLMs, reducing biases from individual models.
An Elo rating system is used to generate stable and interpretable rankings.
Adjustable agreement thresholds—from full unanimity to majority voting—allow flexible control over evaluation confidence and coverage.
The method’s effectiveness is demonstrated through evaluating competency profiles extracted from scientific abstracts.
Preliminary results show that automatically derived rankings correlate well with expert judgments, significantly reducing the need for extensive human intervention.
By offering a scalable, consistent, and domain-agnostic evaluation layer, the framework supports more efficient and reliable quality assessments of LLM outputs across diverse applications.

(Towards) Scalable Reliable Automated Evaluation with Large Language Models

The power of Large Language Models (LLMs) in user workflows has increased the desire to access such technology in everyday work. While the ability to interact with models provides noticeable benefits, it also presents challenges in terms of how much trust a user should put in the system’s responses. This is especially true for external commercial and proprietary models where there is seldom direct access and only a response from an API is provided. While standard evaluation metrics, such as accuracy, provide starting points, they often may not provide enough information to users in settings where the confidence in a system’s response is important due to downstream or real-world impact, such as in Question & Answering (Q&A) workflows. To support users in assessing how accurate Q&A responses from such black-box LLMs scenarios are, we develop an uncertainty estimation framework that provides users with an analysis using a Dirichlet mixture model accessed from probabilities derived from a zero-shot classification model. We apply our framework to responses on the BoolQ Yes/No questions from GPT models, finding the resulting clusters allow a better quantification of uncertainty, providing a more fine-grained quantification of accuracy and precision across the space of model output while still being computationally practical. We further demonstrate its generalizability and reusability of the uncertainty model by applying it to a small set of Q&A collected from U.S. government websites.

Clustering Zero-Shot Uncertainty Estimations to Assess LLM Response Accuracy for Yes/No Q&A

The LLM-as-a-judge paradigm offers a potential solution to scalability issues in human evaluation of large language models (LLMs), but there are still many open questions about its strengths, weaknesses, and potential biases. This study investigates thirteen models, ranging in size and family, as 'judge models' evaluating answers from nine base and instruction-tuned 'exam-taker models'. We find that only the best (and largest) models show reasonable alignment with humans, though they still differ with up to 5 points from human-assigned scores. Our research highlights the need for alignment metrics beyond percent agreement, as judges with high agreement can still assign vastly different scores. We also find that smaller models and the lexical metric contains can provide a reasonable signal in ranking the exam-taker models. Further error analysis reveals vulnerabilities in judge models, such as sensitivity to prompt complexity and a bias toward leniency. Our findings show that even the best judge models differ from humans in this fairly sterile setup, indicating that caution is warranted when applying judge models in more complex scenarios.

Downloads

Next from ACL 2025

Measure only what is measurable: towards conversation requirements for evaluating task-oriented dialogue systems

Similar lecture

Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ACL 2025

Measure only what is measurable: towards conversation requirements for evaluating task-oriented dialogue systems

Similar lecture

Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models

Downloads