United States

Using medical data for Deep Learning models can be highly beneficial, but protecting sensitive and personal patient information in the clinical field is critical. One of the most common ways to use this data while protecting patient privacy is by generating synthetic text with Large Language Models (LLMs) using differential privacy (DP). Although DP techniques, such as the Differentially Private Stochastic Gradient Descent (DP-SGD), are often assumed to guarantee privacy, they require specific conditions to be met. This study shows how memorization in LLMs can occur when these privacy guarantees are compromised, potentially leading to the leakage of personal and sensitive information in generated clinical re017 ports. If these gaps are addressed, DP could offer more reliable safeguards for clinical data, improving privacy without sacrificing utility

EMNLP 2024

Analyzing the Leakage of Personal Information in Synthetic Clinical Spanish Texts

synthetic text generation; clinical nlp; llm memorization; privacy preserving; differential privacy;differentially private stochastic gradient descent; spanish nlp

Welcome to the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL). Continuing its mission of expanding and involving the science community of all European countries, EACL has selected the Malta community for the 18th EACL. Considering the importance of physical interaction among researchers, the conference will be held at the Hotel Radisson Blu, St. Julians, in Malta, from 17 to 22 of March, 2024. The conference will also feature the possibility to participate virtually. As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language. 
#### [Conference Handbook](https://drive.google.com/file/d/13Wn38q0yev6U3RTgnf4Tn5T9GuAuZ-xW/view?usp=sharing) *downloadable*

To access the EACL 2024 event page you need to register [**here**](https://acl.swoogo.com/EACL2024). 

Please register!

EACL 2024

As the flagship European conference in the field of computational linguistics, EACL welcomes European and international researchers covering a broad spectrum of research areas that are concerned with computational approaches to natural language.

## Welcome to EMNLP 2024! 
We are excited to welcome you to one of the most prominent conferences in the field of Natural Language Processing. This year, EMNLP 2024 is being held in a hybrid format,
offering both virtual and in-person participation in beautiful Miami. Due to a record-breaking number of submissions, we've expanded the total number of accepted papers to accommodate more cutting-edge research from around the globe.
### [Conference Handbook](https://drive.google.com/file/d/1WPROgxjLAC96AJL7Ugy0tEnYm7dkrbHt/view?usp=sharing)

You are required to register for this event. **Please register [here](https://2024.emnlp.org/registration/).** The EMNLP 2024 event page on Underline will be open to public one week prior to the event.

EMNLP 2024 will take place in Miami, Florida from Nov 12th to Nov 16th, 2024, at the Hyatt Regency Miami Hote and on Underline for remote participants.

Modern language models predict the next token in the sequence by considering the past text through a powerful function. However, language models have no explicit mechanism that allows them to spend computation time for planning long-distance future text, leading to a suboptimal token prediction. In this paper, we propose a planner that predicts a latent plan for many sentences into the future. By sampling multiple plans at once, we condition the language model on an accurate approximation of the distribution of text continuations, which leads to better next token prediction accuracy. In effect, this allows trading computation time for prediction accuracy.

Improving Language Modeling by Increasing Test-time Planning Compute

In this study, we propose several prompt-based methods for predicting ratings of reviews using large language models, specifically GPT-3.5 Turbo and Dorna. We utilize BaSalam dataset, sourced from an Iranian online marketplace. Our approach includes zero-shot and few-shot prompting, as well as Retrieval-Augmented Generation. We evaluate the effectiveness of our methods by comparing them to baseline models, demonstrating superior performance in terms of mean absolute error (MAE) and mean squared error (MSE). After conducting several experiment, the results shows the superiority of in-context learning method with examples retrieved.

Farsi Review Rating Prediction Using Prompt Engineering and RAG

Theory of Mind (ToM) is considered essential in understanding the intentions and beliefs of others. Recent advancements in large language models (LLMs) like ChatGPT have sparked claims that these models exhibit ToM capabilities. However, follow-up studies reveal that these capabilities vanish with slight task variations. This paper introduces a novel dataset comprising 68 tasks across 10 complexity classes, probing ToM in four open-source LLMs. Our results show that ToM abilities in these models are still limited. We highlight challenges and suggest future research directions.

Probing the Robustness of Theory of Mind in Large Language Models

Hate speech studies often focus on binary categories, neglecting its nuanced spectrum. This research introduces a dataset of 8,258 Amharic tweets annotated for category classification, hate target identification, and ratings of intensities. Most tweets are found to be less offensive, highlighting the need for early interventions. The dataset reveals significant ethnic and political hatred overlaps, reflecting Ethiopia's complex sociopolitical dynamics. The study shows that hate speech requires continuous analysis rather than simplistic classifications. The Afro-XLMR-large model achieved an F1 score of 75.30% for category classification, demonstrating effective performance in this area.

Investigating Hate Speech Beyond Detection and Classification: Uncovering Complex Intensities and Targets

Large language models (LLMs) can help elucidate hate, violence, and other toxicity. However, labeling harmful events is challenging due to the subjectivity of labels such as toxicity'' and hate.'' Motivated by the rise of antisemitism, this paper studies the capability of LLMs to discover reports of antisemitic events. We pilot the task of hateful event classification on the AMCHA Corpus---a continuously updated dataset with expert-labeled instances of fine-grained types of antisemitism---and show that incorporating domain knowledge from fine-grained taxonomies is needed to make LLMs more effective. Our experiments find that providing precise definitions from a taxonomy can steer GPT-4 and Llama-3 to somewhat improve on tagging antisemitic event descriptions, with GPT-4 achieving up to a 14% increase in mean weighted F1. However, LLMs are still far from perfect at understanding antisemitic events, suggesting avenues for future work on LLM alignment and precise definition of antisemitism.

On the Importance of Nuanced Taxonomies for LLM-Based Understanding of Harmful Events: A Case Study on Antisemitism

Social movements use social media to draw attention to their cause, disseminate information, coordinate offline action, and build collective identity (Harlow, 2012; Jost et al., 2018). Social media further provides a space for activists, bystanders and opponents to collectively construct and contest narratives of their movement via collective action framing (Snow et al., 1988). Focusing on Twitter (X) messages from social movements surrounding three issues (guns, immigration, and LGBTQ rights), we create a codebook, annotated dataset, and computational models to detect diagnostic (problem identification and attribution), prognostic (proposed solutions and tactics) and motivational (calls to action) framing strategies. We conduct an in-depth unsupervised linguistic analysis of each strategy and uncover cross-movement similarities in associations between framing and linguistic features such as personal pronouns. We then analyze frame variation across a broad set of sociocultural and interactional contexts.

Framing Social Movements on Social Media: Unpacking Diagnostic, Prognostic, and Motivational Strategies

Recent advancements in massively multilingual machine translation systems have significantly enhanced translation accuracy; however, even the best performing systems still generate hallucinations, severely impacting user trust. Detecting hallucinations in Machine Translation (MT) remains a critical challenge, particularly since existing methods excel with High-Resource Languages (HRLs) but exhibit substantial limitations when applied to Low-Resource Languages (LRLs). This paper evaluates sentence-level hallucination detection approaches using Large Language Models (LLMs) and semantic similarity within massively multilingual embeddings. Our study spans 16 language directions, covering HRLs, LRLs, with diverse scripts. We find that the choice of model is essential for performance. On average, for HRLs, Llama3-70B outperforms the previous state of the art by as much as 0.16 MCC (Matthews Correlation Coefficient). However, for LRLs we observe that Claude Sonnet outperforms other LLMs on average by 0.03 MCC. The key takeaway from our study is that LLMs can achieve performance comparable or even better than previously proposed models, despite not being explicitly trained for any machine translation task. However, their advantage is less significant for LRLs.

Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models

News headline generation has seen significant advancements in resource-rich languages like the English language, leveraging sophisticated natural language processing (NLP) techniques. However, similar progress has not been observed for low-resource Indian languages, particularly Telugu. We focus on implementing news headline generation given the news article using an abstractive summarization approach, which enables the generation of contextually rich headlines by interpreting and rephrasing content using deep learning techniques. We create a dataset of articles and headlines scraped from the Telugu news website `Sakshi'. We use pre-trained language models such as mBART50, MT5, and IndicBART for fine-tuning using our dataset. Our findings show the effectiveness of fine-tuning pre-trained models for news headline-generation tasks in Telugu.

News Headline Generation in Telugu

Automatic metrics are extensively used to evaluate natural language processing systems. However, there has been increasing focus on how they are used and reported by practitioners within the field. In this paper, we have con ducted a survey on the use of automatic metrics, focusing particularly on natural language gen eration (NLG) tasks. We inspect which metrics are used as well as why they are chosen and how their use is reported. Our findings from this survey reveal significant shortcomings, in cluding inappropriate metric usage, lack of im plementation details and missing correlations with human judgements. We conclude with rec ommendations that we believe authors should follow to enable more rigour within the field.

Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices

Modern biomedical concept representations are mostly trained on synonymous concept names from a biomedical knowledge base, ignoring the inter-concept interactions and a concept's local neighborhood in a knowledge base graph. In this paper, we introduce Biomedical Entity Representation with a Graph-Augmented Multi-Objective Transformer (BERGAMOT), which adopts the power of pre-trained language models (LMs) and graph neural networks to capture both inter-concept and intra-concept interactions from the multilingual UMLS graph. We apply contrastive loss on textual and graph representations to make them less sensitive to surface forms and enable intermodal knowledge exchange between two uni-modal encoders. BERGAMOT achieves state-of-the-art results in zero-shot entity linking without task-specific supervision on three monolingual datasets and Mantra multilingual benchmark.

Next from EMNLP 2024

Improving Language Modeling by Increasing Test-time Planning Compute

Similar lecture

Differentially Private Natural Language Models: Recent Advances and Future Directions