Taiwan

Reasoning with preconditions such as “glass can be used for drinking water unless the glass is shattered” remains an open problem for language models. The main challenge lies in the scarcity of preconditions data and the model’s lack of support for such reasoning. We present PInKS, Preconditioned Commonsense Inference with WeaK Supervision, an improved model for reasoning with preconditions through minimum supervision. We show, both empirically and theoretically, that PInKS improves the results on benchmarks focused on reasoning with the preconditions of commonsense knowledge (up to 40% Macro-F1 scores). We further investigate PInKS through PAC-Bayesian informativeness analysis, precision measures, and ablation study.

To view the proceedings, please [click here](https://aclanthology.org/2022.aacl-main.26/)

IJCNLP-AACL 2022

PInKS: Preconditioned Commonsense Inference with Minimal Supervision

preconditions

natural language inference

commonsense

The purpose of the AAAI conference is to promote research in artificial intelligence (AI) and scientific exchange among AI researchers, practitioners, scientists, and engineers in affiliated disciplines. 

AAAI 2021

poster

The AACL, Asia-Pacific Chapter of the Association for Computational Linguistics, and the AFNLP, Asian Federation of Natural Language Processing, invite you to participate in the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL) and the 10th International Joint Conference on Natural Language Processing (IJCNLP). 
<br>
<br>

A registration is required to access this event. Access to the event site will be granted to all registered attendees on **November 16th**. If you have not already done so, please register using the link below:
<br>
<br>https://conference.iis.sinica.edu.tw/servlet/Register?ConferenceID=411

Registration Is Required

The AACL, Asia-Pacific Chapter of the Association for Computational Linguistics, and the AFNLP, Asian Federation of Natural Language Processing, invite you to participate in the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL) and the 10th International Joint Conference on Natural Language Processing (IJCNLP). 

In this research, we propose CNN based CoCNN+ model can beat transformer models in Bengali and Hindi language modeling - two of the most popular languages in the world. These languages have high inflection, flexible word order, and a lot of compound characters making them morphologically rich. Base CoCNN model has two components – 1D CNN based vocabulary learner works at word level, while terminal coordinator works at document level. We analyze CNN, LSTM and Transformer based models for Bengali language modeling using perplexity score. CNNs do a better job than LSTMs because of the flexible word order in Bengali. CoCNN has on par performance with pretrained BERT. We extend the base CoCNN to CoCNN+ by incorporating residual connection in the shallow layers of vocabulary learner and by introducing parallel branches of different convolution widths in the terminal coordinator. These modifications help CoCNN beat pretrained BERT for Bengali and Hindi next word prediction, text classification and sentiment analysis tasks.

To view the proceedings, please [click here](https://aclanthology.org/2022.aacl-main.4/)

CNN for Modeling Sanskrit Originated Bengali and Hindi Language

Word embeddings and pre-trained language models have become essential technical elements in natural language processing. While the general practice is to use or fine-tune publicly available models, there are significant advantages in creating or pre-training unique models that match the domain. The performance of the models degrades as language changes or evolves continuously (semantic shift), but the high cost of model building inhibits regular re-training, especially for the language models. This study designs a methodology for observing time-series performance degradation of word embeddings and pre-trained language models using semantic shift in a corpus. We define an efficiently computable metric named Semantic Shift Stability based on the degree of semantic shift. In the experiments, we create models that vary by time series and reveal the performance degradation in two datasets, Japanese and English. Several case studies demonstrate that Semantic Shift Stability supports decision-making as to whether a model should be re-trained. The source code is available at https://github.com/Nikkei/semantic-shift-stability.

To view the proceedings, please [click here](https://aclanthology.org/2022.aacl-main.17/)

Semantic Shift Stability: Efficient Way to Detect Performance Degradation of Word Embeddings and Pre-trained Language Models

As we begin to see low-powered computing paradigms (Neuromorphic Computing, Spiking Neural Networks, etc.) becoming more popular, learning binary word embeddings has become increasingly important for supporting NLP applications at the edge. Existing binary word embeddings are mostly derived from pretrained real-valued embeddings through different simple transformations, which often break the semantic consistency and the so-called "arithmetic'' properties learned by the original, real-valued embeddings. This paper aims to address this limitation by introducing a new approach to learn binary embeddings from scratch, preserving the semantic relationships between words as well as the arithmetic properties of the embeddings themselves. To achieve this, we propose a novel genetic algorithm to learn the relationships between words from existing word analogy data-sets, carefully making sure that the arithmetic properties of the relationships are preserved. Evaluating our generated 16, 32, and 64-bit binary word embeddings on Mikolov's word analogy task shows that more than 95% of the time, the best fit for the analogy is ranked in the top 5 most similar words in terms of cosine similarity.

To view the proceedings, please [click here](https://aclanthology.org/2022.aacl-main.52/)

Analogy-Guided Evolutionary Pretraining of Binary Word Embeddings

In this work, we introduced a novel inference problem that calls for research towards unifying the Representations of Knowledge Graphs and Expert Rules for Machine Learning and Reasoning. Our work is motivated by the following overarching question: With a knowledge graph and a set of if-then rules, can we reason about the conclusions given a set of observations w/ or w/o any labeled data? Based on our work, the short answer to this question is, yes, we can. And we show that the proposed framework has great potential in performing complex inference tasks through empirical study.

Note that our work should be viewed as a proof of concept of the general idea. Please check out the paper for more details. Thank you!

To view the proceedings, please [click here](https://aclanthology.org/2022.aacl-main.20/)

Towards Integrating Knowledge Graph and Expert Rules for Machine Learning and Reasoning

We aim to develop methods for understanding how multimedia news exposure can affect people’s emotional responses, and we especially focus on news content related to gun violence, a very important yet polarizing issue in the U.S. We created the dataset NEmo+ by significantly extending the U.S. gun violence news-to-emotions dataset, BU-NEmo, from 320 to 1,297 news headline and lead image pairings and collecting 38,910 annotations in a large crowdsourcing experiment. In curating the NEmo+ dataset, we developed methods to identify news items that will trigger similar versus divergent emotional responses. For news items that trigger similar emotional responses, we compiled them into the NEmo+-Consensus dataset. We benchmark models on this dataset that predict a person's dominant emotional response toward the target news item (single-label prediction). On the full NEmo+ dataset, containing news items that would lead to both differing and similar emotional responses, we also benchmark models for the novel task of predicting the distribution of evoked emotional responses in humans when presented with multi-modal news content. Our single-label and multi-label prediction models outperform baselines by large margins across several metrics.

To view the proceedings, please [click here](https://aclanthology.org/2022.aacl-main.29/)

Prediction of People's Emotional Response towards Multi-modal News

Current language models achieve low perplexity but their resulting generations still suffer from toxic responses, repetitiveness, and contradictions. The standard language modeling setup fails to address these issues. In this paper, we introduce a new architecture, Director, that consists of a unified generator-classifier with both a language modeling and a classification head for each output token. Training is conducted jointly using both standard language modeling data, and data labeled with desirable and undesirable sequences. Experiments in several settings show that the model has competitive training and decoding speed compared to standard language models while yielding superior results, avoiding undesirable behaviors while maintaining generation quality. It also outperforms existing model-guiding approaches in terms of both accuracy and efficiency.

To view the proceedings, please [click here](https://aclanthology.org/2022.aacl-main.39/)

Director: Generator-Classifiers For Supervised Language Modeling

Speech translation for subtitling (SubST) is the task of automatically translating speech data into well-formed subtitles by inserting subtitle breaks compliant to specific displaying guidelines. Similar to speech translation (ST), model training requires parallel data comprising audio inputs paired with their textual translations. In SubST, however, the text has to be also annotated with subtitle breaks. So far, this requirement has represented a bottleneck for system development, as confirmed by the dearth of publicly available SubST corpora. To fill this gap, we propose a method to convert existing ST corpora into SubST resources without human intervention. We build a segmenter model that automatically segments texts into proper subtitles by exploiting audio and text in a multimodal fashion, achieving high segmentation quality in zero-shot conditions. Comparative experiments with SubST systems respectively trained on manual and automatic segmentations result in similar performance, showing the effectiveness of our approach.

To view the proceedings, please [click here](https://aclanthology.org/2022.aacl-short.59/)

Dodging the Data Bottleneck: Automatic Subtitling with Automatically Segmented ST Corpora

Encoder-decoder architecture is widely adopted for sequence-to-sequence modeling tasks. For machine translation, despite the evolution from long short-term memory networks to Transformer networks, plus the introduction and development of attention mechanism, encoder-decoder is still the de facto neural network architecture for state-of-the-art models. While the motivation for decoding information from some hidden space is straightforward, the strict separation of the encoding and decoding steps into an encoder and a decoder in the model architecture is not necessarily a must. Compared to the task of autoregressive language modeling in the target language, machine translation simply has an additional source sentence as context. Given the fact that neural language models nowadays can already handle rather long contexts in the target language, it is natural to ask whether simply concatenating the source and target sentences and training a language model to do translation would work. In this work, we investigate the aforementioned concept for machine translation. Specifically, we experiment with bilingual translation, translation with additional target monolingual data, and multilingual translation. In all cases, this alternative approach performs on par with the baseline encoder-decoder Transformer, suggesting that an encoder-decoder architecture might be redundant for neural machine translation.

To view the proceedings, please [click here](https://aclanthology.org/2022.aacl-main.43/)

Is Encoder-Decoder Redundant for Neural Machine Translation?

Given the challenges and complexities introduced while dealing with Dialect Arabic (DA) variations, Transformer based models, e.g., BERT, outperformed other models in dealing with the DA identification task. However, to fine-tune these models, a large corpus is required. Getting a large number high quality labeled examples for some Dialect Arabic classes is challenging and time-consuming. In this paper, we address the Dialect Arabic Identification task. We extend the transformer-based models, ARBERT and MARBERT, with unlabeled data in a generative adversarial setting using Semi-Supervised Generative Adversarial Networks (SS-GAN).
Our model enabled producing high-quality embeddings for the Dialect Arabic examples and aided the model to better generalize for the downstream classification task given a few labeled examples. Experimental results showed that our model reached better performance and faster convergence when only a few labeled examples are available.

To view the proceedings, please [click here](https://aclanthology.org/2022.aacl-main.16/)

Arabic Dialect Identification with a Few Labeled Examples Using Generative Adversarial Networks

Recently, emerging approaches have been proposed to deal with robotic navigation problems, especially vision-and-language navigation task which is one of the most realistic indoor navigation challenge tasks. This task can be modelled as a sequential decision-making problem, which is suitable to be solved by deep reinforcement learning. Unfortunately, the observations provided from the simulator in this task are not fully observable states, which exacerbate the difficulty of implementing reinforcement learning. To deal with this challenge, this paper presents a novel method, called as attentive variational state tracker (AVAST), a variational approach to approximate belief state distribution for the construction of a reinforced navigator. The variational approach is introduced to improve generalization to the unseen environment which barely achieved by traditional deterministic state tracker. In order to stabilize the learning procedure, a fine-tuning process using policy optimization is proposed. From the experimental results, the proposed AVAST does improve the generalization relative to previous works in vision-and-language navigation task. A significant performance is achieved without requiring any additional exploration in the unseen environment.

To view the proceedings, please [click here](https://aclanthology.org/2022.aacl-main.33/)

Downloads

Next from IJCNLP-AACL 2022

CNN for Modeling Sanskrit Originated Bengali and Hindi Language

Similar lecture

Revealing Hidden Preconditions and Effects of Compound HTN Planning Tasks – A Complexity Analysis