Spain

This paper describes our system submitted to task 4 of SemEval 2020: Commonsense Validation and Explanation (ComVE) which consists of three sub-tasks. The challenge is to directly validate whether the system can recognize natural language statements that make sense from those that do not and also require to generate reasonable explanation. Based on BERT architecture with the multi-task setting, we propose an effective and interpretable “Explain, Reason and Predict” (ERP) system to solve the three sub-tasks about commonsense: (a) Validation, and (c) Explanation, (b) Reasoning, following the order of the competition. Inspired by cognitive studies of common sense, our system ﬁrst generates a reason or understanding of the sentences and then choose which one statement makes sense, which is achieved by multi-task learning. The rational experiment validates our assumption and boosts performance. During the post-evaluation, our system has reached 92.9% accuracy in subtask A (rank 11), 89.7% accuracy in subtask B (rank 8), and a BLEU score of 12.9 in subtask C (rank 9).

COLING 2020

CommonSense Explanation, Reasoning and Prediction with Multi-task Learning

explain reason and predict

commonsense knowledge

multi-task learning

stages 

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

First established in 1965, the biennial COLING conference is held in diverse parts of the globe and attracts participants from both top-ranked research centers and emerging countries. Today, the most important developments in our field are taking place not only in universities and academic research institutes, but also in industrial research departments and in technological startups. COLING conferences provide opportunities for all these communities to showcase their exciting developments.

COLING, the International Conference on Computational Linguistics, is one of the premier conferences for natural language processing and computational linguistics. Often grouped within the field of artificial intelligence, but actually pre-dating the development of artificial intelligence, advances in computational linguistics and natural language processing are now some of the major drivers behind the use of artificial intelligence for commercial and social applications – for example, on-line search, machine translation and with voice-assisted conversational devices.

poster

Given the semantics of words captured by temporal word embeddings in different time periods, we investigate the use of unsupervised methods to detect when the target word has gained or lost senses.
To this end, we define a new algorithm based on Gaussian Mixture Models to cluster the target similarities computed over the two periods. We compare the proposed approach with a number of similarity-based thresholds. We found that, although the performance of the detection methods varies across the word embedding algorithms, the combination of Gaussian Mixture with Temporal Referencing resulted in our best system.

Gaussian Mixtures Cross Temporal Similarity Clustering

Task 12 of SemEval 2020 consisted of 3 subtasks, namely offensive language identification (Subtask A), categorization of offense type (Subtask B), and offense target identification (Subtask C). This paper presents the results our classifiers obtained for the English language in the 3 subtasks. The classifiers used by us were BERT and BiLSTM. On the test set, our BERT classifier obtained a macro F1 score of 0.90707 for subtask A, and 0.65279 for subtask B. The BiLSTM classifier obtained a macro F1 score of 0.57565 for subtask C. The paper also performs an analysis of the errors made by our classifiers. We conjecture that the presence of a few misleading instances in the dataset is affecting the performance of the classifiers. Our analysis also discusses the need for temporal context and world knowledge to determine the offensiveness of a few comments.

Comparison of BERT and BiLSTM in Detecting Offensive Language

We present a novel rule-based, language independent method for determining lexical entailment relations using semantic representations built from Wiktionary definitions. Combined with a simple WordNet-based method our system achieves top scores on the English and Italian datasets of the Semeval-2020 task “Predicting Multilingual and Cross-lingual (graded) Lexical Entailment” (Glavas et al., 2020). A detailed error analysis of our output uncovers future directions for improving both the semantic parsing method and the inference process on semantic graphs. reported in this document.


Lexical entailment with semantic graphs

We present our submitted systems for SemEval-2020 Task 8 titled "Memotion Analysis". For this task we used a multimodal architecture to determine the emotion expressed in a meme. Our architecture utilizes both textual and visual information present in a meme. To extract image features we experimented with pre-trained VGG-16 and Inception-V3 classifiers and to extract text features we used LSTM and BERT classifiers. Both FastText and GloVe embeddings were experimented with for the LSTM classifier. The best F1 scores our classifier obtained on the official analysis results are 0.3309, 0.4752, and 0.2897 for Task A, B, and C respectively. In our study, we found that combining both textual and visual information expressed in a meme improves the performance of the classifier as opposed to using standalone classifiers that use only text or visual data.

A Multimodal Approach to Detect Offensive, Sarcastic and Humorous Memes

In this paper, we describe an approach for modelling causal reasoning in natural language by detecting counterfactuals in text using multi-head self-attention weights. We use pre-trained transformer models to extract contextual embeddings and self-attention weights from the text. We show the use of convolutional layers to extract task-specific features from these self-attention weights. Further, we describe a fine-tuning approach with a common base model for knowledge sharing between the two closely related sub-tasks for counterfactual detection. We analyze and compare the performance of various transformer models in our experiments. Finally, we perform a qualitative analysis with the multi-head self-attention weights to interpret our models' dynamics.

Modelling Causal Reasoning in Language with Multi-Head Self-Attention Weights Based Counterfactual Detection

- Proposing a domain-tuned contextualized language model for
Offensive Language Detection task.
- Experimenting with different components and configurations (e.g.,
a multi-view SVM) stacked upon BERT models for specific sub-tasks.
- Achieving F1 scores of 91.7% in Sub-task A, 66.5% in Sub-task B,
and 63.2% in Sub-task C, ranking 5th, 6th, and 11th respectively.
- Performing error analysis to show common misclassification errors
made by our model and outlining research directions for future.

Domain-Tuned Contextualized Models for Offensive Language Detection

We describe the system submitted by the SWAGex team to the SemEval-2020 Commonsense Validation and Explanation Task. We used multiple methods on the pre-trained language model BERT (Devlin et al., 2018) for tasks that require the system to recognize sentences against commonsense and justify the reasoning behind this decision. Our best performing model is BERT trained on SWAG fine-tuned for the task. We investigate the ability to transfer commonsense knowledge from SWAG to SemEval-2020 by training a model for the Explanation task with Next Event Prediction data.

Commonsense Explanation as Next Event Prediction

This poster describes the NTUAAILS submission for SemEval 2020 Task 11 Detection of Propaganda Techniques in News Articles. Using deep learning techniques to detect and classify propaganda techniques, we can automatically provide the means to address propaganda and fake news.
In order to achieve this goal, two separate models were created using bidirectional LSTMs and pre-trained word embeddings like ELMo and GloVe. 
At first, the detection sub-task (Span identification) was addressed as Named Entity Recognition problem (NER). The entities consisted of two categories (token-level) propaganda and no-propaganda.
After the detection, the second sub-task's (Technique Classification) goal was to classify the given propaganda fragments in the 14 propaganda classes.
The models achieved rank 28 among 36 teams with F1 score of 0.335 and ranked 25 among 31 teams with 0.463 F1 scores for Span Identification and Technique Classification sub-tasks respectively. Our results indicate that the proposed deep learning models, although relatively simple in architecture and fast to train, achieve satisfactory results in the tasks on hand.

Propaganda Detection and Classification: BiLSTMs and ELMo approach

OffensEval 2: Multilingual Offensive Language Identification in Social Media is Task 12 of SemEval-2020 international workshop on Semantic Evaluation. Our team created a solution dedicated to this task by leveraging two pre-trained state-of-the-art models: BERT and XLNet. The models were fine-tuned for each subtask separately and features taken from their hidden layers were combined and fed into a fully connected neural network. The model using aggregated Transformer features can serve as a powerful tool for offensive language identification problem. Our team was ranked 7th out of 40 in Sub-task C - Offense target identification with 64.727% macro F1-score and 64th out of 85 in Sub-task A - Offensive language identification (89.726% F1-score).

Aggregation of Transformer-based models’ features for offensive language recognition

We consider detection of the span of antecedents and consequents in argumentative prose a structural, grammatical task. Our system comprises a set of stacked Bi-LSTMs trained on two complementary linguistic annotations. We explore the effectiveness of grammatical features (POS and clause type) through ablation. The reported experiments suggest that a multi-task learning approach using this external, grammatical knowledge is useful for detecting the extent of antecedents and consequents and performs nearly as well without the use of word embeddings.

Downloads

Next from COLING 2020

Gaussian Mixtures Cross Temporal Similarity Clustering

Similar lecture

Commonsense Validation and Explanation

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from COLING 2020

Gaussian Mixtures Cross Temporal Similarity Clustering

Similar lecture

Commonsense Validation and Explanation

Downloads