United States

Weakly-Supervised Dense Video Captioning (WSDVC) aims to localize and describe all events of interest in a video without requiring annotations of event boundaries. This setting poses a great challenge in accurately locating the temporal location of event, as the relevant supervision is unavailable. Existing methods rely on explicit alignment constraints between event locations and captions, which involve complex event proposal procedures during both training and inference. To tackle this problem, we propose a novel implicit location-caption alignment paradigm by complementary masking, which simplifies the complex event proposal and localization process while maintaining effectiveness. Specifically, our model comprises two components: a dual-mode video captioning module and a mask generation module. The dual-mode video captioning module captures global event information and generates descriptive captions,
while the mask generation module generates differentiable positive and negative masks for localizing the events. These masks enable the implicit alignment of event locations and captions by ensuring that captions generated from positively and negatively masked videos are complementary, thereby forming a complete video description. In this way, even under weak supervision, the event location and event caption can be aligned implicitly. Extensive experiments on the public datasets demonstrate that our method outperforms existing weakly-supervised methods and achieves competitive results compared to fully-supervised methods. Code will be released.

AAAI 2025

Implicit Location-Caption Alignment via Complementary Masking for Weakly-Supervised Dense Video Captioning

mining of visual multimedia

dmkm

multimodal data

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

ACL-IJCNLP 2021

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

technical paper

We are pleased to announce the Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI-25), which will be held in Philadelphia, Pennsylvania at the Pennsylvania Convention Center from February 25 to March 4, 2025.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

### [Invited Speakers](https://aaai.org/conference/aaai/aaai-25/aaai-25-invited-speakers/)

Register [here](https://aaai.org/conference/aaai/aaai-25/registration/)

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-25 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.



The field of music generation has seen a surge of interest from both academia and industry, with innovative platforms such as Suno, Udio, and SkyMusic earning widespread recognition. However, the challenge of music infilling—modifying specific music segments without reconstructing the entire piece—remains a significant hurdle for both audio-based and symbolic-based models, limiting their adaptability and practicality. In this paper, we address symbolic music infilling by introducing the Collaborative Music Inpainter (CMI), an advanced human-in-the-loop (HITL) model for music infilling. The CMI features the Joint Embedding Predictive Autoregressive Generative Architecture (JEP-AGA), which learns the high-level predictive representations of the masked part that needs to be infilled during the autoregressive generative process, akin to how humans perceive and interpret music. The newly developed Dynamic Interaction Learner (DIL) achieves HITL by iteratively refining the infilled output based on user interactions alone, significantly reducing the interaction cost without requiring further input. Experimental results confirm CMI’s superior performance in music infilling, demonstrating its efficiency in producing high-quality music.

Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling

Knowledge compilation is a method of transforming knowledge into a compressed and tractable form, permitting more efficient operations. For Boolean functions, numerous representations have been proposed to enhance succinctness and tractability. In this paper, we introduce a new representation named structured Decomposable And-Sum Circuit (st-DASC), which employs $\wedge$ and $+$ nodes with signed edges, in place of the standard $\wedge$ and $\vee$ nodes with unsigned edges. Notably, incorporating negative signs permits polytime logical negation. By following the knowledge compilation map, we show that st-DASC is more succinct than the Sentential Decision Diagram (SDD) while still maintaining support for all of the operations on the knowledge compilation map that SDD supports. Furthermore, st-DASC is even more succinct than structured d-DNNF (st-d-DNNF), which itself is more succinct but supports fewer operations than SDD. Accordingly, st-DASC breaks the traditional trade-off between succinctness and tractability.

An and-Sum Circuit with Signed Edges That Is More Succinct than SDD

This paper presents VDAct, a dataset for a Video-grounded Dialogue on Event-driven Activities, alongside VDEval, a session-based context evaluation metric specially designed for the task. Unlike existing datasets, VDAct includes longer and more complex video sequences that depict a variety of event-driven activities that require advanced contextual understanding for accurate response generation. The dataset comprises 3,000 dialogues with over 30,000 question-and-answer pairs, derived from 1,000 videos with diverse activity scenarios. VDAct displays a notably challenging characteristic due to its broad spectrum of activity scenarios and wide range of question types. Empirical studies on state-of-the-art vision foundation models highlight their limitations in addressing certain question types on our dataset. Furthermore, VDEval, which integrates dialogue session history and video content summaries extracted from our supplementary KGs to evaluate individual responses, demonstrates a significantly higher correlation with human assessments on the VDAct dataset than existing evaluation metrics that rely solely on the context of single dialogue turns.

A Video-grounded Dialogue Dataset and Metric for Event-driven Activities

Deep reinforcement learning (DRL) has gained significant attention in autonomous systems, yet its black-box nature and lack of explainability hinder user trust in safety-critical domains such as autonomous driving. Existing experience replay approaches enhance sample efficiency but often fail to capture the internal causality of training data, leading to a convoluted training process that is difficult for humans to explain. In this work, we introduce Experience Replay with Causal Inference (ERCI), an explainable approach that integrates time series representation and causal inference to offer human-aligned explanations for DRL. Specifically, ERCI 1) introduces a novel multivariate time series representation to extract explainable Time Series Causal Factors (TSCF) from experimental data and 2) leverages internal causality in TSCFs with causal inference as a crucial standard for experience replay in DRL training. We evaluate ERCI using multiple baseline algorithms across diverse environments. Results show that ERCI provides human-aligned explanations and further improves sample efficiency through enhanced explainability. Notably, ERCI outperforms other state-of-the-art approaches by 15% in average performance, highlighting its effectiveness and generalizability.

ERCI: An Explainable Experience Replay Approach with Causal Inference for Deep Reinforcement Learning

This study evaluates the usage of OpenAI’s ChatGPT Large Language Model (LLM) as a tool for constructing multiple choice questions for assessing student academic performance through quizzes and exams. We randomly deploy questions constructed with and without use of the LLM tool and gauge the ability of the students to correctly answer, as well as their ability to correctly perceive the difference between human-authored and LLM-authored questions. In determining whether the questions written with the aid of ChatGPT were consistent with the instructor's questions and source text, we computed representative vectors of both the human and ChatGPT questions using SBERT and compared cosine similarity to the course textbook. A non-significant Mann-Whitney U test ($\textit{z}$ = 1.018, $\textit{p}$ = .309) suggests that students were unable to perceive whether questions were written with or without the aid of ChatGPT. However, student scores on LLM-authored questions were almost 9\% lower ($\textit{z}$ = 2.702, $\textit{p}$ $<$ .01). This result may indicate that either the AI questions were more difficult or that the students were more familiar with the instructor’s style of questions. Overall, the study suggests that while there is potential for using LLM tools to aid in the construction of assessments, care must be taken to ensure that the questions are fair, well-composed, and relevant to the course material.

LLMs in the Classroom: Outcomes and Perceptions of Questions Written with the Aid of AI

This paper addresses the problem of preference learning, which aims to align robot behaviors through learning user-specific preferences (e.g., “good pull-over location”) from visual demonstrations. Despite its similarity to learning factual concepts (e.g., “red cube”), preference learning is a fundamentally harder problem due to its subjective nature and the paucity of person-specific training data. We address this problem using a novel framework called Synapse, which is a neuro-symbolic approach designed to efficiently learn preferential concepts from limited data. Synapse represents preferences as neuro-symbolic programs -- facilitating inspection of individual parts for alignment -- in a domain-specific language (DSL) that operates over images and leverages a novel combination of visual parsing, large language models, and program synthesis to learn programs representing individual preferences. We perform extensive evaluations on various preferential concepts as well as a small user case study demonstrating the ability of Synapse to align well with dissimilar user preferences. Our method significantly outperforms baselines, especially in out-of-distribution generalization. We show the importance of the design choices in the framework through multiple ablation studies.

SYNAPSE: SYmbolic Neural-Aided Preference Synthesis Engine

Researchers, policymakers, and developers of artificial intelligence (AI) are actively collaborating to establish trustworthy AI standards that align with broader societal values, particularly in the context of large language models (LLMs). However, the critical discourse on bridging the vast knowledge gap between experts who shape and implement standards for LLMs and end-users whose values are at stake remains largely unaddressed. Taking a ‘bottom-up’ approach and using a mixed-method approach, we first conducted interviews (N = 12) to engage with end-users’ perceptions of normative standards in the context of LLMs. We thereby identified 68 criteria that end-users’ consider when evaluating value congruence. Second, we conducted an online survey (N = 379) to further investigate how end-users prioritise these standards and the identified criteria in conversational LLM-based applications. Findings reveal a clear preference for governance, monitoring, and transparent information provision over other criteria and highlight the necessity of non-technical complements to technical solutions for bridging the knowledge gap. We discuss end-user information requirements and outline actionable steps to effectively communicate trustworthy AI standards.

Bridging the Knowledge Gap: Understanding User Expectations for Trustworthy LLM Standards

Despite significant progress has been made in image deraining, most existing methods are limited to handling only a single type of rain degradation or a specific pattern of rain. However, real-world rain scenarios tend to contain diverse rainy patterns due to variations in the rainfall process and lighting conditions. To address this dilemma and advance this field, we introduce a new task: Universal Rainy Image Restoration (URIR), which aims to handle multiple types of rain degradation on a single model. To benchmark this task, we construct a high-quality dataset called URIR-8K, which contains four patterns: rain streak, raindrop, rain accumulation and nighttime rain. Building upon this dataset, we present a comprehensive study on existing approaches by evaluating their universal deraining capabilities and their effect on downstream object detection task. In addition, we design a multi-scale vision Mamba as a baseline model, leveraging the benefits of multi-scale learning for its robustness to diverse rain appearances. Unlike existing methods that use fixed-scale scanning for feature extraction, we employ a multi-scale 2D scanning technique to better help image restoration in the richer scale space. Extensive experimental analysis demonstrates the potential of our proposed task and the effectiveness of the baseline model. The code and dataset will be available.

Towards Universal Rainy Image Restoration: Benchmark and Baseline

Backdoor attacks have posed a serious threat in machine learning models, wherein adversaries can poison training samples with maliciously crafted triggers to compromise the victim model. Advanced backdoor attack methods have focused on selectively poisoning more vulnerable training samples, achieving a higher attack success rate (ASR). However, we found that when the manipulation strength of the trigger is constrained to a very small value for imperceptible attacks, they suffer from extremely uneven class-wise ASR due to the unequal selection of instances per class. To solve this issue, we propose a novel backdoor attack method based on \underline{I}nfluence-based \underline{F}air \underline{S}election (IFS), including two objectives: 1) selecting samples that significantly contribute to ASR and 2) ensuring class balance during the selection process. Specifically, we adapt Influence Functions, a classic technique in robust statistics, to evaluate the influence of trigger-embedded training samples on ASR. In this case, training samples contributing to reducing the backdoored test risk could possess higher influence scores. Further, a group-based pruning strategy is designed to avoid calculating the influence on ASR for all training samples, thereby significantly reducing the computational cost. Then, based on the influence score, we design an adaptive thresholding scheme to dynamically select samples with higher influence while maintaining class balance. Extensive experiments on four datasets verify the effectiveness of IFS compared with advanced methods.

Influence-Based Fair Selection for Sample-Discriminative Backdoor Attack

Abstract
Reinforcement learning (RL), particularly its combination with deep neural networks referred to as
deep RL (DRL), has shown tremendous promise across a wide range of applications, suggesting its po-
tential for enabling the development of sophisticated robotic behaviors. Robotics problems, however,
pose fundamental difficulties for the application of RL, stemming from the complexity and cost of
interacting with the physical world. These challenges notwithstanding, recent advances have enabled
DRL to succeed at some real-world robotic tasks. However, state-of-the-art DRL solutions’ maturity
varies significantly across robotic applications. In this talk, I will review the current progress of DRL
in real-world robotic applications based on our recent survey paper (with Tang, Abbatematteo, Hu,
Chandra, and Martı́n-Martı́n), with a particular focus on evaluating the real-world successes achieved
with DRL in realizing several key robotic competencies, including locomotion, navigation, stationary
manipulation, mobile manipulation, human-robot interaction, and multi-robot interaction. The anal-
ysis aims to identify the key factors underlying those exciting successes, reveal underexplored areas,
and provide an overall characterization of the status of DRL in robotics. I will also highlight several
important avenues for future work, emphasizing the need for stable and sample-efficient real-world
RL paradigms, holistic approaches for discovering and integrating various competencies to tackle
complex long-horizon, open-world tasks, and principled development and evaluation procedures. The
talk is designed to offer insights for RL practitioners and roboticists toward harnessing RL’s power
to create generally capable real-world robotic systems.

Downloads

Next from AAAI 2025

Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling

Similar lecture

Hierarchical Context-aware Network for Dense Video Event Captioning

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAAI 2025

Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling

Similar lecture

Hierarchical Context-aware Network for Dense Video Event Captioning

Downloads