arXiv subjects

Mohammad Soleymani

Publications and source records attributed to Mohammad Soleymani.

68 records · Page 4Linked to original sources

arXiv2022

A Pre-trained Audio-Visual Transformer for Emotion Recognition

In this paper, we introduce a pretrained audio-visual Transformer trained on more than 500k utterances from nearly 4000 celebrities from the VoxCeleb2 dataset for human behavior understanding. The model aims to capture and extract useful information from the interactions between human facial and auditory behaviors, with application in emotion recognition. We evaluate the model performance on two datasets, namely CREMAD-D (emotion classification) and MSP-IMPROV (continuous emotion regression). Experimental results show that fine-tuning the pre-trained model helps improving emotion classification accuracy by 5-7% and Concordance Correlation Coefficients (CCC) in continuous emotion recognition by 0.03-0.09 compared to the same model trained from scratch. We also demonstrate the robustness of finetuning the pre-trained model in a low-resource setting. With only 10% of the original training set provided, fine-tuning the pre-trained model can lead to at least 10% better emotion recognition accuracy and a CCC score improvement by at least 0.1 for continuous emotion recognition.

cs.MM↗

arXiv2021

Speaker Turn Modeling for Dialogue Act Classification

Dialogue Act (DA) classification is the task of classifying utterances with respect to the function they serve in a dialogue. Existing approaches to DA classification model utterances without incorporating the turn changes among speakers throughout the dialogue, therefore treating it no different than non-interactive written text. In this paper, we propose to integrate the turn changes in conversations among speakers when modeling DAs. Specifically, we learn conversation-invariant speaker turn embeddings to represent the speaker turns in a conversation; the learned speaker turn embeddings are then merged with the utterance embeddings for the downstream task of DA classification. With this simple yet effective mechanism, our model is able to capture the semantics from the dialogue content while accounting for different speaker turns in a conversation. Validation on three benchmark public datasets demonstrates superior performance of our model.

cs.CL↗

arXiv2021

Modeling Dynamics of Facial Behavior for Mental Health Assessment

Facial action unit (FAU) intensities are popular descriptors for the analysis of facial behavior. However, FAUs are sparsely represented when only a few are activated at a time. In this study, we explore the possibility of representing the dynamics of facial expressions by adopting algorithms used for word representation in natural language processing. Specifically, we perform clustering on a large dataset of temporal facial expressions with 5.3M frames before applying the Global Vector representation (GloVe) algorithm to learn the embeddings of the facial clusters. We evaluate the usefulness of our learned representations on two downstream tasks: schizophrenia symptom estimation and depression severity regression. These experimental results show the potential effectiveness of our approach for improving the assessment of mental health symptoms over baseline models that use FAU intensities alone.

cs.CV↗

arXiv2020

Towards A Friendly Online Community: An Unsupervised Style Transfer Framework for Profanity Redaction

Offensive and abusive language is a pressing problem on social media platforms. In this work, we propose a method for transforming offensive comments, statements containing profanity or offensive language, into non-offensive ones. We design a RETRIEVE, GENERATE and EDIT unsupervised style transfer pipeline to redact the offensive comments in a word-restricted manner while maintaining a high level of fluency and preserving the content of the original text. We extensively evaluate our method's performance and compare it to previous style transfer models using both automatic metrics and human evaluations. Experimental results show that our method outperforms other models on human evaluations and is the only approach that consistently performs well on all automatic evaluation metrics.

cs.CL↗

arXiv2020

Improper Gaussian Signaling for the $K$-user MIMO Interference Channels with Hardware Impairments

This paper investigates the performance of improper Gaussian signaling (IGS) for the $K$-user multiple-input, multiple-output (MIMO) interference channel (IC) with hardware impairments (HWI). HWI may arise due to imperfections in the devices like I/Q imbalance, phase noise, etc. With I/Q imbalance, the received signal is a widely linear transformation of the transmitted signal and noise. Thus, the effective noise at the receivers becomes improper, which means that its real and imaginary parts are correlated and/or have unequal powers. IGS can improve system performance with improper noise and/or improper interference. In this paper, we study the benefits of IGS for this scenario in terms of two performance metrics: achievable rate and energy efficiency (EE). We consider the rate region, the sum-rate, the EE region and the global EE optimization problems to fully evaluate the IGS performance. To solve these non-convex problems, we employ an optimization framework based on majorization-minimization algorithms, which allow us to obtain a stationary point of any optimization problem in which either the objective function and/or constraints are linear functions of rates. Our numerical results show that IGS can significantly improve the performance of the $K$-user MIMO IC with HWI and I/Q imbalance, where its benefits increase with the number of users, $K$, and the imbalance level, and decrease with the number of antennas.

eess.SP↗

arXiv2019

Affective Computing for Large-Scale Heterogeneous Multimedia Data: A Survey

The wide popularity of digital photography and social networks has generated a rapidly growing volume of multimedia data (i.e., image, music, and video), resulting in a great demand for managing, retrieving, and understanding these data. Affective computing (AC) of these data can help to understand human behaviors and enable wide applications. In this article, we survey the state-of-the-art AC technologies comprehensively for large-scale heterogeneous multimedia data. We begin this survey by introducing the typical emotion representation models from psychology that are widely employed in AC. We briefly describe the available datasets for evaluating AC algorithms. We then summarize and compare the representative methods on AC of different multimedia types, i.e., images, music, videos, and multimodal data, with the focus on both handcrafted features-based methods and deep learning methods. Finally, we discuss some challenges and future directions for multimedia affective computing.

cs.MM↗

arXiv2019

Improper Signaling for SISO Two-user Interference Channels with Additive Asymmetric Hardware Distortion

Hardware non-idealities are among the main performance restrictions for upcoming wireless communication systems. Asymmetric hardware distortions (HWD) happen when the impairments of the I/Q branches are correlated or imbalanced, which in turn generate improper additive interference at the receiver side. When the interference is improper, as well as in other interference-limited scenarios, improper Gaussian signaling (IGS) has been shown to provide rate and/or power efficiency benefits. In this paper, we investigate the rate benefits of IGS in a two-user interference channel (IC) with additive asymmetric HWD when interference is treated as noise. We propose two iterative algorithms to optimize the parameters of the improper transmit signals. We first rewrite the rate region as an pseudosignal-to-interference-plus-noise-ratio (PSINR) region and employ majorization minimization and fractional programming to find a suboptimal solution for the achievable user rates. Then, we propose a simplified algorithm based on a separate optimization of the powers and complementary variances of the users, which exhibits lower computational complexity. We show that IGS can improve the performance of the two-user IC with additive HWD. Our proposed algorithms outperform proper Gaussian signaling and competing IGS algorithms in the literature that do not consider asymmetric HWD.

eess.SP↗

arXiv2019

Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval

Visual-semantic embedding aims to find a shared latent space where related visual and textual instances are close to each other. Most current methods learn injective embedding functions that map an instance to a single point in the shared space. Unfortunately, injective embedding cannot effectively handle polysemous instances with multiple possible meanings; at best, it would find an average representation of different meanings. This hinders its use in real-world scenarios where individual instances and their cross-modal associations are often ambiguous. In this work, we introduce Polysemous Instance Embedding Networks (PIE-Nets) that compute multiple and diverse representations of an instance by combining global context with locally-guided features via multi-head self-attention and residual learning. To learn visual-semantic embedding, we tie-up two PIE-Nets and optimize them jointly in the multiple instance learning framework. Most existing work on cross-modal retrieval focuses on image-text data. Here, we also tackle a more challenging case of video-text retrieval. To facilitate further research in video-text retrieval, we release a new dataset of 50K video-sentence pairs collected from social media, dubbed MRW (my reaction when). We demonstrate our approach on both image-text and video-text retrieval scenarios using MS-COCO, TGIF, and our new MRW dataset.

cs.CV↗

arXiv2019

AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition

The Audio/Visual Emotion Challenge and Workshop (AVEC 2019) "State-of-Mind, Detecting Depression with AI, and Cross-cultural Affect Recognition" is the ninth competition event aimed at the comparison of multimedia processing and machine learning methods for automatic audiovisual health and emotion analysis, with all participants competing strictly under the same conditions. The goal of the Challenge is to provide a common benchmark test set for multimodal information processing and to bring together the health and emotion recognition communities, as well as the audiovisual processing communities, to compare the relative merits of various approaches to health and emotion recognition from real-life data. This paper presents the major novelties introduced this year, the challenge guidelines, the data used, and the performance of the baseline systems on the three proposed tasks: state-of-mind recognition, depression assessment with AI, and cross-cultural affect sensing, respectively.

cs.HC↗

arXiv2019

Robust Improper Signaling for Two-user SISO Interference Channels

It has been shown that improper Gaussian signaling (IGS) can improve the performance of wireless interference-limited systems when perfect channel state information (CSI) is available. In this paper, we investigate the robustness of IGS against imperfect CSI on the transmitter side in a two-user single-input single-output (SISO) interference channel (IC) as well as in a SISO Z-IC, when interference is treated as noise. We assume that the true channel coefficients belong to a known region around the channel estimates, which we call the uncertainty region. Following a worst-case robustness approach, we study the rate-region boundary of the IC for the worst channel in the uncertainty region. For the two-user IC, we derive a robust design in closed-form, which is independent of the phase of the channels by allowing only one of the users to transmit IGS. For the Z-IC, we provide a closed-form design for the transmission parameters by considering an enlarged uncertainty region and allowing both users to employ IGS. In both cases, the IGS-based designs are ensured to perform no worse than proper Gaussian signaling. Furthermore, we show, through numerical examples, that the proposed robust designs significantly outperform non-robust solutions.

eess.SP↗

arXiv2018

A data-driven approach to mid-level perceptual musical feature modeling

Musical features and descriptors could be coarsely divided into three levels of complexity. The bottom level contains the basic building blocks of music, e.g., chords, beats and timbre. The middle level contains concepts that emerge from combining the basic blocks: tonal and rhythmic stability, harmonic and rhythmic complexity, etc. High-level descriptors (genre, mood, expressive style) are usually modeled using the lower level ones. The features belonging to the middle level can both improve automatic recognition of high-level descriptors, and provide new music retrieval possibilities. Mid-level features are subjective and usually lack clear definitions. However, they are very important for human perception of music, and on some of them people can reach high agreement, even though defining them and therefore, designing a hand-crafted feature extractor for them can be difficult. In this paper, we derive the mid-level descriptors from data. We collect and release a dataset\footnote{https://osf.io/5aupt/} of 5000 songs annotated by musicians with seven mid-level descriptors, namely, melodiousness, tonal and rhythmic stability, modality, rhythmic complexity, dissonance and articulation. We then compare several approaches to predicting these descriptors from spectrograms using deep-learning. We also demonstrate the usefulness of these mid-level features using music emotion recognition as an application.

cs.SD↗

arXiv2018

Cross-Modal Retrieval with Implicit Concept Association

Traditional cross-modal retrieval assumes explicit association of concepts across modalities, where there is no ambiguity in how the concepts are linked to each other, e.g., when we do the image search with a query "dogs", we expect to see dog images. In this paper, we consider a different setting for cross-modal retrieval where data from different modalities are implicitly linked via concepts that must be inferred by high-level reasoning; we call this setting implicit concept association. To foster future research in this setting, we present a new dataset containing 47K pairs of animated GIFs and sentences crawled from the web, in which the GIFs depict physical or emotional reactions to the scenarios described in the text (called "reaction GIFs"). We report on a user study showing that, despite the presence of implicit concept association, humans are able to identify video-sentence pairs with matching concepts, suggesting the feasibility of our task. Furthermore, we propose a novel visual-semantic embedding network based on multiple instance learning. Unlike traditional approaches, we compute multiple embeddings from each modality, each representing different concepts, and measure their similarity by considering all possible combinations of visual-semantic embeddings in the framework of multiple instance learning. We evaluate our approach on two video-sentence datasets with explicit and implicit concept association and report competitive results compared to existing approaches on cross-modal retrieval.

cs.CV↗

arXiv2016

Detecting Cognitive Appraisals from Facial Expressions for Interest Recognition

Interest makes one hold her attention on the object of interest. Automatic recognition of interest has numerous applications in human-computer interaction. In this paper, we study the facial expressions associated with interest and its underlying and closely related components, namely, curiosity, coping potential, novelty and complexity. To this end, we conducted an experiment in which participants watched images and micro-videos while a front-facing camera recorded their expressions. After watching each item they self-reported their level of interest, curiosity, coping potential and perceived novelty and complexity. Using an automated method, we tracked facial action units (AU) and studied the relationship between the presence of facial movements with interest and its related components. We then tracked the facial landmarks, e.g., corners of lips, and extracted features from each response. We trained random forests regression models to detect the level of interest, curiosity, and appraisals. We found a large difference between the way people report and react to interesting visual content. The expressions in response to images and micro-videos were not always pronounced depending on the participants. This makes the direct detection of interest from facial expressions a challenging problem. With this work, for the first time, we demonstrate the feasibility of detecting cognitive appraisals from facial expressions which will open the door for appraisal-driven emotion recognition methods.

cs.HC↗

arXiv2014

Corpus Development for Affective Video Indexing

Affective video indexing is the area of research that develops techniques to automatically generate descriptions of video content that encode the emotional reactions which the video content evokes in viewers. This paper provides a set of corpus development guidelines based on state-of-the-art practice intended to support researchers in this field. Affective descriptions can be used for video search and browsing systems offering users affective perspectives. The paper is motivated by the observation that affective video indexing has yet to fully profit from the standard corpora (data sets) that have benefited conventional forms of video indexing. Affective video indexing faces unique challenges, since viewer-reported affective reactions are difficult to assess. Moreover affect assessment efforts must be carefully designed in order to both cover the types of affective responses that video content evokes in viewers and also capture the stable and consistent aspects of these responses. We first present background information on affect and multimedia and related work on affective multimedia indexing, including existing corpora. Three dimensions emerge as critical for affective video corpora, and form the basis for our proposed guidelines: the context of viewer response, personal variation among viewers, and the effectiveness and efficiency of corpus creation. Finally, we present examples of three recent corpora and discuss how these corpora make progressive steps towards fulfilling the guidelines.

cs.MM↗