arXiv subjects

Philipp Müller

Publications and source records attributed to Philipp Müller.

At least 37 records · Page 2Linked to original sources

arXiv2022

Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation

Body language is an eye-catching social signal and its automatic analysis can significantly advance artificial intelligence systems to understand and actively participate in social interactions. While computer vision has made impressive progress in low-level tasks like head and body pose estimation, the detection of more subtle behaviors such as gesturing, grooming, or fumbling is not well explored. In this paper we present BBSI, the first set of annotations of complex Bodily Behaviors embedded in continuous Social Interactions in a group setting. Based on previous work in psychology, we manually annotated 26 hours of spontaneous human behavior in the MPIIGroupInteraction dataset with 15 distinct body language classes. We present comprehensive descriptive statistics on the resulting dataset as well as results of annotation quality evaluations. For automatic detection of these behaviors, we adapt the Pyramid Dilated Attention Network (PDAN), a state-of-the-art approach for human action detection. We perform experiments using four variants of spatial-temporal features as input to PDAN: Two-Stream Inflated 3D CNN, Temporal Segment Networks, Temporal Shift Module and Swin Transformer. Results are promising and indicate a great room for improvement in this difficult task. Representing a key piece in the puzzle towards automatic understanding of social behavior, BBSI is fully available to the research community.

cs.CV↗

arXiv2022

HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO

To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications. Over the last years, the number of efficient algorithms and tools for HPO grew substantially. At the same time, the community is still lacking realistic, diverse, computationally cheap, and standardized benchmarks. This is especially the case for multi-fidelity HPO methods. To close this gap, we propose HPOBench, which includes 7 existing and 5 new benchmark families, with a total of more than 100 multi-fidelity benchmark problems. HPOBench allows to run this extendable set of multi-fidelity HPO benchmarks in a reproducible way by isolating and packaging the individual benchmarks in containers. It also provides surrogate and tabular benchmarks for computationally affordable yet statistically sound evaluations. To demonstrate HPOBench's broad compatibility with various optimization tools, as well as its usefulness, we conduct an exemplary large-scale study evaluating 13 optimizers from 6 optimization tools. We provide HPOBench here: https://github.com/automl/HPOBench.

cs.LG↗

arXiv2022

MultiMediate '22: Backchannel Detection and Agreement Estimation in Group Interactions

Backchannels, i.e. short interjections of the listener, serve important meta-conversational purposes like signifying attention or indicating agreement. Despite their key role, automatic analysis of backchannels in group interactions has been largely neglected so far. The MultiMediate challenge addresses, for the first time, the tasks of backchannel detection and agreement estimation from backchannels in group conversations. This paper describes the MultiMediate challenge and presents a novel set of annotations consisting of 7234 backchannel instances for the MPIIGroupInteraction dataset. Each backchannel was additionally annotated with the extent by which it expresses agreement towards the current speaker. In addition to a an analysis of the collected annotations, we present baseline results for both challenge tasks.

cs.HC↗

arXiv2022

Gaze-enhanced Crossmodal Embeddings for Emotion Recognition

Emotional expressions are inherently multimodal -- integrating facial behavior, speech, and gaze -- but their automatic recognition is often limited to a single modality, e.g. speech during a phone call. While previous work proposed crossmodal emotion embeddings to improve monomodal recognition performance, despite its importance, an explicit representation of gaze was not included. We propose a new approach to emotion recognition that incorporates an explicit representation of gaze in a crossmodal emotion embedding framework. We show that our method outperforms the previous state of the art for both audio-only and video-only emotion classification on the popular One-Minute Gradual Emotion Recognition dataset. Furthermore, we report extensive ablation experiments and provide detailed insights into the performance of different state-of-the-art gaze representations and integration strategies. Our results not only underline the importance of gaze for emotion recognition but also demonstrate a practical and highly effective approach to leveraging gaze information for this task.

cs.LG↗

arXiv2022

Towards Analyzing the Bias of News Recommender Systems Using Sentiment and Stance Detection

News recommender systems are used by online news providers to alleviate information overload and to provide personalized content to users. However, algorithmic news curation has been hypothesized to create filter bubbles and to intensify users' selective exposure, potentially increasing their vulnerability to polarized opinions and fake news. In this paper, we show how information on news items' stance and sentiment can be utilized to analyze and quantify the extent to which recommender systems suffer from biases. To that end, we have annotated a German news corpus on the topic of migration using stance detection and sentiment analysis. In an experimental evaluation with four different recommender systems, our results show a slight tendency of all four models for recommending articles with negative sentiments and stances against the topic of refugees and migration. Moreover, we observed a positive correlation between the sentiment and stance bias of the text-based recommenders and the preexisting user bias, which indicates that these systems amplify users' opinions and decrease the diversity of recommended news. The knowledge-aware model appears to be the least prone to such biases, at the cost of predictive accuracy.

cs.IR↗

arXiv2021

Multimodal Integration of Human-Like Attention in Visual Question Answering

Human-like attention as a supervisory signal to guide neural attention has shown significant promise but is currently limited to uni-modal integration - even for inherently multimodal tasks such as visual question answering (VQA). We present the Multimodal Human-like Attention Network (MULAN) - the first method for multimodal integration of human-like attention on image and text during training of VQA models. MULAN integrates attention predictions from two state-of-the-art text and image saliency models into neural self-attention layers of a recent transformer-based VQA model. Through evaluations on the challenging VQAv2 dataset, we show that MULAN achieves a new state-of-the-art performance of 73.98% accuracy on test-std and 73.72% on test-dev and, at the same time, has approximately 80% fewer trainable parameters than prior work. Overall, our work underlines the potential of integrating multimodal human-like and neural attention for VQA

cs.CV↗

arXiv2021

Neural Photofit: Gaze-based Mental Image Reconstruction

We propose a novel method that leverages human fixations to visually decode the image a person has in mind into a photofit (facial composite). Our method combines three neural networks: An encoder, a scoring network, and a decoder. The encoder extracts image features and predicts a neural activation map for each face looked at by a human observer. A neural scoring network compares the human and neural attention and predicts a relevance score for each extracted image feature. Finally, image features are aggregated into a single feature vector as a linear combination of all features weighted by relevance which a decoder decodes into the final photofit. We train the neural scoring network on a novel dataset containing gaze data of 19 participants looking at collages of synthetic faces. We show that our method significantly outperforms a mean baseline predictor and report on a human study that shows that we can decode photofits that are visually plausible and close to the observer's mental image.

cs.CV↗

arXiv2020

Neural Model-based Optimization with Right-Censored Observations

In many fields of study, we only observe lower bounds on the true response value of some experiments. When fitting a regression model to predict the distribution of the outcomes, we cannot simply drop these right-censored observations, but need to properly model them. In this work, we focus on the concept of censored data in the light of model-based optimization where prematurely terminating evaluations (and thus generating right-censored data) is a key factor for efficiency, e.g., when searching for an algorithm configuration that minimizes runtime of the algorithm at hand. Neural networks (NNs) have been demonstrated to work well at the core of model-based optimization procedures and here we extend them to handle these censored observations. We propose (i)~a loss function based on the Tobit model to incorporate censored samples into training and (ii) use an ensemble of networks to model the posterior distribution. To nevertheless be efficient in terms of optimization-overhead, we propose to use Thompson sampling s.t. we only need to train a single NN in each iteration. Our experiments show that our trained regression models achieve a better predictive quality than several baselines and that our approach achieves new state-of-the-art performance for model-based optimization on two optimization problems: minimizing the solution time of a SAT solver and the time-to-accuracy of neural networks.

cs.AI↗

arXiv2019

BOAH: A Tool Suite for Multi-Fidelity Bayesian Optimization & Analysis of Hyperparameters

Hyperparameter optimization and neural architecture search can become prohibitively expensive for regular black-box Bayesian optimization because the training and evaluation of a single model can easily take several hours. To overcome this, we introduce a comprehensive tool suite for effective multi-fidelity Bayesian optimization and the analysis of its runs. The suite, written in Python, provides a simple way to specify complex design spaces, a robust and efficient combination of Bayesian optimization and HyperBand, and a comprehensive analysis of the optimization process and its outcomes.

cs.LG↗

arXiv2019

Emergent Leadership Detection Across Datasets

Automatic detection of emergent leaders in small groups from nonverbal behaviour is a growing research topic in social signal processing but existing methods were evaluated on single datasets -- an unrealistic assumption for real-world applications in which systems are required to also work in settings unseen at training time. It therefore remains unclear whether current methods for emergent leadership detection generalise to similar but new settings and to which extent. To overcome this limitation, we are the first to study a cross-dataset evaluation setting for the emergent leadership detection task. We provide evaluations for within- and cross-dataset prediction using two current datasets (PAVIS and MPIIGroupInteraction), as well as an investigation on the robustness of commonly used feature channels (visual focus of attention, body pose, facial action units, speaking activity) and online prediction in the cross-dataset setting. Our evaluations show that using pose and eye contact based features, cross-dataset prediction is possible with an accuracy of 0.68, as such providing another important piece of the puzzle towards emergent leadership detection in the real world.

cs.HC↗

arXiv2018

Forecasting User Attention During Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors

Visual attention is highly fragmented during mobile interactions, but the erratic nature of attention shifts currently limits attentive user interfaces to adapting after the fact, i.e. after shifts have already happened. We instead study attention forecasting -- the challenging task of predicting users' gaze behaviour (overt visual attention) in the near future. We present a novel long-term dataset of everyday mobile phone interactions, continuously recorded from 20 participants engaged in common activities on a university campus over 4.5 hours each (more than 90 hours in total). We propose a proof-of-concept method that uses device-integrated sensors and body-worn cameras to encode rich information on device usage and users' visual scene. We demonstrate that our method can forecast bidirectional attention shifts and predict whether the primary attentional focus is on the handheld mobile device. We study the impact of different feature sets on performance and discuss the significant potential but also remaining challenges of forecasting user attention during mobile interactions.

cs.HC↗

arXiv2018

Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour

Rapport, the close and harmonious relationship in which interaction partners are "in sync" with each other, was shown to result in smoother social interactions, improved collaboration, and improved interpersonal outcomes. In this work, we are first to investigate automatic prediction of low rapport during natural interactions within small groups. This task is challenging given that rapport only manifests in subtle non-verbal signals that are, in addition, subject to influences of group dynamics as well as inter-personal idiosyncrasies. We record videos of unscripted discussions of three to four people using a multi-view camera system and microphones. We analyse a rich set of non-verbal signals for rapport detection, namely facial expressions, hand motion, gaze, speaker turns, and speech prosody. Using facial features, we can detect low rapport with an average precision of 0.7 (chance level at 0.25), while incorporating prior knowledge of participants' personalities can even achieve early prediction without a drop in performance. We further provide a detailed analysis of different feature sets and the amount of information contained in different temporal segments of the interactions.

cs.CY↗

arXiv2017

Spectral properties of single photons from quantum emitters

Quantum networks require flying qubits that transfer information between the nodes. This may be implemented by means of single atoms (the nodes) that emit and absorb single photons (the flying qubits) and requires full control of photon absorption and emission by the individual emitters. In this work, we theoretically characterize the wave packet of a photon emitted by a single atom undergoing a spontaneous Raman transition in a three-level scheme. We investigate several excitation schemes that are experimentally relevant and discuss control parameters that allow one to tailor the spectrum of the emitted photon wave packet.

quant-ph↗

arXiv2016

Color Change Effect in an Organic-Inorganic Hybrid Material Based on a Porphyrin Diacid

Porphyrinic materials show a range of interesting and useful optical and electrical properties. The less well-known sub-class of porphyrin diacids has been used in this work to construct an ionic hybrid organic-inorganic material in combination with a halogenidometalate anion. The resulting compound, $[H_6TPyP][BiCl_6]_2$ (1) (TPyP = tetra(4-pyridyl)porphyrin) has been obtained via a facile solution based synthesis in single crystalline form. The material exhibits a broad photoluminescence emission band between 650 and 850 nm at room temperature. Single crystals of $[H_6TPyP][BiCl_6]_2$ show a photocurrent in the fA and a much higher dark current in the nA range. They also display an unexpected reversible color change upon wetting with different liquids. This phenomenon has been investigated with optical spectroscopy, SEM, XPS and NEXAFS techniques, showing that a surface-based structural coloration effect is the source of the color change. This stands in contrast to other materials where structural coloration typically has to be introduced through elaborate, multi-step processes or the use of natural templates. Additionally, it underscores the potential of self-assembly of porphyrinic hybrid compounds in the fabrication of materials with unusual optical properties.

cond-mat.mtrl-sci↗

arXiv2016

Programmable atom-photon quantum interface

We present the implementation of a programmable atom-photon quantum interface, employing a single trapped $^{40}$Ca$^+$ ion and single photons. Depending on its mode of operation, the interface serves as a bi-directional atom-photon quantum-state converter, as a source of entangled atom-photon states, or as a quantum frequency converter of single photons. The interface lends itself particularly to interfacing ions with single or entangled photons in hybrid quantum networks.

quant-ph↗

arXiv2015

Doubly-heralded single-photon absorption by a single atom

We report on a single-photon-to-single-atom interface, where a single photon generated by Spontaneous Parametric Down Conversion (SPDC) is absorbed by a single trapped ion. The photon is heralded by its time-correlated partner generated in the SPDC process, while the absorption event is heralded by a single photon emitted in its course. Coincidence detection marks doubly-heralded absorption, enabling photon-to-atom quantum state transfer [1, 2]. Background in the coincidence measurement is strongly suppressed by a new method that discriminates real absorption events from dark count-induced coincidences.

quant-ph↗

arXiv2015

Telecom-heralded single photon absorption by a single atom

We present, characterize, and apply a photonic quantum interface between the near infrared and telecom spectral regions. A singly resonant optical parametric oscillator (OPO) operated below threshold, in combination with external filters, generates high-rate ($>2.5\cdot10^6~{\rm s}^{-1}$) narrowband photon pairs ($\sim 7$ MHz bandwidth); the signal photons are tuned to resonance with an atomic transition in Ca$^+$, while the idler photons are at telecom wavelength. Quantum interface operation is demonstrated through high-rate absorption of single photons by a single trapped ion ($\sim 670~{\rm s}^{-1}$), heralded by coincident telecom photons.

quant-ph↗

arXiv2014

Experimental protocol for high-fidelity heralded photon-to-atom quantum state transfer

A quantum network combines the benefits of quantum systems regarding secure information transmission and calculational speed-up by employing quantum coherence and entanglement to store, transmit, and process information. A promising platform for implementing such a network are atom-based quantum memories and processors, interconnected by photonic quantum channels. A crucial building block in this scenario is the conversion of quantum states between single photons and single atoms through controlled emission and absorption. Here we present an experimental protocol for photon-to-atom quantum state conversion, whereby the polarization state of an absorbed photon is mapped onto the spin state of a single absorbing atom with >95% fidelity, while successful conversion is heralded by a single emitted photon. Heralded high-fidelity conversion without affecting the converted state is a main experimental challenge, in order to make the transferred information reliably available for further operations. We record >80/s successful state transfer events out of 18,000/s repetitions.

quant-ph↗