Michael Iber | Searcharxiv

arXiv2024

The Future is Meta: Metadata, Formats and Perspectives towards Interactive and Personalized AV Content

The production of media content has undergone tremendous changes in recent years. Multiple daily content updates are just as common for some platforms as is processing the provided content specifically for their target audiences. Such features are made possible through metadata, which make information accessible by categorizing it. In conjunction with AI-supported tools, metadata are shaping the future of audio-visual content production, distribution and consumption. It allows editors to effectively search through archives like in the Tailored Media Project, broadcasters to provide content that is adapted to users' surroundings like in the ARD Audiothek unterwegs project, or give users the ability to experience audio-visual content from different perspectives like in the ORPHEUS project. Although these projects provide comprehensive insight into the potential of metadata, their integration in existing infrastructures meets several limitations. For example, content-related metadata may initially be generated at some point during the production process but will then be lost at later stages due to current standards and incomplete software implementations. In our contribution, we will discuss requirements and potential approaches and give an outlook on possible fields of application and use-cases.

cs.MM↗

arXiv2024

Open Your Ears and Take a Look: A State-of-the-Art Report on the Integration of Sonification and Visualization

The research communities studying visualization and sonification for data display and analysis share exceptionally similar goals, essentially making data of any kind interpretable to humans. One community does so by using visual representations of data, and the other community employs auditory (non-speech) representations of data. While the two communities have a lot in common, they developed mostly in parallel over the course of the last few decades. With this STAR, we discuss a collection of work that bridges the borders of the two communities, hence a collection of work that aims to integrate the two techniques into one form of audiovisual display, which we argue to be "more than the sum of the two." We introduce and motivate a classification system applicable to such audiovisual displays and categorize a corpus of 57 academic publications that appeared between 2011 and 2023 in categories such as reading level, dataset type, or evaluation system, to mention a few. The corpus also enables a meta-analysis of the field, including regularly occurring design patterns such as type of visualization and sonification techniques, or the use of visual and auditory channels, showing an overall diverse field with different designs. An analysis of a co-author network of the field shows individual teams without many interconnections. The body of work covered in this STAR also relates to three adjacent topics: audiovisual monitoring, accessibility, and audiovisual data art. These three topics are discussed individually in addition to the systematically conducted part of this research. The findings of this report may be used by researchers from both fields to understand the potentials and challenges of such integrated designs while hopefully inspiring them to collaborate with experts from the respective other field.

cs.HC↗