arXiv subjects

André Kaup

Publications and source records attributed to André Kaup.

At least 127 records · Page 7Linked to original sources

arXiv2022

Modeling of Energy Consumption and Streaming Video QoE using a Crowdsourcing Dataset

In the past decade, we have witnessed an enormous growth in the demand for online video services. Recent studies estimate that nowadays, more than 1% of the global greenhouse gas emissions can be attributed to the production and use of devices performing online video tasks. As such, research on the true power consumption of devices and their energy efficiency during video streaming is highly important for a sustainable use of this technology. At the same time, over-the-top providers strive to offer high-quality streaming experiences to satisfy user expectations. Here, energy consumption and QoE partly depend on the same system parameters. Hence, a joint view is needed for their evaluation. In this paper, we perform a first analysis of both end-user power efficiency and Quality of Experience of a video streaming service. We take a crowdsourced dataset comprising 447,000 streaming events from YouTube and estimate both the power consumption and perceived quality. The power consumption is modeled based on previous work which we extended towards predicting the power usage of different devices and codecs. The user-perceived QoE is estimated using a standardized model. Our results indicate that an intelligent choice of streaming parameters can optimize both the QoE and the power efficiency of the end user device. Further, the paper discusses limitations of the approach and identifies directions for future research.

eess.IV↗

arXiv2022

Boosting Neural Image Compression for Machines Using Latent Space Masking

Today, many image coding scenarios do not have a human as final intended user, but rather a machine fulfilling computer vision tasks on the decoded image. Thereby, the primary goal is not to keep visual quality but maintain the task accuracy of the machine for a given bitrate. Due to the tremendous progress of deep neural networks setting benchmarking results, mostly neural networks are employed to solve the analysis tasks at the decoder side. Moreover, neural networks have also found their way into the field of image compression recently. These two developments allow for an end-to-end training of the neural compression network for an analysis network as information sink. Therefore, we first roll out such a training with a task-specific loss to enhance the coding performance of neural compression networks. Compared to the standard VVC, 41.4% of bitrate are saved by this method for Mask R-CNN as analysis network on the uncompressed Cityscapes dataset. As a main contribution, we propose LSMnet, a network that runs in parallel to the encoder network and masks out elements of the latent space that are presumably not required for the analysis network. By this approach, additional 27.3% of bitrate are saved compared to the basic neural compression network optimized with the task loss. In addition, we are the first to utilize a feature-based distortion in the training loss within the context of machine-to-machine communication, which allows for a training without annotated data. We provide extensive analyses on the Cityscapes dataset including cross-evaluation with different analysis networks and present exemplary visual results. Inference code and pre-trained models are published at https://github.com/FAU-LMS/NCN_for_M2M.

eess.IV↗

arXiv2022

Sweet Streams are Made of This: The System Engineer's View on Energy Efficiency in Video Communications

In recent years, the global use of online video services has increased rapidly. Today, a manifold of applications, such as video streaming, video conferencing, live broadcasting, and social networks, make use of this technology. A recent study found that the development and the success of these services had as a consequence that, nowadays, more than 1% of the global greenhouse-gas emissions are related to online video, with growth rates close to 10% per year. This article reviews the latest findings concerning energy consumption of online video from the system engineer's perspective, where the system engineer is the designer and operator of a typical online video service. We discuss all relevant energy sinks, highlight dependencies with quality-of-service variables as well as video properties, review energy consumption models for different devices from the literature, and aggregate these existing models into a global model for the overall energy consumption of a generic online video service. Analyzing this model and its implications, we find that end-user devices and video encoding have the largest potential for energy savings. Finally, we provide an overview of recent advances in energy efficiency improvement for video streaming and propose future research directions for energy-efficient video streaming services.

eess.IV↗

arXiv2022

Increasing the Accuracy of a Neural Network Using Frequency Selective Mesh-to-Grid Resampling

Neural networks are widely used for almost any task of recognizing image content. Even though much effort has been put into investigating efficient network architectures, optimizers, and training strategies, the influence of image interpolation on the performance of neural networks is not well studied. Furthermore, research has shown that neural networks are often sensitive to minor changes in the input image leading to drastic drops of their performance. Therefore, we propose the use of keypoint agnostic frequency selective mesh-to-grid resampling (FSMR) for the processing of input data for neural networks in this paper. This model-based interpolation method already showed that it is capable of outperforming common interpolation methods in terms of PSNR. Using an extensive experimental evaluation we show that depending on the network architecture and classification task the application of FSMR during training aids the learning process. Furthermore, we show that the usage of FSMR in the application phase is beneficial. The classification accuracy can be increased by up to 4.31 percentage points for ResNet50 and the Oxflower17 dataset.

cs.CV↗

arXiv2022

Synthesizing Annotated Image and Video Data Using a Rendering-Based Pipeline for Improved License Plate Recognition

An insufficient number of training samples is a common problem in neural network applications. While data augmentation methods require at least a minimum number of samples, we propose a novel, rendering-based pipeline for synthesizing annotated data sets. Our method does not modify existing samples but synthesizes entirely new samples. The proposed rendering-based pipeline is capable of generating and annotating synthetic and partly-real image and video data in a fully automatic procedure. Moreover, the pipeline can aid the acquisition of real data. The proposed pipeline is based on a rendering process. This process generates synthetic data. Partly-real data bring the synthetic sequences closer to reality by incorporating real cameras during the acquisition process. The benefits of the proposed data generation pipeline, especially for machine learning scenarios with limited available training data, are demonstrated by an extensive experimental validation in the context of automatic license plate recognition. The experiments demonstrate a significant reduction of the character error rate and miss rate from 73.74% and 100% to 14.11% and 41.27% respectively, compared to an OCR algorithm trained on a real data set solely. These improvements are achieved by training the algorithm on synthesized data solely. When additionally incorporating real data, the error rates can be decreased further. Thereby, the character error rate and miss rate can be reduced to 11.90% and 39.88% respectively. All data used during the experiments as well as the proposed rendering-based pipeline for the automated data generation is made publicly available under (URL will be revealed upon publication).

cs.CV↗

arXiv2022

3D Rendering Framework for Data Augmentation in Optical Character Recognition

In this paper, we propose a data augmentation framework for Optical Character Recognition (OCR). The proposed framework is able to synthesize new viewing angles and illumination scenarios, effectively enriching any available OCR dataset. Its modular structure allows to be modified to match individual user requirements. The framework enables to comfortably scale the enlargement factor of the available dataset. Furthermore, the proposed method is not restricted to single frame OCR but can also be applied to video OCR. We demonstrate the performance of our framework by augmenting a 15% subset of the common Brno Mobile OCR dataset. Our proposed framework is capable of leveraging the performance of OCR applications especially for small datasets. Applying the proposed method, improvements of up to 2.79 percentage points in terms of Character Error Rate (CER), and up to 7.88 percentage points in terms of Word Error Rate (WER) are achieved on the subset. Especially the recognition of challenging text lines can be improved. The CER may be decreased by up to 14.92 percentage points and the WER by up to 18.19 percentage points for this class. Moreover, we are able to achieve smaller error rates when training on the 15% subset augmented with the proposed method than on the original non-augmented full dataset.

cs.CV↗

arXiv2022

Quality Assurance of Weld Seams Using Laser Triangulation Imaging and Deep Neural Networks

In this paper, a novel optical inspection system is presented that is directly suitable for Industry 4.0 and the implementation on IoT-devices controlling the manufacturing process. The proposed system is capable of distinguishing between erroneous and faultless weld seams, without explicitly defining measurement criteria . The developed system uses a deep neural network based classifier for the class prediction. A weld seam dataset was acquired and labelled by an expert committee. Thereby, the visual impression and assessment of the experts is learnt accurately. In the scope of this paper laser triangulation images are used. Due to their special characteristics, the images must be pre-processed to enable the use of a deep neural network. Furthermore, two different approaches are investigated to enable an inspection of differently sized weld seams. Both approaches yield very high classification accuracies of up to 96.88\%, which is competitive to current state of the art optical inspection systems. Moreover, the proposed system enables a higher flexibility and an increased robustness towards systematic errors and environmental conditions due to its ability to generalize. A further benefit of the proposed system is the fast decision process enabling the usage directly within the manufacturing line. Furthermore, standard hardware is used throughout the whole presented work, keeping the roll-out costs for the proposed system as low as possible.

eess.IV↗

arXiv2022

Decoding Energy Modeling For Versatile Video Coding

In previous research, it was shown that the software decoding energy demand of High Efficiency Video Coding (HEVC) can be reduced by 15$\%$ by using a decoding-energy-rate-distortion optimization algorithm. To achieve this, the energy demand of the decoder has to be modeled by a bit stream feature-based model with sufficiently high accuracy. Therefore, we propose two bit stream feature-based models for the upcoming Versatile Video Coding (VVC) standard. The newly introduced models are compared with models from literature, which are used for HEVC. An evaluation of the proposed models reveals that the mean estimation error is similar to the results of the literature and yields an estimation error of 1.85% with 10-fold cross-validation.

eess.IV↗

arXiv2022

A Comparative Analysis of the Time and Energy Demand of Versatile Video Coding and High Efficiency Video Coding Reference Decoders

This paper investigates the decoding energy and decoding time demand of VTM-7.0 in relation to HM-16.20. We present the first detailed comparison of two video codecs in terms of software decoder energy consumption. The evaluation shows that the energy demand of the VTM decoder is increased significantly compared to HM and that the increase depends on the coding configuration. For the coding configuration randomaccess, we find that the decoding energy is increased by over 80% at a decoding time increase of over 70%. Furthermore, results indicate that the energy demand increases by up to 207% when Single Instruction Multiple Data (SIMD) instructions are disabled, which corresponds to the HM implementation style. By measurements, it is revealed that the coding tools MIP, AMVR, TPM, LFNST, and MTS increase the energy efficiency of the decoder. Furthermore, we propose a new coding configuration based on our analysis, which reduces the energy demand of the VTM decoder by over 17% on average.

eess.IV↗

arXiv2022

Extending Video Decoding Energy Models for 360° and HDR Video Formats in HEVC

Research has shown that decoder energy models are helpful tools for improving the energy efficiency in video playback applications. For example, an accurate feature-based bit stream model can reduce the energy consumption of the decoding process. However, until now only sequences of the SDR video format were investigated. Therefore, this paper shows that the decoding energy of HEVC-coded bit streams can be estimated precisely for different video formats and coding bit depths. Therefore, we compare a state-of-the-art model from the literature with a proposed model. We show that bit streams of the 360°, HDR, and fisheye video format can be estimated with a mean estimation error lower than 3.88% if the setups have the same coding bit depth. Furthermore, it is shown that on average, the energy demand for the decoding of bit streams with a bit depth of 10-bit is 55% higher than with 8-bit.

eess.IV↗

arXiv2022

Advanced Design Space Exploration for Joint Energy and Quality Optimization for VVC

In recent studies, it could be shown that the energy demand of Versatile Video Coding (VVC) decoders can be twice as high as comparable High Efficiency Video Coding (HEVC) decoders. A significant part of this increase in complexity is attributed to the usage of new coding tools. By using a design space exploration algorithm, it was shown that the energy demand of VVC-coded sequences could be reduced if different coding tool profiles were used for the encoding process. This work extends the algorithm with several optimization strategies, methodological adjustments to optimize perceptual quality, and a new minimization criterion. As a result, we significantly improve the Pareto front, and the rate-distortion and energy efficiency of the state-of-the-art design space exploration. Therefore, we show an energy demand reduction of up to 47% with less than 30% additional bit rate, or a reduction of over 35% with approximately 6% additional bit rate.

eess.IV↗

arXiv2022

Matched Quality Evaluation of Temporally Downsampled Videos with Non-Integer Factors

Recent research has shown that temporal downsampling of high-frame-rate sequences can be exploited to improve the rate-distortion performance in video coding. However, until now, research only targeted downsampling factors of powers of two, which greatly restricts the potential applicability of temporal downsampling. A major reason is that traditional, objective quality metrics such as peak signal-to-noise ratio or more recent approaches, which try to mimic subjective quality, can only be evaluated between two sequences whose frame rate ratio is an integer value. To relieve this problem, we propose a quality evaluation method that allows calculating the distortion between two sequences whose frame rate ratio is fractional. The proposed method can be applied to any full-reference quality metric.

eess.IV↗

arXiv2022

Hyperspectral Image Reconstruction from Multispectral Images Using Non-Local Filtering

Using light spectra is an essential element in many applications, for example, in material classification. Often this information is acquired by using a hyperspectral camera. Unfortunately, these cameras have some major disadvantages like not being able to record videos. Therefore, multispectral cameras with wide-band filters are used, which are much cheaper and are often able to capture videos. However, using multispectral cameras requires an additional reconstruction step to yield spectral information. Usually, this reconstruction step has to be done in the presence of imaging noise, which degrades the reconstructed spectra severely. Typically, same or similar pixels are found across the image with the advantage of having independent noise. In contrast to state-of-the-art spectral reconstruction methods which only exploit neighboring pixels by block-based processing, this paper introduces non-local filtering in spectral reconstruction. First, a block-matching procedure finds similar non-local multispectral blocks. Thereafter, the hyperspectral pixels are reconstructed by filtering the matched multispectral pixels collaboratively using a reconstruction Wiener filter. The proposed novel procedure even works under very strong noise. The method is able to lower the spectral angle up to 18% and increase the peak signal-to-noise-ratio up to 1.1dB in noisy scenarios compared to state-of-the-art methods. Moreover, the visual results are much more appealing.

eess.IV↗

arXiv2022

Spatio-spectral Image Reconstruction Using Non-local Filtering

In many image processing tasks it occurs that pixels or blocks of pixels are missing or lost in only some channels. For example during defective transmissions of RGB images, it may happen that one or more blocks in one color channel are lost. Nearly all modern applications in image processing and transmission use at least three color channels, some of the applications employ even more bands, for example in the infrared and ultraviolet area of the light spectrum. Typically, only some pixels and blocks in a subset of color channels are distorted. Thus, other channels can be used to reconstruct the missing pixels, which is called spatio-spectral reconstruction. Current state-of-the-art methods purely rely on the local neighborhood, which works well for homogeneous regions. However, in high-frequency regions like edges or textures, these methods fail to properly model the relationship between color bands. Hence, this paper introduces non-local filtering for building a linear regression model that describes the inter-band relationship and is used to reconstruct the missing pixels. Our novel method is able to increase the PSNR on average by 2 dB and yields visually much more appealing images in high-frequency regions.

eess.IV↗

arXiv2022

Structure-Preserving Spectral Reflectance Estimation using Guided Filtering

Light spectra are a very important source of information for diverse classification problems, e.g., for discrimination of materials. To lower the cost for acquiring this information, multispectral cameras are used. Several techniques exist for estimating light spectra out of multispectral images by exploiting properties about the spectrum. Unfortunately, especially when capturing multispectral videos, the images are heavily affected by noise due to the nature of limited exposure times in videos. Therefore, models that explicitly try to lower the influence of noise on the reconstructed spectrum are highly desirable. Hence, a novel reconstruction algorithm is presented. This novel estimation method is based on the guided filtering technique which preserves basic structures, while using spatial information to reduce the influence of noise. The evaluation based on spectra of natural images reveals that this new technique yields better quantitative and subjective results in noisy scenarios than other state-of-the-art spatial reconstruction methods. Specifically, the proposed algorithm lowers the mean squared error and the spectral angle up to 46% and 35% in noisy scenarios, respectively. Furthermore, it is shown that the proposed reconstruction technique works out-of-the-box and does not need any calibration or training by reconstructing spectra from a real-world multispectral camera with nine channels.

eess.IV↗

arXiv2022

Motion-Adapted Three-Dimensional Frequency Selective Extrapolation

It has been shown, that high resolution images can be acquired using a low resolution sensor with non-regular sampling. Therefore, post-processing is necessary. In terms of video data, not only the spatial neighborhood can be used to assist the reconstruction, but also the temporal neighborhood. A popular and well performing algorithm for this kind of problem is the three-dimensional frequency selective extrapolation (3D-FSE) for which a motion adapted version is introduced in this paper. This proposed extension solves the problem of changing content within the area considered by the 3D-FSE, which is caused by motion within the sequence. Because of this motion, it may happen that regions are emphasized during the reconstruction that are not present in the original signal within the considered area. By that, false content is introduced into the extrapolated sequence, which affects the resulting image quality negatively. The novel extension, presented in the following, incorporates motion data of the sequence in order to adapt the algorithm accordingly, and compensates changing content, resulting in gains of up to 1.75 dB compared to the existing 3D-FSE.

eess.IV↗

arXiv2022

Orthogonality Deficiency Compensation for Improved Frequency Selective Image Extrapolation

This paper describes a very efficient algorithm for image signal extrapolation. It can be used for various applications in image and video communication, e.g. the concealment of data corrupted by transmission errors or prediction in video coding. The extrapolation is performed on a limited number of known samples and extends the signal beyond these samples. Therefore the signal from the known samples is iteratively projected onto different basis functions in order to generate a model of the signal. As the basis functions are not orthogonal with respect to the area of the known samples we propose a new extension, the orthogonality deficiency compensation, to cope with the non-orthogonality. Using this extension, very good extrapolation results for structured as well as for smooth areas are achievable. This algorithm improves PSNR up to 2 dB and gives a better visual quality for concealment of block losses compared to extrapolation algorithms existent so far.

eess.IV↗

arXiv2022

Spatio-temporal prediction in video coding by best approximation

Within the scope of this contribution we propose a novel efficient spatio-temporal prediction algorithm for video coding. The algorithm operates in two stages. First, motion compensation is performed on the block to be predicted in order to exploit temporal correlations. Afterwards, in order to exploit spatial correlations, this preliminary estimate is spatially refined by forming a joint model of the motion compensated block and spatially adjacent already decoded blocks. Compared to an earlier refinement algorithm, the novel one only needs very little iteration, leading to a speedup of factor 17. The implementation of this new algorithm into the H.264/AVC leads to a maximum reduction in data rate of up to nearly 13% for the considered sequences.

eess.IV↗