arXiv subjects

Jukka K. Nurminen

Publications and source records attributed to Jukka K. Nurminen.

At least 19 recordsLinked to original sources

arXiv2026

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked with open-domain question answering (QA) datasets containing questions and corresponding short reference answers. First, an LLM is used to generate answers to questions within the QA dataset. Then, some automated labeling strategy is used to label these answers as hallucinated or not by comparing them with the reference answers in the dataset. This evaluation setting creates a methodological ambiguity between two criteria: reference faithfulness (whether the answer is fully supported by the reference) and factual correctness (whether the answer is free from contradictions and factually false specific claims). In practice, automated labelers may apply the former criterion even when the intended target is the latter. We study this potential criterion mismatch using 900 human-labeled question-answer pairs spanning three commonly used QA datasets and three generator models, with labels targeting answer-level factual correctness. We evaluate lexical similarity metrics, a reference-entailment NLI baseline, and seven LLM judges under controlled prompt variants as automated labelers. Our experiments reveal substantial disagreement both among automated labeling strategies and between these labels and human annotations. Many strategies also exhibit strong directional error biases, and for most judge-generator pairs, replacing a faithfulness-oriented prompt with a factual-correctness prompt improves agreement with human annotations and reduces false-positive dominance, indicating that automated hallucination labels depend strongly on how the target criterion is specified. Label-source choice should therefore be considered a fundamental part of benchmark design and made explicit, validated, and matched with the benchmark goal.

cs.CL↗

arXiv2026

Architecture-aware Unitary Synthesis

We present a novel architecture-aware transpilation method for exact general unitary gate synthesis on superconducting quantum hardware. Our approach is tightly integrated with the optimized block-ZXZ decomposition, exploiting its recursive structure to make hardware-aware decisions at each level of the recursion rather than treating transpilation as an independent post-processing step. The method introduces three key techniques: a greedy qubit mapping strategy that minimizes pairwise distances between physical qubits, an adaptive Gray code selection combined with qubit swapping that optimizes the construction of uniformly controlled Rz gates for the target topology, and a heuristic for reducing CNOT gates by exploiting the structure of long-range CNOT ladders. We benchmark our method against TKet, Qiskit, and Pennylane on the 20-qubit IQM Garnet (square lattice) and the 156-qubit IBM Marrakesh (heavy-hex) architectures with qubit counts ranging from 3 to 11. Our method achieves CNOT count reductions of up to 36 percent on the IQM Garnet and up to 34 percent on the IBM Marrakesh compared to the best competing transpiler, while simultaneously achieving transpilation speedups of up to 553x. Furthermore, our method is the only one capable of transpiling circuits beyond 10 qubits within a 30-minute time limit across both architectures.

quant-ph↗

arXiv2026

The Average Relative Entropy and Transpilation Depth determines the noise robustness in Variational Quantum Classifiers

Variational Quantum Algorithms (VQAs) have been extensively researched for applications in Quantum Machine Learning (QML), Optimization, and Molecular simulations. Although designed for Noisy Intermediate-Scale Quantum (NISQ) devices, VQAs are predominantly evaluated classically due to uncertain results on noisy devices and limited resource availability. Raising concern over the reproducibility of simulated VQAs on noisy hardware. While prior studies indicate that VQAs may exhibit noise resilience in specific parameterized shallow quantum circuits, there are no definitive measures to establish what defines a shallow circuit or the optimal circuit depth for VQAs on a noisy platform. These challenges extend naturally to Variational Quantum Classification (VQC) algorithms, a subclass of VQAs for supervised learning. In this article, we propose a relative entropy-based metric to verify whether a VQC model would perform similarly on a noisy device as it does on simulations. We establish a strong correlation between the average relative entropy difference in classes, transpilation circuit depth, and their performance difference on a noisy quantum device. Our results further indicate that circuit depth alone is insufficient to characterize shallow circuits. We present empirical evidence to support these assertions across a diverse array of techniques for implementing VQC, datasets, and multiple noisy quantum devices.

quant-ph↗

arXiv2026

Performance Estimation in Binary Classification Using Calibrated Confidence

Model monitoring is a critical component of the machine learning lifecycle, safeguarding against undetected drops in the model's performance after deployment. Traditionally, performance monitoring has required access to ground truth labels, which are not always readily available. This can result in unacceptable latency or render performance monitoring altogether impossible. Recently, methods designed to estimate the accuracy of classifier models without access to labels have shown promising results. However, there are various other metrics that might be more suitable for assessing model performance in many cases. Until now, none of these important metrics has received similar interest from the scientific community. In this work, we address this gap by presenting CBPE, a novel method that can estimate any binary classification metric defined using the confusion matrix. In particular, we choose four metrics from this large family: accuracy, precision, recall, and F$_1$, to demonstrate our method. CBPE treats the elements of the confusion matrix as random variables and leverages calibrated confidence scores of the model to estimate their distributions. The desired metric is then also treated as a random variable, whose full probability distribution can be derived from the estimated confusion matrix. CBPE is shown to produce estimates that come with strong theoretical guarantees and valid confidence intervals.

cs.LG↗

arXiv2025

Influence of Data Dimensionality Reduction Methods on the Effectiveness of Quantum Machine Learning Models

Data dimensionality reduction techniques are often utilized in the implementation of Quantum Machine Learning models to address two significant issues: the constraints of NISQ quantum devices, which are characterized by noise and a limited number of qubits, and the challenge of simulating a large number of qubits on classical devices. It also raises concerns over the scalability of these approaches, as dimensionality reduction methods are slow to adapt to large datasets. In this article, we analyze how data reduction methods affect different QML models. We conduct this experiment over several generated datasets, quantum machine algorithms, quantum data encoding methods, and data reduction methods. All these models were evaluated on the performance metrics like accuracy, precision, recall, and F1 score. Our findings have led us to conclude that the usage of data dimensionality reduction methods results in skewed performance metric values, which results in wrongly estimating the actual performance of quantum machine learning models. There are several factors, along with data dimensionality reduction methods, that worsen this problem, such as characteristics of the datasets, classical to quantum information embedding methods, percentage of feature reduction, classical components associated with quantum models, and structure of quantum machine learning models. We consistently observed the difference in the accuracy range of 14% to 48% amongst these models, using data reduction and not using it. Apart from this, our observations have shown that some data reduction methods tend to perform better for some specific data embedding methodologies and ansatz constructions.

quant-ph↗

arXiv2025

Perspectives on Utilization of Measurements in Quantum Algorithms

Measurement is a fundamental operation in quantum computing and has many important use cases in quantum algorithms. This article provides a comprehensive overview of the basic measurement operations in quantum computing and represents a selected set of their applications in quantum algorithms. Our goal is to provide one of the first algorithmic overviews of measurement processes in quantum computing. From the quantum information-theoretical perspective, measurements are either a method to access the result of a quantum computation or a technique to modify a quantum state. We also identify measurement-based methods to solve quantum computational challenges, such as error mitigation and circuit cutting. We discuss three main categories of measurements: performing measurements in static quantum circuits, modifying the quantum state in dynamic quantum circuits via measurements, and addressing challenges in quantum computing with measurements. Based on the reviewed topics, the measurement operations are frequently not at the center of the quantum algorithm design. However, the most novel and error-prone quantum algorithms will likely require sophisticated measurement schemes. Thus, the central message of this article is to broaden the view of measurement operations and highlight their importance at every level of quantum algorithm design.

quant-ph↗

arXiv2025

Optimizing State Preparation for Variational Quantum Regression on NISQ Hardware

The execution of quantum algorithms on modern hardware is often constrained by noise and qubit decoherence, limiting the circuit depth and the number of gates that can be executed. Circuit optimization techniques help mitigate these limitations, enhancing algorithm feasibility. In this work, we implement, optimize, and execute a variational quantum regression algorithm using a novel state preparation method. By leveraging ZX-calculus-based optimization techniques, such as Pauli pushing, phase folding, and Hadamard pushing, we achieve a more efficient circuit design. Our results demonstrate that these optimizations enable the successful execution of the quantum regression algorithm on current hardware. Furthermore, the techniques presented are broadly applicable to other quantum circuits requiring arbitrary real-valued state preparation, advancing the practical implementation of quantum algorithms.

quant-ph↗

arXiv2025

How Large Language Models Are Changing MOOC Essay Answers: A Comparison of Pre- and Post-LLM Responses

The release of ChatGPT in late 2022 caused a flurry of activity and concern in the academic and educational communities. Some see the tool's ability to generate human-like text that passes at least cursory inspections for factual accuracy ``often enough'' a golden age of information retrieval and computer-assisted learning. Some, on the other hand, worry the tool may lead to unprecedented levels of academic dishonesty and cheating. In this work, we quantify some of the effects of the emergence of Large Language Models (LLMs) on online education by analyzing a multi-year dataset of student essay responses from a free university-level MOOC on AI ethics. Our dataset includes essays submitted both before and after ChatGPT's release. We find that the launch of ChatGPT coincided with significant changes in both the length and style of student essays, mirroring observations in other contexts such as academic publishing. We also observe -- as expected based on related public discourse -- changes in prevalence of key content words related to AI and LLMs, but not necessarily the general themes or topics discussed in the student essays as identified through (dynamic) topic modeling.

cs.CY↗

arXiv2025

Confidence-based Estimators for Predictive Performance in Model Monitoring

After a machine learning model has been deployed into production, its predictive performance needs to be monitored. Ideally, such monitoring can be carried out by comparing the model's predictions against ground truth labels. For this to be possible, the ground truth labels must be available relatively soon after inference. However, there are many use cases where ground truth labels are available only after a significant delay, or in the worst case, not at all. In such cases, directly monitoring the model's predictive performance is impossible. Recently, novel methods for estimating the predictive performance of a model when ground truth is unavailable have been developed. Many of these methods leverage model confidence or other uncertainty estimates and are experimentally compared against a naive baseline method, namely Average Confidence (AC), which estimates model accuracy as the average of confidence scores for a given set of predictions. However, until now the theoretical properties of the AC method have not been properly explored. In this paper, we try to fill this gap by reviewing the AC method and show that under certain general assumptions, it is an unbiased and consistent estimator of model accuracy with many desirable properties. We also compare this baseline estimator against some more complex estimators empirically and show that in many cases the AC method is able to beat the others, although the comparative quality of the different estimators is heavily case-dependent.

cs.LG↗

arXiv2024

The Impact of Feature Embedding Placement in the Ansatz of a Quantum Kernel in QSVMs

Designing a useful feature map for a quantum kernel is a critical task when attempting to achieve an advantage over classical machine learning models. The choice of circuit architecture, i.e. how feature-dependent gates should be interwoven with other gates is a relatively unexplored problem and becomes very important when using a model of quantum kernels called Quantum Embedding Kernels (QEK). We study and categorize various architectural patterns in QEKs and show that existing architectural styles do not behave as the literature supposes. We also produce a novel alternative architecture based on the old ones and show that it performs equally well while containing fewer gates than its older counterparts.

quant-ph↗

arXiv2023

Quantum Software Engineering Challenges from Developers' Perspective: Mapping Research Challenges to the Proposed Workflow Model

Despite the increasing interest in quantum computing, the aspect of development to achieve cost-effective and reliable quantum software applications has been slow. One barrier is the software engineering of quantum programs, which can be approached from two directions. On the one hand, many software engineering practices, debugging in particular, are bound to classical computing. On the other hand, quantum programming is closely associated with the phenomena of quantum physics, and consequently, the way we express programs resembles the early days of programming. Moreover, much of the software engineering research today focuses on agile development, where computing cycles are cheap and new software can be rapidly deployed and tested, whereas in the quantum context, executions may consume lots of energy, and test runs may require lots of work to interpret. In this paper, we aim at bridging this gap by starting with the quantum computing workflow and by mapping existing software engineering research to this workflow. Based on the mapping, we then identify directions for software engineering research for quantum computing.

cs.SE↗

arXiv2021

On Misbehaviour and Fault Tolerance in Machine Learning Systems

Machine learning (ML) provides us with numerous opportunities, allowing ML systems to adapt to new situations and contexts. At the same time, this adaptability raises uncertainties concerning the run-time product quality or dependability, such as reliability and security, of these systems. Systems can be tested and monitored, but this does not provide protection against faults and failures in adapted ML systems themselves. We studied software designs that aim at introducing fault tolerance in ML systems so that possible problems in ML components of the systems can be avoided. The research was conducted as a case study, and its data was collected through five semi-structured interviews with experienced software architects. We present a conceptualisation of the misbehaviour of ML systems, the perceived role of fault tolerance, and the designs used. Common patterns to incorporating ML components in design in a fault tolerant fashion have started to emerge. ML models are, for example, guarded by monitoring the inputs and their distribution, and enforcing business rules on acceptable outputs. Multiple, specialised ML models are used to adapt to the variations and changes in the surrounding world, and simpler fall-over techniques like default outputs are put in place to have systems up and running in the face of problems. However, the general role of these patterns is not widely acknowledged. This is mainly due to the relative immaturity of using ML as part of a complete software system: the field still lacks established frameworks and practices beyond training to implement, operate, and maintain the software that utilises ML. ML software engineering needs further analysis and development on all fronts.

cs.SE↗

arXiv2021

Systematic Literature Review of Validation Methods for AI Systems

Context: Artificial intelligence (AI) has made its way into everyday activities, particularly through new techniques such as machine learning (ML). These techniques are implementable with little domain knowledge. This, combined with the difficulty of testing AI systems with traditional methods, has made system trustworthiness a pressing issue. Objective: This paper studies the methods used to validate practical AI systems reported in the literature. Our goal is to classify and describe the methods that are used in realistic settings to ensure the dependability of AI systems. Method: A systematic literature review resulted in 90 papers. Systems presented in the papers were analysed based on their domain, task, complexity, and applied validation methods. Results: The validation methods were synthesized into a taxonomy consisting of trial, simulation, model-centred validation, and expert opinion. Failure monitors, safety channels, redundancy, voting, and input and output restrictions are methods used to continuously validate the systems after deployment. Conclusions: Our results clarify existing strategies applied to validation. They form a basis for the synthesization, assessment, and refinement of AI system validation in research and guidelines for validating individual systems in practice. While various validation strategies have all been relatively widely applied, only few studies report on continuous validation. Keywords: artificial intelligence, machine learning, validation, testing, V&V, systematic literature review.

cs.SE↗

arXiv2020

Testing the Robustness of AutoML Systems

Automated machine learning (AutoML) systems aim at finding the best machine learning (ML) pipeline that automatically matches the task and data at hand. We investigate the robustness of machine learning pipelines generated with three AutoML systems, TPOT, H2O, and AutoKeras. In particular, we study the influence of dirty data on accuracy, and consider how using dirty training data may help create more robust solutions. Furthermore, we also analyze how the structure of the generated pipelines differs in different cases.

cs.LG↗

arXiv2020

Validation Frameworks for Self-Driving Vehicles: A Survey

As a part of the digital transformation, we interact with more and more intelligent gadgets. Today, these gadgets are often mobile devices, but in the advent of smart cities, more and more infrastructure---such as traffic and buildings---in our surroundings becomes intelligent. The intelligence, however, does not emerge by itself. Instead, we need both design techniques to create intelligent systems, as well as approaches to validate their correct behavior. An example of intelligent systems that could benefit smart cities are self-driving vehicles. Self-driving vehicles are continuously becoming both commercially available and common on roads. Accidents involving self-driving vehicles, however, have raised concerns about their reliability. Due to these concerns, the safety of self-driving vehicles should be thoroughly tested before they can be released into traffic. To ensure that self-driving vehicles encounter all possible scenarios, several millions of hours of testing must be carried out; therefore, testing self-driving vehicles in the real world is impractical. There is also the issue that testing self-driving vehicles directly in the traffic poses a potential safety hazard to human drivers. To tackle this challenge, validation frameworks for testing self-driving vehicles in simulated scenarios are being developed by academia and industry. In this chapter, we briefly introduce self-driving vehicles and give an overview of validation frameworks for testing them in a simulated environment. We conclude by discussing what an ideal validation framework at the state of the art should be and what could benefit validation frameworks for self-driving vehicles in the future.

cs.SE↗

arXiv2014

Techniques and tools for measuring energy efficiency of scientific software applications

The scale of scientific High Performance Computing (HPC) and High Throughput Computing (HTC) has increased significantly in recent years, and is becoming sensitive to total energy use and cost. Energy-efficiency has thus become an important concern in scientific fields such as High Energy Physics (HEP). There has been a growing interest in utilizing alternate architectures, such as low power ARM processors, to replace traditional Intel x86 architectures. Nevertheless, even though such solutions have been successfully used in mobile applications with low I/O and memory demands, it is unclear if they are suitable and more energy-efficient in the scientific computing environment. Furthermore, there is a lack of tools and experience to derive and compare power consumption between the architectures for various workloads, and eventually to support software optimizations for energy efficiency. To that end, we have performed several physical and software-based measurements of workloads from HEP applications running on ARM and Intel architectures, and compare their power consumption and performance. We leverage several profiling tools (both in hardware and software) to extract different characteristics of the power use. We report the results of these measurements and the experience gained in developing a set of measurement techniques and profiling tools to accurately assess the power consumption for scientific workloads.

cs.DC↗

arXiv2014

Mobile Multimedia Streaming Techniques : QoE and Energy Consumption Perspective

Multimedia streaming to mobile devices is challenging for two reasons. First, the way content is delivered to a client must ensure that the user does not experience a long initial playback delay or a distorted playback in the middle of a streaming session. Second, multimedia streaming applications are among the most energy hungry applications in smartphones. The energy consumption mostly depends on the delivery techniques and on the power management techniques of wireless access technologies (Wi-Fi, 3G, and 4G). In order to provide insights on what kind of streaming techniques exist, how they work on different mobile platforms, their efforts in providing smooth quality of experience, and their impact on energy consumption of mobile phones, we did a large set of active measurements with several smartphones having both Wi-Fi and cellular network access. Our analysis reveals five different techniques to deliver the content to the video players. The selection of a technique depends on the mobile platform, device, player, quality, and service. The results from our traffic and power measurements allow us to conclude that none of the identified techniques is optimal because they take none of the following facts into account: access technology used, user behavior, and user preferences concerning data waste. We point out the technique with optimal playback buffer configuration, which provides the most attractive trade-offs in particular situations.

cs.MM↗

arXiv2014

Saving Energy in Mobile Devices for On-Demand Multimedia Streaming -- A Cross-Layer Approach

This paper proposes a novel energy-efficient multimedia delivery system called EStreamer. First, we study the relationship between buffer size at the client, burst-shaped TCP-based multimedia traffic, and energy consumption of wireless network interfaces in smartphones. Based on the study, we design and implement EStreamer for constant bit rate and rate-adaptive streaming. EStreamer can improve battery lifetime by 3x, 1.5x and 2x while streaming over Wi-Fi, 3G and 4G respectively.

cs.MM↗