arXiv subjects

Shaukat Ali

Publications and source records attributed to Shaukat Ali.

At least 55 records · Page 3Linked to original sources

arXiv2025

Bloch Vector Assertions for Debugging Quantum Programs

Quantum programs must be reliable to ensure trustworthy results, yet debugging them is notoriously challenging due to quantum-specific faults like gate misimplementations and hardware noise, as well as their inherently probabilistic nature. Assertion-based debugging provides a promising solution by enabling localized correctness checks during execution. However, current approaches face challenges including manual assertion generation, reliance on mid-circuit-measurements, and poor scalability. In this paper, we present Bloq, a scalable, automated fault localization approach introducing Bloch-vector-based assertions utilizing expectation value measurements of Pauli operators, enabling low-overhead fault localization without mid-circuit measurements. In addition, we introduce AutoBloq, a component of Bloq for automatically generating assertion schemes from quantum algorithms. An experimental evaluation over 684432 programs using two algorithms (Quantum Fourier Transform (QFT) and Grover) shows that Bloq consistently outperforms the state-of-the-art approach Proq, notably as circuit depth and noise increase. For Grover, Bloq achieves a mean F1 score across all experimental instances of 0.74 versus 0.38 for Proq under ideal conditions, and maintains performance under noise (0.43 versus 0.06). Bloq also reduces Proq's runtime by a factor of 5 and circuit depth overhead by a factor of 23. These results underline Bloq's potential to make assertion-based debugging scalable and effective for near-term quantum devices.

quant-ph↗

arXiv2025

Quantum Circuit Mutants: Empirical Analysis and Recommendations

As a new research area, quantum software testing lacks systematic testing benchmarks to assess testing techniques' effectiveness. Recently, some open-source benchmarks and mutation analysis tools have emerged. However, there is insufficient evidence on how various quantum circuit characteristics (e.g., circuit depth, number of quantum gates), algorithms (e.g., Quantum Approximate Optimization Algorithm), and mutation characteristics (e.g., mutation operators) affect the detection of mutants in quantum circuits. Studying such relations is important to systematically design faulty benchmarks with varied attributes (e.g., the difficulty in detecting a seeded fault) to facilitate assessing the cost-effectiveness of quantum software testing techniques efficiently. To this end, we present a large-scale empirical evaluation with more than 700K faulty benchmarks (quantum circuits) generated by mutating 382 real-world quantum circuits. Based on the results, we provide valuable insights for researchers to define systematic quantum mutation analysis techniques. We also provide a tool to recommend mutants to users based on chosen characteristics (e.g., a quantum algorithm type) and the required difficulty of detecting mutants. Finally, we also provide faulty benchmarks that can already be used to assess the cost-effectiveness of quantum software testing techniques.

cs.SE↗

arXiv2025

Using quantum annealing to generate test cases for cyber-physical systems

Quantum computing has emerged as a powerful tool to efficiently solve computational challenges, particularly in simulation and optimisation. However, hardware limitations prevent quantum computers from achieving the full theoretical potential. Among the quantum algorithms, quantum annealing is a prime candidate to solve optimisation problems. This makes it a natural candidate for search-based software testing in the Cyber-Physical Systems (CPS) domain, which demands effective test cases due to their safety-critical nature. This work explores the use of quantum annealing to enhance test case generation for CPS through a mutation-based approach. We encode test case mutation as a binary optimisation problem, and use quantum annealing to identify and target critical regions of the test cases for improvement. Our approach mechanises this process into an algorithm that uses D-Wave's quantum annealer to find the solution. As a main contribution, we offer insights into how quantum annealing can advance software testing methodologies by empirically evaluating the correlation between problem size, hardware limitations, and the effectiveness of the results. Moreover, we compare the proposed method against state-of-the-art classical optimisation algorithms, targeting efficiency (time to generate test cases) and effectiveness (fault detection rates). Results indicate that quantum annealing enables faster test case generation while achieving comparable fault detection performance to state-of-the-art alternatives.

cs.ET↗

arXiv2025

Digital Twin-based Out-of-Distribution Detection in Autonomous Vessels

An autonomous vessel (AV) is a complex cyber-physical system (CPS) with software enabling many key functionalities, e.g., navigation software enables an AV to autonomously or semi-autonomously follow a path to its destination. Digital twins of such AVs enable advanced functionalities such as running what-if scenarios, performing predictive maintenance, and enabling fault diagnosis. Due to technological improvements, real-time analyses using continuous data from vessels' real-time operations have become increasingly possible. However, the literature has little explored developing advanced analyses in real-time data in AVs with digital twins built with machine learning techniques. To this end, we present a novel digital twin-based approach (ODDIT) to detect future out-of-distribution (OOD) states of an AV before reaching them, enabling proactive intervention. Such states may indicate anomalies requiring attention (e.g., manual correction by the ship master) and assist testers in scenario-centered testing. The digital twin consists of two machine-learning models predicting future vessel states and whether the predicted state will be OOD. We evaluated ODDIT with five vessels across waypoint and zigzag maneuvering under simulated conditions, including sensor and actuator noise and environmental disturbances i.e., ocean current. ODDIT achieved high accuracy in detecting OOD states, with AUROC and TNR@TPR95 scores reaching 99\% across multiple vessels.

cs.RO↗

arXiv2025

Envisioning Responsible Quantum Software Engineering and Quantum Artificial Intelligence

The convergence of Quantum Computing (QC), Quantum Software Engineering (QSE), and Artificial Intelligence (AI) presents transformative opportunities across various domains. However, existing methodologies inadequately address the ethical, security, and governance challenges arising from this technological shift. This paper highlights the urgent need for interdisciplinary collaboration to embed ethical principles into the development of Quantum AI (QAI) and QSE, ensuring transparency, inclusivity, and equitable global access. Without proactive governance, there is a risk of deepening digital inequalities and consolidating power among a select few. We call on the software engineering community to actively shape a future where responsible QSE and QAI are foundational for ethical, accountable, and socially beneficial technological progress.

cs.SE↗

arXiv2025

LLMs in the Heart of Differential Testing: A Case Study on a Medical Rule Engine

The Cancer Registry of Norway (CRN) uses an automated cancer registration support system (CaReSS) to support core cancer registry activities, i.e, data capture, data curation, and producing data products and statistics for various stakeholders. GURI is a core component of CaReSS, which is responsible for validating incoming data with medical rules. Such medical rules are manually implemented by medical experts based on medical standards, regulations, and research. Since large language models (LLMs) have been trained on a large amount of public information, including these documents, they can be employed to generate tests for GURI. Thus, we propose an LLM-based test generation and differential testing approach (LLMeDiff) to test GURI. We experimented with four different LLMs, two medical rule engine implementations, and 58 real medical rules to investigate the hallucination, success, time efficiency, and robustness of the LLMs to generate tests, and these tests' ability to find potential issues in GURI. Our results showed that GPT-3.5 hallucinates the least, is the most successful, and is generally the most robust; however, it has the worst time efficiency. Our differential testing revealed 22 medical rules where implementation inconsistencies were discovered (e.g., regarding handling rule versions). Finally, we provide insights for practitioners and researchers based on the results.

cs.SE↗

arXiv2024

Quantum Software Engineering: Roadmap and Challenges Ahead

As quantum computers advance, the complexity of the software they can execute increases as well. To ensure this software is efficient, maintainable, reusable, and cost-effective -key qualities of any industry-grade software-mature software engineering practices must be applied throughout its design, development, and operation. However, the significant differences between classical and quantum software make it challenging to directly apply classical software engineering methods to quantum systems. This challenge has led to the emergence of Quantum Software Engineering as a distinct field within the broader software engineering landscape. In this work, a group of active researchers analyse in depth the current state of quantum software engineering research. From this analysis, the key areas of quantum software engineering are identified and explored in order to determine the most relevant open challenges that should be addressed in the next years. These challenges help identify necessary breakthroughs and future research directions for advancing Quantum Software Engineering.

cs.SE↗

arXiv2024

Testing Medical Rules Web Services in Practice

The Cancer Registry of Norway (CRN) collects and processes cancer-related data for patients in Norway. For this, it employs a sociotechnical software system that evolves with changing requirements and medical standards. The current practice is to manually test CRN's system to prevent faults and ensure its dependability. This paper focuses on automatically testing GURI, the CRN's medical rule engine, using a system-level testing tool, EvoMaster, in both its black-box and white-box modes, and a novel CRN-specific EvoMaster-based tool, EvoGURI. We empirically evaluate the tools' effectiveness regarding code coverage, errors found, domain-specific rule coverage, and ability to identify artificial faults ten versions of GURI. Our results show that all the tools achieve similar code coverage and identified a similar number of errors. For rule coverage, EvoGURI and EvoMaster's black-box mode produce test suites that cover the highest number of rules with Pass, Fail, and Warning results. The test suites of EvoGURI and two EvoMaster white-box tools identify the most faults in a mutation testing experiment. Based on our findings, we recommend using EvoGURI in CRN's current practice. Finally, we present key takeaways and outline open research questions for the research community.

cs.SE↗

arXiv2024

MeDeT: Medical Device Digital Twins Creation with Few-shot Meta-learning

Testing healthcare Internet of Things (IoT) applications at system and integration levels necessitates integrating numerous medical devices of various types. Challenges of incorporating medical devices are: (i) their continuous evolution, making it infeasible to include all device variants, and (ii) rigorous testing at scale requires multiple devices and their variants, which is time-intensive, costly, and impractical. Our collaborator, Oslo City's health department, faced these challenges in developing automated test infrastructure, which our research aims to address. In this context, we propose a meta-learning-based approach (MeDeT) to generate digital twins (DTs) of medical devices and adapt DTs to evolving devices. We evaluate MeDeT in OsloCity's context using five widely-used medical devices integrated with a real-world healthcare IoT application. Our evaluation assesses MeDeT's ability to generate and adapt DTs across various devices and versions using different few-shot methods, the fidelity of these DTs, the scalability of operating 1000 DTs concurrently, and the associated time costs. Results show that MeDeT can generate DTs with over 96% fidelity, adapt DTs to different devices and newer versions with reduced time cost (around one minute), and operate 1000 DTs in a scalable manner while maintaining the fidelity level, thus serving in place of physical devices for testing.

cs.SE↗

arXiv2024

Quantum Neural Network Classifier for Cancer Registry System Testing: A Feasibility Study

The Cancer Registry of Norway (CRN) is a part of the Norwegian Institute of Public Health (NIPH) and is tasked with producing statistics on cancer among the Norwegian population. For this task, CRN develops, tests, and evolves a software system called Cancer Registration Support System (CaReSS). It is a complex socio-technical software system that interacts with many entities (e.g., hospitals, medical laboratories, and other patient registries) to achieve its task. For cost-effective testing of CaReSS, CRN has employed EvoMaster, an AI-based REST API testing tool combined with an integrated classical machine learning model. Within this context, we propose Qlinical to investigate the feasibility of using, inside EvoMaster, a Quantum Neural Network (QNN) classifier, i.e., a quantum machine learning model, instead of the existing classical machine learning model. Results indicate that Qlinical can achieve performance comparable to that of EvoClass. We further explore the effects of various QNN configurations on performance and offer recommendations for optimal QNN settings for future QNN developers.

cs.SE↗

arXiv2024

Faster and Better Quantum Software Testing through Specification Reduction and Projective Measurements

Quantum computing promises polynomial and exponential speedups in many domains, such as unstructured search and prime number factoring. However, quantum programs yield probabilistic outputs from exponentially growing distributions and are vulnerable to quantum-specific faults. Existing quantum software testing (QST) approaches treat quantum superpositions as classical distributions. This leads to two major limitations when applied to quantum programs: (1) an exponentially growing sample space distribution and (2) failing to detect quantum-specific faults such as phase flips. To overcome these limitations, we introduce a QST approach, which applies a reduction algorithm to a quantum program specification. The reduced specification alleviates the limitations (1) by enabling faster sampling through quantum parallelism and (2) by performing projective measurements in the mixed Hadamard basis. Our evaluation of 143 quantum programs across four categories demonstrates significant improvements in test runtimes and fault detection with our reduction approach. Average test runtimes improved from 169.9s to 11.8s, with notable enhancements in programs with large circuit depths (383.1s to 33.4s) and large program specifications (464.8s to 7.7s). Furthermore, our approach increases mutation scores from 54.5% to 74.7%, effectively detecting phase flip faults that non-reduced specifications miss. These results underline our approach's importance to improve QST efficiency and effectiveness.

cs.SE↗

arXiv2024

Uncertainty-Aware Environment Simulation of Medical Devices Digital Twins

Smart medical devices are an integral component of the healthcare Internet of Things (IoT), providing patients with various healthcare services through an IoT-based application. Ensuring the dependability of such applications through system and integration-level testing mandates the physical integration of numerous medical devices, which is costly and impractical. In this context, digital twins of medical devices play an essential role in facilitating testing automation. Testing with digital twins without accounting for uncertain environmental factors of medical devices leaves many functionalities of IoT-based healthcare applications untested. In addition, digital twins operating without environmental factors remain out of sync and uncalibrated with their corresponding devices functioning in the real environment. To deal with these challenges, in this paper, we propose a model-based approach (EnvDT) for modeling and simulating the environment of medical devices' digital twins under uncertainties. We empirically evaluate the EnvDT using three medicine dispensers, Karie, Medido, and Pilly connected to a real-world IoT-based healthcare application. Our evaluation targets analyzing the coverage of environment models and the diversity of uncertain scenarios generated for digital twins. Results show that EnvDT achieves approximately 61% coverage of environment models and generates diverse uncertain scenarios (with a near-maximum diversity value of 0.62) during multiple environmental simulations.

cs.SE↗

arXiv2024

Quantum Program Testing Through Commuting Pauli Strings on IBM's Quantum Computers

The most promising applications of quantum computing are centered around solving search and optimization tasks, particularly in fields such as physics simulations, quantum chemistry, and finance. However, the current quantum software testing methods face practical limitations when applied in industrial contexts: (i) they do not apply to quantum programs most relevant to the industry, (ii) they require a full program specification, which is usually not available for these programs, and (iii) they are incompatible with error mitigation methods currently adopted by main industry actors like IBM. To address these challenges, we present QOPS, a novel quantum software testing approach. QOPS introduces a new definition of test cases based on Pauli strings to improve compatibility with different quantum programs. QOPS also introduces a new test oracle that can be directly integrated with industrial APIs such as IBM's Estimator API and can utilize error mitigation methods for testing on real noisy quantum computers. We also leverage the commuting property of Pauli strings to relax the requirement of having complete program specifications, making QOPS practical for testing complex quantum programs in industrial settings. We empirically evaluate QOPS on 194,982 real quantum programs, demonstrating effective performance in test assessment compared to the state-of-the-art with a perfect F1-score, precision, and recall. Furthermore, we validate the industrial applicability of QOPS by assessing its performance on IBM's three real quantum computers, incorporating both industrial and open-source error mitigation methods.

cs.SE↗

arXiv2024

HITA: An Architecture for System-level Testing of Healthcare IoT Applications

System-level testing of healthcare Internet of Things (IoT) applications requires creating a test infrastructure with integrated medical devices and third-party applications. A significant challenge in creating such test infrastructure is that healthcare IoT applications evolve continuously with the addition of new medical devices from different vendors and new services offered by different third-party organizations following different architectures. Moreover, creating test infrastructure with a large number of different types of medical devices is time-consuming, financially expensive, and practically infeasible. Oslo City's healthcare department faced these challenges while working with various healthcare IoT applications. To address these challenges, this paper presents a real-world test infrastructure software architecture (HITA) designed for healthcare IoT applications. We evaluated HITA's digital twin (DT) generation component implemented using model-based and machine learning (ML) approaches in terms of DT fidelity, scalability, and time cost of generating DTs. Results show that the fidelity of DTs created using model-based and ML approaches reach 94% and 95%, respectively. Results from operating 100 DTs concurrently show that the DT generation component is scalable and ML-based DTs have a higher time cost.

cs.SE↗

arXiv2024

A Machine Learning-Based Error Mitigation Approach For Reliable Software Development On IBM'S Quantum Computers

Quantum computers have the potential to outperform classical computers for some complex computational problems. However, current quantum computers (e.g., from IBM and Google) have inherent noise that results in errors in the outputs of quantum software executing on the quantum computers, affecting the reliability of quantum software development. The industry is increasingly interested in machine learning (ML)--based error mitigation techniques, given their scalability and practicality. However, existing ML-based techniques have limitations, such as only targeting specific noise types or specific quantum circuits. This paper proposes a practical ML-based approach, called Q-LEAR, with a novel feature set, to mitigate noise errors in quantum software outputs. We evaluated Q-LEAR on eight quantum computers and their corresponding noisy simulators, all from IBM, and compared Q-LEAR with a state-of-the-art ML-based approach taken as baseline. Results show that, compared to the baseline, Q-LEAR achieved a 25% average improvement in error mitigation on both real quantum computers and simulators. We also discuss the implications and practicality of Q-LEAR, which, we believe, is valuable for practitioners.

cs.SE↗

arXiv2024

Evaluating Search-Based Software Microbenchmark Prioritization

Ensuring that software performance does not degrade after a code change is paramount. A solution is to regularly execute software microbenchmarks, a performance testing technique similar to (functional) unit tests, which, however, often becomes infeasible due to extensive runtimes. To address that challenge, research has investigated regression testing techniques, such as test case prioritization (TCP), which reorder the execution within a microbenchmark suite to detect larger performance changes sooner. Such techniques are either designed for unit tests and perform sub-par on microbenchmarks or require complex performance models, drastically reducing their potential application. In this paper, we empirically evaluate single- and multi-objective search-based microbenchmark prioritization techniques to understand whether they are more effective and efficient than greedy, coverage-based techniques. For this, we devise three search objectives, i.e., coverage to maximize, coverage overlap to minimize, and historical performance change detection to maximize. We find that search algorithms (SAs) are only competitive with but do not outperform the best greedy, coverage-based baselines. However, a simple greedy technique utilizing solely the performance change history (without coverage information) is equally or more effective than the best coverage-based techniques while being considerably more efficient, with a runtime overhead of less than 1%. These results show that simple, non-coverage-based techniques are a better fit for microbenchmarks than complex coverage-based techniques.

cs.SE↗

arXiv2024

Search-based Automated Program Repair of CPS Controllers Modeled in Simulink-Stateflow

Stateflow models are widely used in the industry to model the high-level control logic of Cyber-Physical Systems (CPSs) in Simulink--the defacto CPS simulator. Many approaches exist to test Simulink models, but once a fault is detected, the process to repair it remains manual. Such a manual process increases the software development cost, making it paramount to develop novel techniques that reduce this cost. Automated Program Repair (APR) techniques can significantly reduce the time for fixing bugs by automatically generating patches. However, current approaches face scalability issues to be applicable in the CPS context. To deal with this problem, we propose an automated search-based approach called FlowRepair, explicitly designed to repair Stateflow models. The novelty of FlowRepair includes, (1) a new algorithm that combines global and local search for patch generation; (2) a definition of novel repair objectives (e.g., the time a fault remained active) specifically designed for repairing CPSs; and (3) a set of mutation operators to repair Stateflow models automatically. We evaluated FlowRepair with three different case study systems and a total of nine faulty stateflow models. Our experiments suggest that (1) Flo wRepaircan fix bugs in stateflow models, including models with multiple faults; (2) FlowRepair surpasses or performs similarly to a baseline APR technique inspired by a well-known CPS program repair approach. Besides, we provide both a replication package and a live repository, paving the way towards the APR of CPSs modeled in Simulink.

cs.SE↗

arXiv2024

Pretrain, Prompt, and Transfer: Evolving Digital Twins for Time-to-Event Analysis in Cyber-physical Systems

Cyber-Physical Systems (CPSs), e.g., elevator systems and autonomous driving systems, are progressively permeating our everyday lives. To ensure their safety, various analyses need to be conducted, such as anomaly detection and time-to-event analysis (the focus of this paper). Recently, it has been widely accepted that digital Twins (DTs) can serve as an efficient method to aid in the development, maintenance, and safe and secure operation of CPSs. However, CPSs frequently evolve, e.g., with new or updated functionalities, which demand their corresponding DTs be co-evolved, i.e., in synchronization with the CPSs. To that end, we propose a novel method, named PPT, utilizing an uncertainty-aware transfer learning for DT evolution. Specifically, we first pretrain PPT with a pretraining dataset to acquire generic knowledge about the CPSs, followed by adapting it to a specific CPS with the help of prompt tuning. Results highlight that PPT is effective in time-to-event analysis in both elevator and ADSs case studies, on average, outperforming a baseline method by 7.31 and 12.58 in terms of Huber loss, respectively. The experiment results also affirm the effectiveness of transfer learning, prompt tuning and uncertainty quantification in terms of reducing Huber loss by at least 21.32, 3.14 and 4.08, respectively, in both case studies.

cs.SE↗