arXiv subjects

Markus Steimle

Publications and source records attributed to Markus Steimle.

5 recordsLinked to original sources

arXiv2024

A Concept for Semi-Automatic Configuration of Sufficiently Valid Simulation Setups for Automated Driving Systems

As simulation is increasingly used in scenario-based approaches to test Automated Driving Systems, the credibility of simulation results is a major concern. Arguably, credibility depends on the validity of the simulation setup and simulation models. When selecting appropriate simulation models, a trade-off must be made between validity, often connected to the model's fidelity, and cost of computation. However, due to the large number of test cases, expert-based methods to create sufficiently valid simulation setups seem infeasible. We propose using design contracts in order to semi-automatically compose simulation setups for given test cases from simulation models and to derive requirements for the simulation models, supporting separation of concerns between simulation model developers and users. Simulation model contracts represent their validity domains by capturing a validity guarantee and the associated operating conditions in an assumption. We then require the composition of the simulation model contracts to refine a test case contract. The latter contract captures the operating conditions of the test case in its assumption and validity requirements in its guarantee. Based on this idea, we present a framework that supports the compositional configuration of simulation setups based on the contracts and a method to derive runtime monitors for these simulation setups.

eess.SY↗

arXiv2022

Toward Generating Sufficiently Valid Test Case Results: A Method for Systematically Assigning Test Cases to Test Bench Configurations in a Scenario-Based Test Approach for Automated Vehicles

To successfully launch automated vehicles into the consumer market, there must be credible proof that the vehicles will operate safely. However, finding a method to validate the vehicles' safe operation is a challenging problem. While scenario-based test approaches seem to be possible solutions, they require execution of a large number of test cases. Several test benches, ranging from actual test vehicles to partly or fully simulated environments, are available to execute these test cases. Each test bench provides different elements, which in turn, have different parameters and parameter ranges. The composition of elements with their specific parameter values at a specific test bench that is used to execute a test case is referred to as a test bench configuration. However, selecting the most suitable test bench configuration is difficult. The selected test bench configuration determines whether the execution of a specific test case provides sufficiently valid test case results with respect to the intended purpose, for example, validating a vehicle's safe operation. The effective and efficient execution of a large number of test cases requires a method for systematically assigning test cases to the most suitable test bench configuration. Based on a proposed method for classifying test bench configurations, we propose and illustrate a method for systematically assigning test cases to test bench configurations in a scenario-based test approach for automated vehicles. This assignment method allows for the effective and efficient execution of a large number of test cases while generating sufficiently valid test case results.

cs.SE↗

arXiv2021

A taxonomy for quality in simulation-based development and testing of automated driving systems

Ensuring the quality of automated driving systems is a major challenge the automotive industry is facing. In this context, quality defines the degree to which an object meets expectations and requirements. Especially, automated vehicles at SAE level 4 and 5 will be expected to operate safely in various contexts and complex situations without misconduct. Thus, a systematic approach is needed to show their safe operation. A way to address this challenge is simulation-based testing as pure physical testing is not feasible. During simulation-based testing, the data used to evaluate the actual quality of an automated driving system are generated using a simulation. However, to rely on these simulation data, the overall simulation, which also includes its simulation models, must provide a certain quality level. This quality level depends on the intended purpose for which the generated simulation data should be used. Therefore, three categories of quality can be considered: quality of the automated driving system and simulation quality, consisting of simulation model quality and scenario quality. Hence, quality must be determined and evaluated in various process steps in developing and testing automated driving systems, the overall simulation, and the simulation models used for the simulation. In this paper, we propose a taxonomy to serve a better understanding of the concept of quality in the development and testing process to have a clear separation and insight where further testing is needed -- both in terms of automated driving systems and simulation, including their simulation models and scenarios used for testing.

cs.SE↗

arXiv2021

Toward a Consistent Taxonomy for Scenario-Based Development and Test Approaches for Automated Vehicles: A Proposal for a Structuring Framework, a Basic Vocabulary, and its Application

Ensuring and validating the safe operation of automated vehicles are key challenges for their market launch. Scenario-based development and test approaches are currently being pursued as possible solutions. An essential prerequisite for researching, applying, and standardizing these approaches is a consistent and agreed-upon taxonomy. This taxonomy must include relevant terms, their description, and the relationships between the respective terms. To the best of our knowledge, such a taxonomy does not yet exist, which often leads to misunderstandings, for example, in coordination processes and discussions. This publication contributes to this taxonomy. For this purpose, we propose a framework for structuring the taxonomy. Within this framework, we propose a basic vocabulary by identifying and describing terms that we consider particularly relevant for an overview of such scenario-based development and test approaches. Additionally, we visualize the proposed terms and the relationships between these terms with UML diagrams and explain the application of the proposed basic vocabulary with an example.

cs.SE↗

arXiv2019

A Method for Classifying Test Bench Configurations in a Scenario-Based Test Approach for Automated Vehicles

The introduction of automated vehicles demands a way to prove their safe operation. However, validating the safety of automated vehicles is still an unsolved problem. While the scenario-based test approach seems to provide a possible solution, it requires the execution of a high amount of test cases. Several test benches, from actual test vehicles to partly or fully simulated environments, are available, but choosing the optimal test bench, e.g. in terms of required execution time and costs, is a difficult task. Every test bench provides different elements, e.g. simulation models which can be used for test case execution. The composition of elements at a specific test bench is called test bench configuration. This test bench configuration determines the actual performance of a test bench and, therefore, whether the run of a particular test case provides valid test case results with respect to the intended purpose, e.g. a safety validation. For an effective and efficient test case execution, a method is required to assign test cases to the most appropriate test bench configuration. Therefore, it is indispensable to have a method to classify test bench configurations in a clear and reproducible manner. In this paper, we propose a method for classifying test benches and test bench configurations and illustrate the classification method with some examples. The classification method serves as a basis for a systematic assignment of test cases to test bench configurations which allows for an effective and efficient test case execution.

eess.SP↗