Software Engineering (cs.SE)

Abstract: Scenario-based testing is envisioned as a key approach for the safety assurance of autonomous vehicles. In scenario-based testing, relevant (driving) scenarios are the basis of tests. Many recent works focus on specification, variation, generation and execution of individual scenarios. In this work, we address the open challenges of classifying sets of scenarios and measuring coverage of theses scenarios in recorded test drives. Technically, we define logic-based classifiers that compute features of scenarios on complex data streams and combine these classifiers into feature trees that describe sets of scenarios. We demonstrate the expressiveness and effectiveness of our approach by defining a scenario classifier for urban driving and evaluating it on data recorded from simulations.

Abstract: Benchmarks are among the main drivers of progress in software engineering research, especially in software testing and debugging. However, current benchmarks in this field could be better suited for specific research tasks, as they rely on weak system oracles like crash detection, come with few unit tests only, need more elaborative research, or cannot verify the outcome of system tests. Our Tests4Py benchmark addresses these issues. It is derived from the popular BugsInPy benchmark, including 30 bugs from 5 real-world Python applications. Each subject in Tests4Py comes with an oracle to verify the functional correctness of system inputs. Besides, it enables the generation of system tests and unit tests, allowing for qualitative studies by investigating essential aspects of test sets and extensive evaluations. These opportunities make Tests4Py a next-generation benchmark for research in test generation, debugging, and automatic program repair.