July 29, 2019
Virtual test bed automation and adaptive exploration of driving scenarios using pSeven
Industry: Automotive | Product: pSeven | Company: FDTech
Introduction
FDTech GmbH is developing the simulation-based testing environment of automated driving systems. Testing in a virtual environment has several advantages compared to real world testing:
- Large scalability
- Exact determination of testing scenario
- Controllable variation of environment by defined parameters
- Exact repetition of test points possible
Virtual test beds imitate a whole variety of real-world driving scenarios, including the environment, sensor and control systems, testing car and external objects dynamics.
Virtual driving test beds allow to consider huge variety of scenarios
Described by the set of parameters, such driving scenarios are suitable for automated parametric analysis, i.e design of experiments methods.
However, in order to perform such extensive studies, the virtual test bed should be controlled by external tool to update the configuration, control the execution and analyze the results. Massive parallel study should be possible in order to reduce the overall testing time.
Number of parameters in consideration is huge, so special techniques should be used to cover test spaces efficiently and to iteratively cut out non-interesting areas of test spaces. In this study, we applied the Adaptive DoE methods in pSeven to focus on the most important parts of design space. Another goal is to cover remaining test space with continuous predictive (approximation) models in order to be able to predict the responses with acceptable accuracy for any driving scenario.
1. Virtual Test Bed Automation
Challenges
To enable scalable test beds:
- Test bed automation layer that schedules instances of virtual ECU, simulation etc. must be controlled externally
- All testing activities must be traceable and reusable
- All steps of test preparation, execution and analysis must be automated
Simulation is described by scenario XML files describing environment and other objects, as well as initial values. All entries should be exposed as potential variables for DoE study.
Solution
Fully automated workflow was successfully created in pSeven. It allows modifying xml. files with scenario setup, launching the virtual simulation and tracking the execution, collecting, storing and reusing the data. pSeven workflow is running Test automation layer (TAL), which:
- controls simulation and other tools that are needed for test execution
- obtains parameter settings from pSeven and generates executable test cases
- records all measures from simulation and virtual ECU and calculates usable responses
- responses are read by pSeven
pSeven workflow for virtual test bed automation and driving scenarios exploration
Once created, such automated chain enables the whole variety of DoE studies. In the example above, we perform two DoE studies in a single run one after another in order to use the data of the preliminary study in further detailed one.
The results are stored in pSeven database and can be accessed later on or exported to neutral format.
Benefit
Full automation of the virtual testing and DoE studies over it together with flexible setup of the study itself allows performing massive parallel virtual testing. Reuse of such typical workflows will reduce the effort during further testing campaings. These workflows can be used as a part of a bigger system or even generated automatically for the given task.
2. Adaptive Exploration of Driving Scenarios
Challenges
Number of driving scenarios increases exponentially with number of parameters in consideration. Testing every combination is
- not possible, even with roughly gridded parameters
- not needed, because large parameter areas would not cause differences in system behavior.
Therefore, some advanced DoE methodology is required to focus on most interesting combinations of parameters and design areas. The final goal is to explore those design areas in more detail and to construct an accurate predictive model of the responses.
Solution
We considered the following test scenario:
- Ego vehicle is driving on the right lane with velocity (parameter 1)
- Co (Player) is initially driving on the left lane with another velocity (parameter 2)
- Simulation starts when Co is at certain distance in front of Ego (parameter 3)
- If a certain distance is reached (parameter 4), Co is doing a lane change (cuts in).
Two responses are considered: Dive-In-Depth (minimal safety distance measure) and Maximal Acceleration. Depending on the nature of the response, the predictive models built on uniform DoE samples demonstrate different accuracy:
Dive-in-Depth (left) and Maximum Acceleration (right) prediction
The Maximum Acceleration response has discontinuous behavior, and the uniform DoE does not provide enough information on the shape of the function. Moreover, there are ranges in the response values, which are not useful for the goals of the study (safe areas). Therefore, in order to improve the coverage of the important domain, the Adaptive DoE technique was used with the previous results as initial sample. Adaptive DoE allows specifying the particular response as a target one, and the technique will place more testing points in the areas of most significant changes of such response. This technique might help to reveal more interesting scenarios with a limited number of test runs.
However, it can be noticed from the scatter plots that there might be responses with severely non-linear and non-smooth behavior, when even adaptive design of experiments techniques does not allow obtaining a suitable approximation model.
For such case, special approximation techniques could be used to complement the designs exploration routine. The MoA – Mixture of Approximations – is a technique, which was specially developed to handle the responses with different behavior in different regions of the design space, like steps, peaks etc.
Based on the idea of smooth combining of different techniques, MoA can significantly improve the prediction even for discontinues responses. Moreover, the technique can accept the initial approximation model and improve the prediction quality, acting as a second step in building the accurate response surface in special cases.
Scatter (left) and Quantile (right) plots for standard technique (red) and MoA (green)
For considered response, the RMS error of the MoA model is almost four times lower, compared to the single-technique (HDA) model.
Benefit
Flexible setup of DoE studies in pSeven allows switching from standard uniform sampling to more sophisticated adaptive techniques in the case it is required to track special response behavior with a limited number of testing points.
The predictive modeling toolkit enables fast and easy model training using the collected datasets. To complement the adaptive design exploration, the effective methods to approximate severely non-linear responses can be used, resulting in accurate approximations even for the most complex responses.
By Anton Saratov, Head of Application Engineering, DATADVANCE and Leonhard Herrmann, Development Engineer, FDTech GmbH
About FDTech
FDTech GmbH are ADAS and AD Experts with international customer experience (i.e. VW Group, BMW, RSA, PSA, FCA). Company’s encouraged teams have vast experience in automotive engineering and high-tech industries, competent and motivated employees for projects in the areas of driver assistance systems and automated driving:
- Architecture: merging monolithic vehicle architectures with flexible IT architectures into dynamic self-learning solutions
- Algorithms: edge computing for efficient data processing, machine learning for permanent adaptation and enhancement
- Agility: speed by agile development methods in small cross-functional teams
- Advisory: transformation of top level ideas into real functions through early concepts with high level functional prototypes
