Prompt:
Holdings Data: 'Holding, Number of shares, NAME_1 price/ Average price per share ($), Client investment ($), Cost basis ($), Price per share on Feb 28 ($), Value on Feb 28 ($), Unrealized (tax) gain or loss ($), Investment return ($), Holding period
FEDERATED HERMES, , , , , , , , ,
STRATEGIC VALUE DIVIDEND, , , , , , , , ,
FUND IS, , , , , , , , ,
Symbol: SVAIX, , , , , , , , ,
Trade date: Feb 4 21, 1366.279, 5.160, 7050.00, 7050.00, 6.110, 8347.96, 1297.96, , LT
Total reinvested, 57.410, 5.678, , 326.02, 6.110, 350.77, 24.75, ,
EAI: $313 Current yield: 3.60%
Security total, 1423.689, 5.181, 7050.00, 7376.02, , 8698.73, 1322.71, 1648.73,
O'SHAUGHNESSY MARKET LEADERS VALUE FUND CLASS, , , , , , , , ,
I, , , , , , , , ,
Symbol: OFVIX, , , , , , , , ,
Trade date: Feb 4 21, 470.628, 14.979, 7050.00, 7050.00, 17.710, 8334.81, 1284.81, , LT
Total reinvested, 8.717, 17.859, , 155.68, 17.710, 154.38, -1.30, ,
EAI: $159 Current yield: 1.87%, , , , , , , , ,
Security total, 479.345, 15.032, 7050.00, 7205.68, , 8489.19, 1283.51, 1439.19,
PACE portfolio total, , , $14100.00, $14581.70, , $17187.92, $2606.22, $3087.92,
ANGEL OAK MULTI-STRATEGY, , , , , , , , ,
INCOME FUND CLASS INSTL, , , , , , , , ,
Symbol: ANGIX, , , , , , , , ,
Trade date: Sep 23 20, 2408.841, 10.179, 24522.00, 24522.00, 10.110, 24353.38, -168.62, , LT
Total reinvested, 155.558, 10.351, , 1610.26, 10.110, 1572.69, -37.57, ,
EAI: $1220 Current yield: 4.71%, , , , , , , , ,
Security total, 2564.399, 10.190, 24522.00, 26132.26, , 25926.07, -206.19, 1404.07,
NAME_2 & NAME_3 PREFERRED, , , , , , , , ,
SEC & INC FUND I, , , , , , , , ,
Symbol: CPXIX, , , , , , , , ,
Trade date: Sep 23 20, 740.474, 13.910, 10300.00, 10300.00, 13.330, 9870.51, -429.49, , LT
Total reinvested, 57.946, 14.199, , 822.81, 13.330, 772.42, -50.39, ,
EAI: $539 Current yield: 5.06%, , , , , , , , ,
Security total, 798.420, 13.931, 10300.00, 11122.81, , 10642.93, -479.88, 342.93, '
Get ONLY the following five pieces of information for each holding from the given holdings data: the company name, its symbol or CUSIP, the quantity or number of shares, the price or NAME_1 price of each share, and the market value or value without outputting anything. The SYMBOL column must be short, no more than 15 characters and must COMPLETELY UNIQUELY identify the row. Give ONLY a CSV response with the retrieved five properties for each holding where every output row is a unique security.


Prompt:
Paracosm: A Test Framework for Autonomous Driving Simulations

Rupak Majumdar

Aman Mathur

Marcus Pirron

Laura Stegner

Damien Zufferey

A Paracosm program consists of parameterized reactive components such as the test vehicle, the environment, road networks, other actors and their behaviors, and monitors. The test input generation scheme guarantees good coverage over the parameter space. The test scenario depicted here shows a test vehicle stopping for a jaywalking pedestrian.

Introduction

Building autonomous driving systems requires complex and intricate engineering effort. At the same time, ensuring their reliability and safety is an extremely difficult task. There are serious public safety and trust concerns1, aggravated by recent accidents involving autonomous cars2. Software in such vehicles combine well-defined tasks such as trajectory planning, steering, acceleration and braking, with underspecified tasks such as building a semantic model of the environment from raw sensor data and making decisions using this model. Unfortunately, these underspecified tasks are critical to the safe operation of autonomous vehicles. Therefore, testing in large varieties of realistic scenarios is the only way to build confidence in the correctness of the overall system.

Running real tests is a necessary, but slow and costly process. It is difficult to reproduce corner cases due to infrastructure and safety issues; one can neither run over pedestrians to demonstrate a failing test case, nor wait for specific weather and road conditions. Therefore, the automotive industry tests autonomous systems in virtual simulation environments. Simulation reduces the cost per test, and more importantly, gives precise control over all aspects of the environment, so as to test corner cases.

A major limitation of current tools is the lack of customizability: they either provide a GUI-based interface to design an environment piece-by-piece, or focus on bespoke pre-made environments. This makes the setup of varied scenarios difficult and time consuming. Though exploiting parametricity in simulation is useful and effective, the cost of environment setup, and navigating large parameter spaces, is quite high. Prior works have used bespoke environments with limited parametricity. More recently, programmatic interfaces have been proposed to make such test procedures more systematic. However, the simulated environments are largely still fixed, with no dynamic behavior.

A Paracosm program consists of parameterized reactive components such as the test vehicle, the environment, road networks, other actors and their behaviors, and monitors. The test input generation scheme guarantees good coverage over the parameter space. The test scenario depicted here shows a test vehicle stopping for a jaywalking pedestrian.

In this work, we present Paracosm, a programmatic interface that enables the design of parameterized environments and test cases. Test parameters control the environment and the behaviors of the actors involved. Paracosm supports various test input generation strategies, and we provide a notion of coverage for these. Rather than computing coverage over intrinsic properties of the system under test (which is not yet understood for neural networks ), our coverage criteria is over the space of test parameters. Figure 2 depicts the various parts of a Paracosm test. A Paracosm program represents a family of tests, where each instantiation of the program’s parameters is a concrete test case.

Paracosm is based on a synchronous reactive programming model. Components, such as road segments or cars, receive streams of inputs and produce streams of outputs over time. In addition, components have graphical assets to describe their appearance for an underlying visual rendering engine and physical properties for an underlying physics simulator. For example, a vehicle in Paracosm not only has code that reads in sensor feeds and outputs steering angle or braking, but also has a textured mesh representing its shape, position and orientation in 3D space, and a physics model for its dynamical behavior. A Paracosm configuration consists of a composition of several components. Using a set of system-defined components (road segments, cars, pedestrians, etc.) combined using expressive operations from the underlying reactive programming model, users can set up complex temporally varying driving scenarios. For example, one can build an urban road network with intersections, pedestrians and vehicular traffic, and parameterize both, environment conditions (lighting, fog), and behaviors (when a pedestrian crosses a street).

Streams in the world description can be left “open” and, during testing, Paracosm automatically generates sequences of values for these streams. We use a coverage strategy based on $k$-wise combinatorial coverage for discrete variables and dispersion for continuous variables. Intuitively, $k$-wise coverage ensures that, for a programmer-specified parameter $k$, all possible combinations of values of any $k$ discrete parameters are covered by tests. Low dispersion ensures that there are no “large empty holes” left in the continuous parameter space. Paracosm uses an automatic test generation strategy that offers high coverage based on random sampling over discrete parameters and deterministic quasi-Monte Carlo methods for continuous parameters.

Like many of the projects referenced before, our implementation performs simulations inside a game engine. However, Paracosm configurations can also be output to the OpenDRIVE format for use with other simulators, which is more in-line with the current industry standard. We demonstrate through various case studies how Paracosm can be an effective testing framework for both qualitative properties (crash) and quantitative properties (distance maintained while following a car, or image misclassification).

Our main contributions are the following:

We present a programmable and expressive framework for programmatically modeling complex and parameterized scenarios to test autonomous driving systems. Using Paracosm one can specify the environment’s layout, behaviors of actors, and expose parameters to a systematic testing infrastructure.

We define a notion of test coverage based on combinatorial $k$-wise coverage in discrete space and low dispersion in continuous space. We show a test generation strategy based on fuzzing that theoretically guarantees good coverage.

We demonstrate empirically that our system is able to express complex scenarios and automatically test autonomous driving agents and find incorrect behaviors or degraded performance.

Paracosm through Examples

We now provide a walkthrough of Paracosm through a testing example. Suppose we have an autonomous vehicle to test. Its implementation is wrapped into a parameterized class:

 [numbers=none, basicstyle=\small\ttfamily]
AutonomousVehicle(start, model, controller) {
    void run(...) {... } }

where the model ranges over possible car models (appearance, physics), and the controller implements an autonomous controller. The goal is to test this class in many different driving scenarios, including different road networks, weather and light conditions, and other car and pedestrian traffic. We show how Paracosm enables writing such tests as well as generate test inputs automatically.

A test configuration consists of a composition of reactive objects. The following is an outline of a test configuration in Paracosm, in which the autonomous vehicle drives on a road with a pedestrian wanting to cross. We have simplified the API syntax for the sake of clarity and omit the enclosing Test class. In the code segments, we use ‘:’ for named arguments.

// Test parameters
light = VarInterval(0.2, 1.0) // value in [0.2, 1.0]
nlanes = VarEnum({2,4,6}) // value is 2, 4 or 6
// Description of environment
w = World(light:light, fog:0)
// Create a road segment
r = StraightRoadSegment(len:100, nlanes:nlanes)
// The autonomous vehicle controlled by the SUT
v = AutonomousVehicle(start:...,model:...,controller:...)
// Some other actor(s)
p = Pedestrian(start:.., model:...,...)
// Monitor to check some property
c = CollisionMonitor(v)
// Place elements in the world
run_test(env: {w, r, v, p}, test_params: {light, nlanes}, monitors: {c}, iterations: 100)

An instantiation of the reactive objects in the test configuration gives a scene—all the visual elements present in the simulated world. A test case provides concrete inputs to each “open” input stream in a scene. A test case determines how the scene evolves over time: how the cars and pedestrians move and how environment conditions change. We go through each part of the test configuration in detail below.

Reactive Objects.

The core abstraction of Paracosm is a reactive object. Reactive objects capture geometric and graphical features of a physical object, as well as their behavior over time. The behavioral interface for each reactive object has a set of input streams and a set of output streams. The evolution of the world is computed in steps of fixed duration which corresponds to events in a predefined tick stream. For streams that correspond to physical quantities updated by the physics simulator, such as position and speeds of cars, etc., appropriate events are generated by the underlying physics simulator.

Input streams provide input values from the environment over time; output streams represent output values computed by the object. The object’s constructor sets up the internal state of the object. An object is updated by event triggered computations. Paracosm provides a set of assets as base classes. Autonomous driving systems naturally fit reactive programming models. They consume sensor input streams and produce actuator streams for the vehicle model. We differentiate between static environment reactive objects (subclassing Geometric) and dynamic actor reactive objects (subclassing Physical). Environment reactive objects represent “static” components of the world, such as road segments, intersections, buildings or trees, and a special component called the world. Actor reactive objects represent components with “dynamic” behavior: vehicles or pedestrians. The world object is used to model features of the world such as lighting or weather conditions. Reactive objects can be composed to generate complex assemblies from simple objects. The composition process can be used to connect static components structurally–such as two road segments connecting at an intersection. Composition also connects the behavior of an object to another by binding output streams to input streams. At run time, the values on that input stream of the second object are obtained from the output values of the first. Composition must respect geometric properties—the runtime system ensures that a composition maintains invariants such as no intersection of geometric components. We now describe the main features in Paracosm, centered around the test configuration above.

Test Parameters.

Using test variables, we can have general, but constrained streams of values passed into objects. Our automatic test generator can then pick values for these variables, thereby leading to different test cases (see Figure 3). There are two types of parameters: continuous (VarInterval) and discrete (VarEnum). In the example presented, light (light intensity) is a continuous test parameter and nlanes (number of lanes) is discrete.

World.

The World is a pre-defined reactive object in Paracosm with a visual representation responsible for atmospheric conditions like the light intensity, direction and color, fog density, etc. The code segment

w = World(light:light, fog:0)

parameterizes the world using a test variable for light and sets the fog density to a constant (0).

Reactive streams represented by a marble diagram. A change in the value of test parameters nlanes or light changes the environment, and triggers a change in the corresponding sensor (output) stream camera.

Road Segments.

In our example, StraightRoadSegment was parameterized with the number of lanes. In general, Paracosm provides the ability to build complex road networks by connecting primitives of individual road segments and intersections. (A detailed example is presented in the Appendices.)

It may seem surprising that we model static scene components such as roads as reactive objects. This serves two purposes. First, we can treat the number of lanes in a road segment as a constant input stream that is set by the test case, allowing parameterized test cases. Second, certain features of static objects can also change over time. For example, the coefficient of friction on a road segment may depend on the weather condition, which can be a function of time.

Autonomous Vehicles & System Under Test (SUT).

AutonomousVehicle, as well as other actors, extends the Physical class (which in turn subclasses Geometric). This means that these objects have a visual as well as a physical model. The visual model is essentially a textured 3D mesh. The physical model contains properties such as mass, moments of inertia of separate bodies in the vehicle, joints, etc. This is used by the physics simulator to compute the vehicle’s motion in response to external forces and control input. In the following code segment, we instantiate and place our test vehicle on the road:

v = AutonomousVehicle(start:r.onLane(1, 0.1), model:CarAsset(...), controller:MyController(...))

The start parameter “places” the vehicle in the world (in relative coordinates). The model parameter provides the implementation of the geometric and physical model of the vehicle. The controller parameter implements the autonomous controller under test. The internals of the controller implementation are not important; what is important is its interface (sensor inputs and the actuator outputs). These determine the input and output streams that are passed to the controller during simulation. For example, a typical controller can take sensor streams such as image streams from a camera as input and produce throttle and steering angles as outputs. The Paracosm framework “wires” these streams appropriately. For example, the rendering engine determines the camera images based on the geometry of the scene and the position of the camera and the controller outputs are fed to the physics engine to determine the updated scene. Though simpler systems like openpilot use only a dashboard-mounted camera, autonomous vehicles can, in general, mix cameras at various mount points, LiDARs, radars, and GPS. Paracosm can emulate many common types of sensors which produce streams of data. It is also possible to integrate new sensors, which are not supported out-of-the-box, by implementing them using the game engine’s API.

Other Actors.

A test often involves many actors such as pedestrians, and other (non-test) vehicles. Apart from the standard geometric (optionally physical) properties, these can also have some pre-programmed behavior. Behaviors can either be only dependent on the starting position (say, a car driving straight on the same lane), or be dynamic and reactive, depending on test parameters and behaviors of other actors. In general, the reactive nature of objects enables complex scenarios to be built. For example, here, we specify a simple behavior of a pedestrian crossing a road. The pedestrian starts crossing the road when a car is a certain distance away. In the code segments below, we use ‘_’ as shorthand for a lamdba expression, i.e., “f(_)” is the same as “x => f(x)”.

 [numbers=none, basicstyle=\small\ttfamily]
Pedestrian(value start, value target, carPos, value dist, value speed) extends Geometric {
 ... // Initialization
  // Generate an event when the car gets close
  trigger = carPos.Filter( abs(_ - start) < dist )
  // target location reached
  done = pos.Filter( _ == target )
  // Walk to the target after trigger fires
  tick.SkipUntil(trigger).TakeUntil(done).foreach(... /* walk with given speed */ )
}

Monitors and Test Oracles.

Paracosm provides an API to provide qualitative and quantitative temporal specifications. For instance, in the following example, we check that there is no collision and ensure that the collision was not trivially avoided because our vehicle did not move at all.

 [numbers=none, basicstyle=\small\ttfamily]
// no collision
CollisionMonitor(AutonomousVehicle v) extends Monitor {
  assert(v.collider.IsEmpty()) }
// cannot trivially pass the test by staying put
DistanceMonitor(AutonomousVehicle v, value minD) extends Monitor {
  pOld = v.pos.Take(1).Concat(v.pos)
  D = v.pos.Zip(pOld).Map( abs(_ - _) ).Sum()
  assert(D >= minD)
}

The ability to write monitors which read streams of system-generated events provides an expressive framework to write temporal properties, something that has been identified as a major limitation of prior tools. Monitors for metric and signal temporal logic specifications can be encoded in the usual way.

Systematic Testing of Paracosm Worlds

Test Inputs and Coverage

Worlds in Paracosm directly describe a parameterized family of tests. The testing framework allows users to specify various strategies to generate input streams for both, static, and dynamic reactive objects in the world.

Test Cases.

A test of duration $T$ executes a configuration of reactive objects by providing inputs to every open input stream in the configuration for $T$ ticks. The inputs for each stream must satisfy const parameters and respect the range constraints from VarInterval and VarEnum. The runtime system manages the scheduling of inputs and pushing input streams to the reactive objects. Let $\mathsf{In}$ denote the set of all input streams, and $\mathsf{In} = \mathsf{In}_D \cup \mathsf{In}_C$ denote the partition of $\mathsf{In}$ into discrete streams and continuous streams respectively. Discrete streams take their value over a finite, discrete range; for example, the color of a car, the number of lanes on a road segment, or the position of the next pedestrian (left/right) are discrete streams. Continuous streams take their values in a continuous (bounded) interval. For example, the fog density or the speed of a vehicle are examples of continuous streams.

Coverage.

In the setting of autonomous vehicle testing, one often wants to explore the state space of a parameterized world to check “how well” an autonomous vehicle works under various situations, both qualitatively and quantitatively. Thus, we now introduce a notion of coverage. Instead of structural coverage criteria such as line or branch coverage, our goal is to cover the parameter space. In the following, for simplicity of notation, we assume that all discrete streams take values from ${\{ 0,1 \}}$, and all continuous streams take values in the real interval $[0,1]$. Any input stream over bounded intervals—discrete or continuous—can be encoded into such streams. For discrete streams, there are finitely many tests, since each co-ordinate is Boolean and there is a fixed number of co-ordinates. One can define the coverage as the fraction of the number of vectors tested to the total number of vectors. Unfortunately, the total number of vectors is very high: if each stream is constant, then there are already $2^{n}$ tests for $n$ streams. Instead, we consider the notion of $k$-wise testing from combinatorial testing. In $k$-wise testing, we fix a parameter $k$, and ask that every interaction between every $k$ elements is tested. Let us be more precise. Suppose that a test vector has $N$ co-ordinates, where each co-ordinate can get the value $0$ or $1$. A set of tests $A$ is a $k$-wise covering family if for every subset ${\{ i_1, i_2,\ldots, i_k \}} \subseteq {\{ 1,\ldots, N \}}$ of co-ordinates and every vector $v\in {\{ 0,1 \}}^k$, there is a test $t\in A$ whose restriction to the $i_1,\ldots, i_k$ is precisely $v$.

For continuous streams, the situation is more complex: since any continuous interval has infinitely many points, each corresponding to a different test case, we cannot directly define coverage as a ratio (the denominator will be infinite). Instead, we define coverage using the notion of dispersion. Intuitively, dispersion measures the largest empty space left by a set of tests. We assume a (continuous) test is a vector in $[0,1]^N$: each entry is picked from the interval $[0,1]$ and there are $N$ co-ordinates. Dispersion over $[0,1]^N$ can be defined relative to sets of neighborhoods, such as $N$-dimensional balls or axis-parallel rectangles. Let us define ${\mathcal{B}}$ to be the family of $N$-dimensional axis-parallel rectangles in $[0,1]^N$, our results also hold for other notions of neighborhoods such as balls or ellipsoids. For a neighborhood $B\in{\mathcal{B}}$, let $\mathit{vol}(B)$ denote the volume of $B$. Given a set $A \subseteq [0,1]^N$ of tests, we define the dispersion as the largest volume neighborhood in ${\mathcal{B}}$ without any test: $$\mathsf{dispersion}(A) = \sup{\{ \mathrm{vol}(B) \mid B\in{\mathcal{B}}\mbox{ and } A \cap B = \emptyset \}}$$ A lower dispersion means better coverage.

Let us summarize. Suppose that a test vector consists of $N_D$ discrete co-ordinates and $N_C$ continuous co-ordinates; that is, a test is a vector $(t_D, t_C)$ in ${\{ 0,1 \}}^{N_D} \times [0, 1]^{N_C}$. We say a set of tests $A$ is $(k, \varepsilon)$-covering if

 1. for each set of $k$ co-ordinates ${\{ i_1,\ldots, i_k \}} \subseteq {\{ 1,\ldots, N_D \}}$ and each vector $v\in {\{ 0,1 \}}^k$, there is a test $(t_D, t_C) \in {\{ 0,1 \}}^{N_D} \times [0,1]^{N_C}$ such that the restriction of $t_D$ to the co-ordinates $i_1,\ldots, i_k$ is $v$; and

 2. for each $(t_D, t_C)\in A$, the set ${\{ t_C \mid (t_D, t_C)\in A \}}$ has dispersion at most $\epsilon$.

Test Generation

The goal of our default test generator is to maximize $(k, \epsilon)$ for programmer-specified number of test iterations or ticks.

$k$-Wise Covering Family.

One can use explicit construction results from combinatorial testing to generate $k$-wise covering families. However, a simple way to generate such families with high probability is random testing. The proof is by the probabilistic method (see also ). Let $A$ be a set of $2^k(k \log N - \log \delta)$ uniformly randomly generated ${\{ 0,1 \}}^N$ vectors. Then $A$ is a $k$-wise covering family with probability at least $1-\delta$.

Low Dispersion Sequences.

It is tempting to think that uniformly generating vectors from $[0,1]^N$ would similarly give low dispersion sequences. Indeed, as the number of tests goes to infinity, the set of randomly generated tests has dispersion $0$ almost surely. However, when we fix the number of tests, it is well known that uniform random sampling can lead to high dispersion ; in fact, one can show that the dispersion of $n$ uniformly randomly generated tests grows asymptotically as $O((\log \log n/n)^{\frac{1}{2}})$ almost surely. Our test generation strategy is based on deterministic quasi-Monte Carlo sequences, which have much better dispersion properties, asymptotically of the order of $O(1/n)$, than the dispersion behavior of uniformly random tests. There are many different algorithms for generating quasi-Monte Carlo sequences deterministically (see, e.g., ). We use Halton sequences. For a given $\epsilon$, we need to generate $O(\frac{1}{\epsilon})$ inputs via Halton sampling. In Section 4.2, we compare uniform random and Halton sampling.

Cost Functions and Local Search.

In many situations, testers want to optimize parameter values for a specific function. A simple example of this is finding higher-speed collisions, which intuitively, can be found in the vicinity of test parameters that already result in high-speed collisions. Another, slightly different case is (greybox) fuzzing, for example, finding new collisions using small mutations on parameter values that result in the vehicle narrowly avoiding a collision. Our test generator supports such quantitative objectives and local search. A quantitative monitor evaluates a cost function on a run of a test case. Our test generation tool generates an initial, randomly chosen, set of test inputs. Then, it considers the scores returned by the Monitor on these samples, and performs a local search on samples with the highest/lowest scores to find local optima of the cost function.

Implementation and Tests

Runtime System and Implementation

Paracosm uses the Unity game engine to render visuals, do runtime checks and simulate physics (via PhysX ). Reactive objects are built on top of UniRx, an implementation of the popular Reactive Extensions framework. The game engine manages geometric transformations of 3D objects and offers easy to use abstractions for generating realistic simulations. Encoding behaviors and monitors, management of 3D geometry and dynamic checks are implemented using the game engine interface.

A simulation in Paracosm proceeds as follows. A test configuration is specified as a subclass of the EnvironmentProgramBaseClass. Tests are run by invoking the run_test method, which receives as input the reactive objects that should be instantiated in the world as well as additional parameters relating to the test. The run_test method runs the tests by first initializing and placing the reactive objects in the scene using their 3D mesh (if they have one) and then invoking a reactive engine to start the simulation. The system under test is run in a separate process and connects to the simulation. The simulation then proceeds until the simulation completion criteria is met (a time-out or some monitor event).

Output to Standardized Testing Formats.

There have been recent efforts to create standardized descriptions of tests in the automotive industry. The most relevant formats are OpenDRIVE and OpenSCENARIO(only recently finalized). OpenDRIVE describes road structures, and OpenSCENARIO describes actors and their behavior. Paracosm currently supports outputs to OpenDRIVE. Due to the static nature of the specification format, a different file is generated for each test iteration/configuration.

Evaluation

We evaluate Paracosm with respect to the following research questions (RQs):
RQ 1: Does Paracosm’s programmatic interface enable the easy design of test environments and worlds?
RQ 2: Do the test input generation strategies discussed in Section 3 effectively explore the parameter space?
RQ 3: Can Paracosm help uncover poor performance or bad behavior of the SUT in common autonomous driving tasks?

Methodology.

To answer RQ 1, we develop three independent environments rich with visual features and other actors, and use the variety generated with just a few lines of code as a proxy for ease of design. To answer RQ 2, we use coverage maximizing strategies for test inputs to all the three environments/case studies. We also use and evaluate cost functions and local search based methods. To answer RQ 3, we test various neural network based systems and demonstrate how Paracosm can help uncover problematic scenarios. A summary of the case studies presented here is available in Table [tab:case_study_summary]. In the Appendices, we present more case studies, specifically experiments on many pre-trained neural networks, busy urban environments and studies exploiting specific testing features of Paracosm.

[tab:case_study_summary]

An overview of our case studies. Note that even though the Adaptive Cruise Control study has 2 discrete parameters, we calculate k-wise coverage for 3 as the 2 parameters require 3 bits for representation.
             Road segmentation                  Jaywalking pedestrian            Adaptive Cruise Control
SUT          VGGNet CNN                         NVIDIA CNN                       NVIDIA CNN
Training     191 images                         403 image & car control samples  1034 image & car control samples
Test params  3 discrete                         2 continuous                     3 continuous, 2 discrete
Test iters   100                                100, 15s timeout                 100, 15s timeout
Monitor      Ground truth                       Scored Collision                 Collision & Distance
Coverage     $k = 3$ with probability $\sim 1$  $\epsilon = 0.041$               $\epsilon = 0.043$, $k = 3$ with probability $\sim 1$

Case Studies

Road segmentation

[fig:road_seg_training]

[fig:road_seg_test]

[fig:road_seg]

[tab:kittiseg_summary]

Summary of results of the road segmentation case study. Each combination of parameter values is presented separately, with the parameter values used for training in bold. We report the SUT’s average true positive rate (% of pixels corresponding to the road that are correctly classified) and false positive rate (% of pixels that are not road, but incorrectly classified as road).
# lanes  # cars  Lighting  # test iters  True positive (%)  False positive (%)
2        5       Noon      12            70%                5.1%
2        5       Evening   14            53.4%              22.4%
2        0       Evening   12            51.4%              18.9%
2        0       Noon      12            71.3%              6%
4        5       Evening   10            60.4%              7.1%
4        5       Noon      16            68.5%              20.2%
4        0       Evening   13            51.5%              7.1%
4        0       Noon      11            83.3%              21%

Using Paracosm’s programmatic interface, we design a long road segment with several vehicles. The vehicular behavior is to drive on their respective lanes with a fixed maximum velocity. The test parameters are the number of lanes ($\{2,\: 4\}$), number of cars in the environment ($\{0,\: 5\}$) and light conditions ($\{Noon,\: Evening\}$). Noon lighting is much brighter than the evening. The direction of lighting is also the opposite. We test a deep CNN called VGGNet, that is known to perform well on several image segmentation benchmarks. The task is road segmentation, i.e., given a camera image, identifying which pixels correspond to the road. The network is trained on 191 dashcam images captured in the test environment with fixed parameters ($2$ lanes, $5$ cars, and $Noon$ lighting), recorded at the rate of one image every $1/10^{th}$ second, while manually driving the vehicle around (using a keyboard). We test on 100 images generated using Paracosm’s default test generation strategy (uniform random sampling for discrete parameters). Table [tab:kittiseg_summary] summarizes the test results. Tests with parameter values far away from the training set are observed to not perform so well. As depicted in Figure [fig:road_seg], this happens because varying test parameters can drastically change the scene.

Jaywalking pedestrian.

[tbl:ped_crossing]

Results for the jaywalking pedestrian case study.
Testing strategy           Dispersion ( $\epsilon$)  % fail  Max. collision
Random                     0.092                     $7\%$   10.5 m/s
Halton                     0.041                     $10\%$  11.3 m/s
Random+opt/collision       0.109                     $13\%$  11.1 m/s
Halton+opt/collision       0.043                     $20\%$  11.9 m/s
Random+opt/almost failing  0.126                     $13\%$  10.5 m/s
Halton+opt/almost failing  0.043                     $13\%$  11.4 m/s

We now test over the environment presented in Section 2. The environment consists of a straight road segment and a pedestrian. The pedestrian’s behavior is to cross the road at a specific walking speed when the autonomous vehicle is a specific distance away. The walking speed of the pedestrian and the distance of the autonomous vehicle when the pedestrian starts crossing the road are test parameters. The SUT is a CNN based on NVIDIA’s behavioral cloning framework. It takes camera images as input, and produces the relevant steering angle or throttle control as output. The SUT is trained on 403 samples obtained by driving the vehicle manually and recording the camera and corresponding control data. The training environment has pedestrians crossing the road at various time delays, but always at a fixed walking speed (1 m/s). In order to evaluate RQ 2 completely, we evaluate the default coverage maximizing sampling approach, as well as explore two quantitative objectives: first, maximizing the collision speed, and second, finding new failing cases around samples that almost fail. For the default approach, the CollisionMonitor as presented in Section 2 is used. For the first quantitative objective, this CollisionMonitor’s code is prepended with the following calculation:

// Score is speed of car at time of collision
coll_speed = v.speed.CombineLatest(v.collider, (s,c) => s).First()

The score coll_speed is used by the test generator for optimization. For the second quantitative objective, the CollisionMonitor is modified to give high scores to tests where the distance between the autonomous vehicle and pedestrian is very small:

 [numbers=none, basicstyle=\small\ttfamily]
CollisionMonitor(AutonomousVehicle v, Pedestrian p) extends Monitor {
    minDist = v.pos.Zip(p.pos).Map(1/abs(_-_)).Min()
    coll_score = v.collider.Map(0)
    // Score is either 0 (collision) or 1/minDist
    score = coll_score.DefaultIfEmpty(minDist)
    assert(v.collider.IsEmpty())
}

We evaluate the following test input generation strategies:

Random sampling

Halton sampling,

Random or Halton sampling with local search for the two quantitative objectives.

We run 100 iterations of each strategy with a 15 second timeout. For random or Halton sampling, we sample 100 times. For the quantitative objectives, we first generate 85 random or Halton samples, then choose the top 5 scores, and finally run 3 simulated annealing iterations on each of these 5 configurations. Table [tbl:ped_crossing] presents results from the various test input generation strategies. Clearly, Halton sampling offers the lowest dispersion (highest coverage) over the parameter space. This can also be visually confirmed from the plot of test parameters (Figure [fig:Pedestrian_halton_100]). There are no big gaps in the parameter space. Moreover, we find that test strategies optimizing for the first objective are successful in finding more collisions with higher speeds. As these techniques perform simulated annealing repetitions on top of already failing tests, they also find more failing tests overall. Finally, test strategies using the second objective are also successful in finding more (newer) failure cases than simple Random or Halton sampling.

Random sampling (no opt.)

[fig:Pedestrian_rand_100]

Random + opt. / maximizing collision.

[fig:Pedestrian_rand_sa]

Random + opt. / almost failing.

[fig:Pedestrian_random_fuzzing]

Halton sampling (no opt.)

[fig:Pedestrian_halton_100]

Halton + opt. / maximizing collision.

[fig:Pedestrian_halton_sa]

Halton + opt. / almost failing.

[fig:Pedestrian_halton_fuzzing]

Adaptive Cruise Control.

Initial offset (X-axis) vs. max. speed (Y-axis).

[fig:ACC_OffsetvSpeed]

Initial offset (X-axis) vs. fog density (Y-axis).

[fig:ACC_OffsetvFog]

Max. speed (X-axis) vs. fog density (Y-axis).

[fig:ACC_SpeedvFog]

[tbl:acc_results]

Parameterized test on Adaptive Cruise Control, separated for each value of discrete parameters, and low and high values of continuous parameters. A test passes if there are no collisions and no inactivity (the overall distance moved by the test vehicle is more than 5 m. The average offset (in m) maintained by the test vehicle to the lead car (for passing tests) is also presented.

(r)2-3(r)4-7 (r)8-9(r)10-11(r)12-13  2     4     Black  Red   Yellow  Blue  $< 24$  $\geq 24$  $< 5.5$  $\geq 5.5$  $< 0.5$  $\geq 0.5$
Test iters                           54    46    24     22    27      27    51      49         52       48          51       49
Collisions                           7     7     3      3     6       2     6       8          8        6           12       0
Inactivity                           12    4     4      4     6       2     9       7          9        7           1        15
Offset (m)                           42.4  43.4  46.5   48.1  39.6    39.1  33.7    52.7       38.4     47.4        36.5     49.8

We now create and test in an environment with our test vehicle following a car (lead car) on the same lane. The lead car’s behavior is programmed to drive on the same lane as the test vehicle, with a certain maximum speed. This is a very typical driving scenario that engineers test their implementations on. We use $5$ test parameters: the initial lead of the lead car to the test vehicle ($[8 \: m,\: 40 \: m]$), the lead car’s maximum speed ($[3 \: m/s,\: 8 \: m/s]$), density of fog3 in the environment ($[0,1]$), number of lanes on the road ($\{2,\: 4\}$), and color of the lead car ($\{Black, \: Red, \: Yello, \: Blue\}$). We use both, CollisionMonitor4 and DistanceMonitor, as presented in Section 2. A test passes if there is no collision and the autonomous vehicle moves atleast 5 m during the simulation duration (15 s).

We use Paracosm’s default test generation strategy, i.e., Halton sampling for continuous parameters and Random sampling for discrete parameters (no optimization or fuzzing). The SUT is the same CNN as in the previous case study. It is trained on 1034 training samples, which are obtained by manually driving behind a red lead car on the same lane of a 2-lane road with the same maximum velocity (5.5 m/s) and no fog.

The results of this case study are presented in Table [tbl:acc_results]. Looking at the discrete parameters, the number of lanes does not seem to contribute towards a risk of collision. Surprisingly, though the training only involves a Red lead car, the results appear to be the best for a Blue lead car. Moving on to the continuous parameters, the fog density appears to have the most significant impact on test failures (collision or vehicle inactivity). In the presence of dense fog, the SUT behaves pessimistically and does not accelerate much (thereby causing a failure due to inactivity). These are all interesting and useful metrics about the performance of our SUT. Plots of the results projected on to continuous parameters are presented in Figure [fig:ACC].

Results and Analysis

We now summarize the results of our evaluation with respect to our RQs:
RQ 1: All the three case studies involve varied, rich and dynamic environments. They are representative of tests engineers would typically want to do, and we parameterize many different aspects of the world and the dynamic behavior of its components. These designs are at most $70$ lines of code. This provides confidence in Paracosm’s ability of providing an easy interface for the design of realistic test environments.
RQ 2: Our default test generation strategies are found to be quite effective at exploring the parameter space systematically, eliminating large unexplored gaps, and at the same time, successfully identifying problematic cases in all the three case studies. The jaywalking pedestrian study demonstrates that optimization and local search are possible on top of these strategies, and are quite effective in finding the relevant scenarios. The adaptive cruise control study tests over $5$ parameters, which is more than most related works, and even guarantees good coverage of this parameter space. Therefore, it is amply clear that Paracosm’s test input generation methods are useful and effective.
RQ 3: The road segmentation case study uses a well-performing neural network for object segmentation, and we are able to detect degraded performance for automatically generated test inputs. Whereas this study focuses on static image classification, the next two, i.e., the jaywalking pedestrian and the adaptive cruise control study uncover poor performance on simulated driving, using a popular neural network architecture for self driving cars. Therefore, we can safely conclude that Paracosm can find bugs in various different kinds of systems related to autonomous driving.

Threats to Validity

The internal validity of our experiments depends on having implemented our system correctly and, more importantly, trained and used the neural networks considered in the case studies correctly. For training the networks, we followed the available documentation and inspected our examples to ensure that we use an appropriate training procedure. We watched some test runs and replays of tests we did not understand. Furthermore, our implementation logs events and we also capture images, which allow us to check a large number of tests.

In terms of threats to external validity, the biggest challenge in this project has been finding systems that we can easily train and test in complex driving scenarios. Publicly available systems have limited capabilities and tend to be brittle. Many networks trained on real world data do not work well in simulation. We therefore re-train these networks in simulation. An alternative is to run fewer tests, but use more expensive and visually realistic simulations. Our test generation strategy maximizes coverage, even when only a few test iterations can be performed due to high simulation cost.

Related Work

Traditionally, test-driven software development paradigms have advocated testing and mocking frameworks to test software early and often. Mocking frameworks and mock objects allow programmers to test a piece of code against an API specification. Typically, mock objects are stubs providing outputs to explicitly provided lists of inputs of simple types, with little functionality of the actual code. Thus, they fall short of providing a rich environment for autonomous driving. Paracosm can be seen as a mocking framework for reactive, physical systems embedded in the 3D world. Our notion of constraining streams is inspired by work on declarative mocking.

Testing Cyber-Physical Systems.

There is a large body of work on automated test generation tools for cyber-physical systems through heuristic search of a high-dimensional continuous state space. While much of this work has focused on low-level controller interfaces rather than the system level, specification and test generation techniques arising from this work—for example, the use of metric and signal temporal logics or search heuristics—can be adapted to our setting. More recently, test generation tools have started targeting autonomous systems under a simulation-based semantic testing framework similar to ours. In most of these works, visual scenarios are either fixed by hand, or are constrained due to the model or coverage criteria. These analyses are shown to be preferable to the application of random noise on the input vector. Additionally, a simulation-based approach filters benign misclassifications from misclassifications that actually lead to bad or dangerous behavior. Our work extends this line of work and provides an expressive language to design parameterized environments and tests. AsFault uses random search and mutation for procedural generation of road networks for testing. AC3R reconstructs test cases from accident reports.

To address problems of high time and infrastructure cost of testing autonomous systems, several simulators have been developed. The most popular is Gazebo for the ROS robotics framework. It offers a modular and extensible architecture, however falls behind on visual realism and complexity of environments that can be generated with it. To counter this, game engines are used. Popular examples are TORCS, CARLA, and AirSim Modern game engines and support creation of realistic urban environments. Though they enable visually realistic simulations and enable detection of infractions such as collisions, the environments themselves are difficult to design. Designing a custom environment involves manual placement of road segments, buildings, and actors (as well as their properties). Performing many systematic tests is therefore time-consuming and difficult. While these systems and Paracosm share the same aims and much of the same infrastructure, Paracosm focuses on procedural design and systematic testing, backed by a relevant coverage criteria.

Adversarial Testing.

Adversarial examples for neural networks introduce perturbations to inputs that cause a classifier to classify “perceptually identical” inputs differently. Much work has focused on finding adversarial examples in the context of autonomous driving as well as on training a network to be robust to perturbations. Tools such as DeepXplore, DeepTest, DeepGauge, and SADL define a notion of coverage for neural networks based on the number of neurons activated during tests compared against the total number of neurons in the network and activation during training. However, these techniques focus mostly on individual classification tasks and apply 2D transformations on images. In comparison, we consider the closed-loop behavior of the system and our parameters directly change the world rather than apply transformations post facto. We can observe, over time, that certain vehicles are not detected, which is more useful to testers than a single misclassification. Furthermore, it is already known that structural coverage criteria may not be an effective strategy for finding errors in classification. We use coverage metrics on the test space, rather than the structure of the neural network. Alternately, there are recent techniques to verify controllers implemented as neural networks through constraint solving or abstract interpretation. While these tools do not focus on the problem of autonomous driving, their underlying techniques can be combined in the test generation phase for Paracosm.

Future Work and Conclusion

Deploying autonomous systems like self-driving cars in urban environments raises several safety challenges. The complex software stack processes sensor data, builds a semantic model of the surrounding world, makes decisions, plans trajectories, and controls the car. The end-to-end testing of such systems requires the creation and simulation of whole worlds, with different tests representing different world and parameter configurations. Paracosm tackles these problems by

enabling procedural construction of diverse scenarios, with precise control over elements like layout roads, physical and visual properties of objects, and behaviors of actors in the system, and

using quasi-random testing to obtain good coverage over large parameter spaces.

In our evaluation, we show that Paracosm enables easy design of environmnents and automated testing of autonomous agents implemented using neural networks. While finding errors in sensing can be done with only a few static images, we show that Paracosm also enables the creation of longer test scenarios which exercise the controller’s feedback on the environment. Our case studies focused on qualitative state space exploration. In future work, we shall perform quantitative statistical analysis to understand the sensitivity of autonomous vehicle behavior on individual parameters.

In the future, we plan to extend Paracosm’s testing infrastructure to also aid in the training of deep neural networks that require large amounts of high quality training data. For instance, we show that small variations in the environment result in widely different results for road segmentation. Generating data is a time consuming and expensive task. Paracosm can easily generate labelled data for static images. For driving scenarios, we can record a user manually driving in a parameterized Paracosm environment and augment this data by varying parameters that should not impact the car’s behavior. For instance, we can vary the color of other cars, positions of pedestrians who are not crossing, or even the light conditions and sensor properties (within reasonable limits).

Acknowledgements

This research was funded in part by the Deutsche Forschungsgemeinschaft project 389792660-TRR 248 and by the European Research Council under the Grant Agreement 610150 (ERC Synergy Grant ImPACT).

In these Appendices to the main paper, we present additional case studies performed using Paracosm. We also provide additional description and a code sample for connection and composition of road elements, and a sample output to OpenDRIVE.

Testing networks trained on standard datasets

Many autonomous driving systems have on-board components for computer vision tasks like road segmentation, traffic light and traffic sign classification, vehicle detection, and optical flow. In the following tests, instead of training the SUT (a deep neural network) inside our simulation environment, we test components trained on real world datasets. We present results for road detection and vehicle detection.

Test environment.

For the tests, we designed a highly parameterized environment using Paracosm’s programmatic interface. The environment consisted of 4 StraightRoadSegments connected by a CrossIntersection.

The test has three discrete parameters and three continuous parameters:

The number of lanes is either $2$ or $4$ (discrete).

The light condition corresponds to a morning, noon, or an evening drive (discrete).

The number of other cars on the road ranges from $2$ to $9$ (discrete).

The camera focal length is in the $[18,22]$ mm interval (continuous).

The height of the mounting point of the camera varies from $1.9$m to $2.2$m (continuous).

Finally, the camera looks slightly down with a pitch angle between $-10$ and $-12$ degrees (continuous).

Many of our parameters correspond to the vehicle’s camera. These were chosen because in preliminary tests, small perturbations to the camera’s properties led to drastically different results (see Figure [fig:focal_width_rd]). We perform 100 test iterations using Paracosm’s default test generation scheme.

[fig:focal10]

[fig:focal34]

[fig:focal_width_rd]

Road segmentation.

The SUTs here take RGB images as input and return those pixels that are estimated to be a part of the road. We tested:

the convolutional neural network from Simonyan and Zisserman (popular as VGGNet),

Multinet from Teichmann, Weber et al., a top performer on the KITTI Road Estimation Benchmark, and

the fully convolutional network by Long, Shelhamer and Darrell,

All three are trained on the KITTI road segmentation dataset (289 images). The first and third networks do not have a name, so we use initials of the authors’ names, SZ and LSD, respectively. Figure [fig:focal_graph] shows the results for the 100 test iterations ($x$-axis). We plot results in the order in which the tests were performed. The $y$-axis shows the percentage of the “ground truth” road identified as road by the method. A cursory look did not reveal any correlation between road segmentation performance and parameter choice. We observe that SZ is the best performer overall. What is quite striking is the results of LSD: in these tests, it either performs well, or not at all. Except for the poorly performing examples of LSD, false positives are not an issue, generally. Our hypothesis is that the networks do not generalize sufficiently from the limited training data and images that are too different from training lead to poor results.

Road segmentation results (% of the ground truth).

[fig:focal_graph]

[fig:car_detection]

Vehicle detection rates for the two SUTs.

Vehicle detection.

The SUTs here take RGB images as input and return bounding boxes around pixels that correspond to vehicles. Figure [fig:car_detection] shows an example of a vehicle detection system’s output. To detect other vehicles in the vicinity of the autonomous car, we used:

the single shot multibox detector (SSD), a deep neural network, trained with the Pascal Object Recognition Database Collection,

Multinet like in the previous experiment.

Figure 4 summarizes the results. The results are again in the order of the tests. In this experiment, we did not observe any false positives. Overall, Multinet performs better than SSD but these systems are much closer than in the previous experiment. While the detection rates may look disappointing, factors like occlusion as seen in Figure [fig:car_detection] make it difficult to detect all the cars. The two experiments presented here highlight the fact that even with quite narrow parameter ranges (especially for the camera), the quality of results can vary widely.

Additional tests on driving behavior

In the case studies that follow, we test the NVIDIA behavioral cloning framework to perform tests on autonomous vehicle behavior. The SUT takes RGB images as input and returns the corresponding throttle or steering control to be applied. The network is trained inside Paracosm’s simulation environment and the specific training procedure is also described for each case study presented. The primary aim of these case studies is to highlight specific testing features of Paracosm.

Random (dispersion is $0.105$)

[fig:Pedestrian_rand_100]

Halton (dispersion is $0.041$)

[fig:Pedestrian_halton_100]

Random vs. Halton sampling for the pedestrian crossing experiment over various test iterations. The test parameters are the walking speed of the pedestrian ([0.5, 10] m/s) and distance from the car when the pedestrian starts crossing ([5, 60] m).

(r)2-3(r)4-5 # tests  Random   Halton   Random    Halton
50                    $0.200$  $0.083$  $6\%$     $8\%$
100                   $0.105$  $0.041$  $6\%$     $8\%$
200                   $0.051$  $0.029$  $7\%$     $8\%$
400                   $0.025$  $0.011$  $8.75\%$  $8.25\%$

Dynamic Pedestrian Behavior (low dispersion sequences revisited).

This case study has already been presented in the main paper. We now present results that underline the importance of low-dispersion sequences and how coverage improves with more test iterations. Note that the parameter ranges here are different (wider) than the study presented in the main paper. However, the SUT is the same. Figure [fig:PedestrianHalton] demonstrates the advantage of low-dispersion sampling over random sampling. Samples are more spread out for the Halton sequence (low-dispersion). In Table [tbl:crossing], we report the difference between random and Halton sampling for various numbers of test iterations. Halton sampling gives much better dispersion and even leads to more failure cases being revealed (especially for fewer test iterations).

Distance covered (Z-axis, [0,120] m) in changing fog (X-axis, $[0,1]$) and light (Y-axis, $[0,1]$) conditions tested with 400 iterations of the Halton sequence. Green dots and red crosses denote the absence or presence of a collision. The car is trained with a fog density of 0 and light intensity of 0.5.

Changing Environmental Settings.

As mentioned in the main paper, reactive variables can be used to parameterize environment settings so as to describe a large class of configurations. To demonstrate this, we train a model at fixed light intensity and no fog. This case study is similar to the Adaptive Cruise Control case study presented in the main paper. Here, we analyse the autonomous vehicle’s performance when the light intensity and fog density are varied. We report the overall distance covered, and whether a collision happened. Each test lasts 15 seconds. Parameter values are generated using the Halton sequence. Results are aggregated in Figure 5. The car performs best around the parameter values it was trained on. The distance that the car covers drops-off fairly quickly as fog density increases. Perturbations to light intensity often lead to scenarios with collisions.

Test vehicle braking on seeing a red car coming from the opposite direction, even though there is a large distance to the lead car (car on the same lane).

[fig:brake_opp_car]

Speed (X-axis, $[0,40]$ km/h) over time (Y-axis, $[0,8]$ sec) of car trained to follow a red car in the presence of another car coming from the opposite direction. Depending on the color of the incoming car, the speed of the car changes vis-à-vis the baseline driving with no other car.

[fig:features_geometric]

Features of Geometric Components.

For the case study above, the SUT is trained to follow a red lead car driving in front of it on a two-lane road. Under ideal conditions (conditions under which the SUT is trained), it is observed that the autonomous vehicle indeed follows the red lead car while maintaining a safe distance and not colliding with it. We now test how the SUT reacts to cars coming from the other direction. Though the test vehicle’s throttle should not be affected by cars coming from the other direction, it is likely that the SUT learnt to slow the car down when there are several red pixels in the camera image. Indeed, this seems to be the case. When we test with a red car coming from the other direction, our autonomous vehicle slows down in response to this car being close (see Figure [fig:brake_opp_car]). Speed is picked up again once this car passes. Perhaps more surprisingly, the vehicle also slows down when the car coming from the other direction is yellow or green, but has no affect when the car is blue. Figure 6 has plots of speed vs. time for the various cases, with the baseline being no car coming from the other direction.

Connections of road segments (and OpenDRIVE output)

T-Intersection with 2 lanes, long road segments, and traffic and pedestrian lights.

T-Intersection with 4 lanes, short road segments, and no traffic or pedestrian lights.

[fig:tintersection-opendrive_simple]

In this section we provide a code example to demonstrate composition of road elements using the connect operation between road elements. Paracosm supports complex road elements such as cross-intersections, T-intersections, and roundabouts. Connections can be established using the connect method, that takes physical connection identifiers and road elements as arguments. The connections are directed in order to compute the positions of the elements. One road element becomes the parent and it’s children are positioned relative to its position and the specified connection points. After an object is connected, a new composite road element which encapsulates all road elements along with requisite transformations (rotations and translations) is returned. The following example shows how road segments can be connected into a road network.

len = VarInterval(5, 100)
nlanes = VarInterval({2, 4})
// Create a parameterized T-intersection and three straight road elements (east, south, west)
t = TIntersection(nlanes:nlanes)
e = StraightRoadSegment(len:len, nlanes:nlanes)
s = StraightRoadSegment(len:len, nlanes:nlanes)
w = StraightRoadSegment(len:len, nlanes:nlanes)
// connect and get new composite object
net = t.connect((t.ONE, e, e.TWO),
                (t.TWO, s, s.ONE),
                (t.THREE, w, w.ONE))

In this example, the T-intersection is not given a specific position or orientation. It is therefore instantiated at the origin. Road elements connecting to it are then positioned with respect to it. After connection, the composite road element net can be used for tests in simulation or to a standardized format (OpenDRIVE). Figure [fig:tintersections] shows some samples in the OpenDRIVE viewer.

Connecting elements has two purposes. First, it allows Paracosm to perform sanity checks like proper positioning of road elements. Second, it creates an overlay graph of the road networks which can easily be followed by environment controlled vehicles. When a road network is created, the runtime system of Paracosm checks that compositions of road elements and intersections are topologically and geometrically valid. All road elements must be connected to a matching road correctly (for example, a 2-lane road segment cannot be connected to a 6-lane road segment directly), there can be no spatial overlaps between road segments, and the positions of the connection points must match.

A large grid world with several connected road elements viewed in the default 3D simulator.

In general, Paracosm inherits all programming features of the underlying imperative programming model as well as reactive programming with streams. Thus, one can build complex urban settings through composition and iteration. For instance, the grid world shown in Figure 8 was created by iterating a simple road network.

This chapter is licensed under the terms of the Creative CommonsAttribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

.28em plus.1em minus.1em The images or other third party material in this chapter are included in the chapter’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons license and your intendeduse is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

 1. https://www.weforum.org/agenda/2019/08/self-driving-vehicles-public-trust/↩

 2. https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf↩

 3. 0 denotes no fog and 1 denotes very dense fog (exponential squared scale).↩

 4. the monitor additionally calculates the mean distance of the test vehicle to the lead car during the test, which is used for later analysis.↩

According to the results on the adaptive cruise control case study, which test parameter had the most significant impact on test failures involving collisions or vehicle inactivity?


Prompt:
Dec 29
ARRAIAL DO CABO SUPEARRAIAL DO CA

$101.50

Dec 28
BOOKING.COM BRASIL SSAO PAULO

$565.86

Dec 28
DROGARIA ARAUJO BELO HORIZONT

$40.48

Dec 28
LAMBEIJOS PET SHOP BELO HORIZONTE

$15.37

Dec 28
UNIVET LTDA BELO HORIZONTE MG

$24.58

Dec 24
HOTEL MONTE FELICE GRAMADO RS

$15.70

Dec 23
HR CAFE GRAMADO LTDAGRAMADO

$83.55

Dec 23
SNOWLAND PARTICIPACOGRAMADO RS

$31.18

Dec 22
APLPAY ADEGA DO JACAGRAMADO BR

$8.73

Dec 21
GRAMADO TERMAS PARK GRAMADO RS

$24.14

Dec 21
GRAMADO TERMAS PARK GRAMADO RS

$30.83

Dec 21
HOTEL MONTE FELICE GRAMADO RS

$159.95

Dec 21
HOTEL NAME_2 GRAMADGRAMADO

$2.75

Dec 21
RESTAURANTE NOVA ERAPORTO ALEGRE

$84.03

Dec 20
CANTINA PASTASCIUTTAGRAMADO RS

$49.71

Dec 19
FARMACIA SAO JOAO GRAMADO

$16.65

Dec 19
LA DIVINA RESTAURANTGRAMADO

$62.69

Dec 18
HOTEL NAME_2 GRAMADGRAMADO

$182.41

Dec 18
MC2 ESTACIONAMENTOS GRAMADO RS

$17.08

Dec 18
OLIVAS GRAMADO BR

$66.17

Dec 18
THE WAFFLE KING GRAMADO BR

$7.21

Dec 16
BARBEARIA SEU ELIAS BELO HORIZONTE MG

$57.88

Dec 16
INTEREST CHARGE ON PAY OVER TIME PURCHASES

$17.80

Dec 16
OURO MINAS BELO HORIZONTE BR

$1.71

Dec 16
OURO MINAS BELO HORIZONTE BR

$56.93

Dec 14
MULTIPLAN ADMINISTRABELO HORIZONTE

$4.58

Dec 14
O QUINTAL GRANU CAFEBELO HORIZONTE MG

$56.67

Dec 13
TOKAI SAVASSI BELO HORIZONTE

$21.53

Dec 11
LATE FEE

$29.00

Dec 10
EPA PLUS 013 BELO HORIZONT

$79.54

Dec 10
SAMS CONTAGEM

$106.09

Dec 9
DROGARIA ARAUJO BELO HORIZONT

$6.20

Dec 8
PIZZARELLA BELO HORIZONT

$28.89

Dec 5
OLEGARIO BELO HORIZONTE MG

$31.76

Dec 3
ESTACIONAMENTO MINASBELO HORIZONT

$3.10

Dec 3
MINAS GRILL BELO HORIZONTE MG

$14.07

Dec 3
POSTO MINAS SHOPPINGBELO HORIZONTE

$41.27

Nov 16
MEMBERSHIP FEE

$695.00

May 31
Pending
STROBEL AMERICA

$900.00

May 31
Pending
THEODORO E FILHOS DISTRIBUIDOR

$5.14

May 30
APLPAY MGPOWER BELO HORIZONT

$149.80

May 30
MERCADOLIVRE*MERCADOSANTANA DE PARNAIBA

$25.74

May 27
DROGARIA ARAUJO BELO HORIZONT

$1.51

May 27
PERSARENTACAR CONTAGEM

$21.57

May 27
SUPERMERCADO E PADARBELO HORIZONTE

$11.40

May 27
SUPERMERCADO E PADARBELO HORIZONTE

$104.23

May 26
DROGARIA ARAUJO BELO HORIZONT

$133.57

May 24
WWW.STROBELAMERICA.CDOVER DE

$1,787.00

May 22
DROGARIA ARAUJO BELO HORIZONT

$2.82

May 22
SUPERMERCADO E PADARBELO HORIZONTE

$42.54

May 21
BETA DE PRATA LTDA ETIRADENTES

$17.20

May 21
JARDINS DE SANTO TIRADENTES

$22.03

May 21
PAG*ICASEIPRESENTES SAO PAULO

$129.51

May 21
POSTO CN LTDA BELO HORIZONTE

$52.35

May 21
POUSADA RECANTO A TIRADENTES

$0.81

May 19
POSTO VIP BELO HORIZONT

$22.73

May 19
P


Prompt:
아래 내용은 시간여행이라는 책을 요약한 내용입니다. 아래의 글을 바탕으로 2000자 내외의 칼럼을 작성해주세요.

사람들은 몇 살 때부터 자기 혼자만의 생각을 즐기지 못하게 됐을까요?

깨어 있으면서도 다른 사람들의 소리를 듣지 않는, 그런 방법이 어디 없을까?

혹시 이런거 생각 안 해 봤니? 네가 하기 싫은 일에 대해 말하면 하고 싶은 일을 할 수 없다는 사실 말야.

반대로 말해 볼게. 네가 하고 싶은 일에 대해 말하거나, 또 하고 싶은 일을 하고 있다고 느낄 때, 어땠니, 즉시 그 일을 할 수 있었지? 하늘을 나는 일 말야.

가장 중요한 것은, 하고 싶은 일을 하고 있다는 느낌이야.

정말로 하고 싶은 일이 있다면, 무엇을 하고 싶으며 왜 그 일을 하고 싶은지에 대해 생각해보란 말이야 . 바로 지금 그 일을 하는 중이라고 느낄 때까지 말야. 이게 바로 네가 배운 가장 중요한 교훈이야.

고통의 사슬을 끊는 방법- 느낌으로 알 수 있다. 하기 싫은 일에 대해 생각하거나 말할 때, 항상 나쁜 감정을 갖게 된다. 고통의 사슬에서 빠져나올 수 있는 효과가 가장 빠른 방법은 감사한 마음을 가지는 것이다.

1. 내가 하기 싫은 일을 알아내는 것
2. 내가 하고 싶은 일을 결정하는 것
3. 하고 싶은 일을 알아내자마자 실제로 그 일을 하고 있다고 느껴야 한다
4. 내 소망을 행동으로 옮기는 단계

우주에 존재하는 모든 사람들과 모든 사물은 끌림의 법칙을 따르고 있다.

매 순간 나는 내가 원하는 상황을, 혹은 내가 원하지 않는 상황을 선택할 수 있다.

내가 행복하거나 감사하는 마음을 가지고 다른 사람이나 사물의 긍정적인 면을 보면, 내가 원하는 상황과 조화를 이루게 된다.

항상. 언제나 내 느낌을 믿어야 한다. 내 느낌이 바로 나의 길잡이다.

내가 무엇에 관심을 가지느냐에 따라 내 기분이 좋아지거나 나빠지는 것이다.

내가 할 일은 행복이 들어올 수 있게 나의 밸브를 열어주는 것

오늘 나의 기분이 좋아지면 내일과 그 다음 날 일어날 상황도 마음에 들 것이다

내 느낌과 생각이 어떻게 연결돼 있는지 이제 알겠어. 상황은 변하지 않아! 내 생각이 변하는거야.

감사하는 마음은 항상 거기에 있다. 원래부터 거기에 있었다.

나 자신이 행복하지 않으면 다른 사람에게도 행복을 나눠줄 수 없다

내 몸은 진정한 내가 아니다. 단지 진정한 내가 재미있게 놀고 성장하고 기뻐할 수 있도록 해주는 도구일 뿐이다

내 생각에 따라 내 기쁨과 고통을 선택할 수 있다는 것이다.

내 주변의 상황이 내 감정을 통제하기 시작하면 난 항상 덫에 걸리게 된다. 내 느낌을 통제할 수 있도록 연습해야 한다. 그래야 내 생각도 통제하게 되고, 결국 진정한 자유를 누릴 수 있다.

내 삶을 아주 소중하게 생각하면, 다른 사람들의 행동이나 말 때문에 괴로워하지 않게 될지도 몰라

내가 원하지 않는 상황에 신경을 쓰면 결국 내가 원하는 상황을 거부하게 되는 것이다. 그렇기 때문에 내가 원하지 않는 상황을 구별해 내고 내가 원하는 상황을 찾아서 ‘좋아’라고 말하는 일이 가장 중요하다.

내가 원하지 않는 상황을 밀어내려고 하기보다 내가 원하는 상황을 느긋하게 생각한다. 물론 말도 방향을 일러주지만, 느낌이야 말로 내가 받아들이는지 거부하는지를 가장 정확하게 알려주는 안내자다. 내가 원하는 상황에 대해 계속 이야기하면 모든 상황이 점점 좋아질 것이다.

행복의 물결은 항상 나를 향해 흐르고 있다. 나는 매 순간 그 물결을 받아들이거나 거부한다. 그 물결을 받아들이거나 거부할 수 있는 사람은 오직 나 뿐이다.

나는 모든 일이 순조롭게 흘러간다는 믿음을 전해야 한다. 나 자신의 선명한 경험을 많은 사람들에게 말하면 사람들은 자신이 밀어내야 할 상황은 세상에 없다는 사실을 이해하게 될것이다. 밀어내는 행위 자체가 행복을 쫓아내는 일이라는 사실도 또한 알게 될것이다.


Prompt:
Take the following text and put it into a Salesforce order JSON structure.

2022 2022 | תנש LA.O.22.072522 שכר תנמזה דדוובבככלל :ןימזמ 'סמ מ"עב הבונת ללאאררששיי תתססננככ ::ההננייממזזממ ההננחחתת הוקת חתפ,300 .ד.ת 21 םייפכ עיגי 4910201 ירשב תסנכ :ןסח מ םש 0508875313 6986028 יגח רומיל :ןופלט ::ממ..עע noyt@tnuva.co.il :ינורטקלא ראוד 33008811 ::םםככררפפססממ 17/01/23:הקפסא .ת 16/01/23 :אתכמסא .ת 03-7367706 :יצרא זכרמ ןופלט הרושל כ"הס 'חיל ריחמ הדימ 'חי תומכ םיטרפ קפס דוק # 3% ןוטרק ל1ירט הבונת בלח ח"ש 1,099.20 4.58 רטיל 240.00 4131074 1 ןירדהמ ח"ש 117.12 7.32 רטיל 16.00 רטיל 1 רשעומ היוס הקשמ 4125578 2 1 ירט יעבט םעטב טייל היוס הקשמ ח"ש 122.72 7.67 רטיל 16.00 4132552 3 רטיל ח"ש 352.10 35.21 הדיחי 10.00 לכימ לקשמב 5% תירגלוב סואריפ 4120818 4 מ"עב ח"ש 113.90 22.78 םרגוליק 5.00 ידסומה קושל יעבט ופוט 10322786 5 הנכות 5% רג 50 עיבג הניבג הבונת תונבל ח"ש 18.72 0.78 הדיחי 24.00 47782 6 צדב תוישעת ח"ש 262.14 43.69 םרגוליק 6.00 גק 3 ידסומ קמע יתיתפ 4121341 7 ח"ש 141.12 8.82 רטיל 16.00 םידקש+ תלוביש הקשמ 10325619 8 340 תויגוע םעטב GO בלח הקשמ גומלא ח"ש 112.20 5.61 רטיל 20.00 10329686 9 ל"מ ח"ש 80.80 4.04 הדיחי 20.00 למ 250 קובקבב הבונת וקוש 4126872 10 תיבמ ח"ש 69.44 4.34 רטיל 16.00 %1 ל1 קקפ +טרק ירט הבונת בלח 42435 11 Almog ח"ש 37.85 37.85 םרגוליק 1.00 הינדעמל 5% תנדועמ תיתפצ 4136291 12 150 עיבג %3 ןבדבוד טרוגוי הלפוי ח"ש 28.16 3.52 הדיחי 8.00 40066 13 ERP םרג 150 עיבג %3 קסרפא טרוגוי הלפוי תועצמאב ח"ש 28.16 3.52 הדיחי 8.00 40080 14 םרג ח"ש 2,583.63 החנה ינפל םוכס קפוה ח"ש -0.15 החנה ח"ש 2,583.63 מ"עמ ינפל כ"הס ח"ש 439.22 מ"עמ 17.00% ח"ש 3,023.00 םולשתל כ"הס 01/05/23 :ךיראת דע םולשתל .ישאר תונמזה זכרמ רושיא אלל וז הנמזהב יוניש עצבל ןיא * .םייקלח םיחולשמ ןיא * .הנמזהה ףוגב אלש םיטירפ לש הקפסא ןיא * 16/01/23 15:12 2 ךותמ 1 דומע 2022 2022 | תנש LA.O.22.072522 שכר תנמזה דדוובבככלל :ןימזמ 'סמ מ"עב הבונת ללאאררששיי תתססננככ ::ההננייממזזממ ההננחחתת הוקת חתפ,300 .ד.ת 21 םייפכ עיגי 4910201 ירשב תסנכ :ןסח מ םש 0508875313 6986028 יגח רומיל :ןופלט ::ממ..עע noyt@tnuva.co.il :ינורטקלא ראוד 33008811 ::םםככררפפססממ 17/01/23:הקפסא .ת 16/01/23 :אתכמסא .ת 03-7367706 :יצרא זכרמ ןופלט מ"עב הנכות תוישעת גומלא תיבמ Almog ERP תועצמאב קפוה בר דובכב לשא.ל 16/01/23 15:12 2 ךותמ 2 דומע