uzh-rpg / rpg_esim

ESIM: an Open Event Camera Simulator

ESIM: an Open Event Camera Simulator

This is the code for the 2018 CoRL paper ESIM: an Open Event Camera Simulator by Henri Rebecq, Daniel Gehrig and Davide Scaramuzza:

@Article{Rebecq18corl,
  author        = {Henri Rebecq and Daniel Gehrig and Davide Scaramuzza},
  title         = {{ESIM}: an Open Event Camera Simulator},
  journal       = {Conf. on Robotics Learning (CoRL)},
  year          = 2018,
  month         = oct
}

You can find a pdf of the paper here. If you use any of this code, please cite this publication.

Python Bindings

Python bindings for the event camera simulator can be found here. We now also support GPU support for fully parallel event generation!

Features

• Accurate event simulation, guaranteed by the tight integration between the rendering engine and the event simulator
• Inertial Measurement Unit (IMU) simulation
• Support for multi-camera systems
• Ground truth camera poses, IMU biases, angular/linear velocities, depth maps, and optic flow maps
• Support for camera distortion (only planar and panoramic renderers)
• Different C+/C- contrast thresholds
• Basic noise simulation for event cameras (based on additive Gaussian noise on the contrast threshold)
• Motion blur simulation
• Publish to ROS and/or save data to rosbag

Install

Installation instructions can be found in our wiki.

Run

Specific instructions to run the simulator depending on the chosen rendering engine can be found in our wiki.

Acknowledgements

We thank Raffael Theiler and Dario Brescianini for their contributions to ESIM. This research was supported by by Swiss National Center of Competence Research Robotics (NCCR), Qualcomm (through the Qualcomm Innovation Fellowship Award 2018), the SNSF-ERC Starting Grant and DARPA FLA.

A significant part of ESIM uses components (spline trajectories, inertial measurement unit simulation, various utility functions) from the ze_oss project. ESIM depends on UnrealCV for the photorealistic rendering engine. We also reused some code samples from the excellent Lean OpenGL tutorial in our OpenGL rendering engine. Finally, ESIM depends on the Open Asset Import Library (assimp) to load 3D models and Blender scenes within the OpenGL rendering engine.

rpg_esim's Issues

Input trajectory from UE4

Hello,

I am wondering wheater there is a way to allow the simulator to follow a trajectory given by UE4 instead of the random or predefined trajectory (trajectory_type 0 and 1). For example, following a spline or just the movement of an actor in the scene. I would like to handle the trajectory of the camera directly from UE4. Is it possible?

rviz no output

I have tried the Multi Objects 2d renderer example, and run
roslaunch esim_ros esim.launch config:=cfg/multi_objects.conf

the out put is:
... logging to /home/wangy0k/.ros/log/99a41850-b74d-11e9-a932-ac1f6bd0c91d/roslaunch-VCC-SYS-7039A-I-4713.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://VCC-SYS-7039A-I:39191/

SUMMARY

PARAMETERS

• /hector_trajectory_server/source_frame_name: cam0
• /hector_trajectory_server/target_frame_name: map
• /hector_trajectory_server/trajectory_publish_rate: 15
• /hector_trajectory_server/trajectory_update_rate: 15
• /optic_flow_viz/arrows_scale: 0.07
• /optic_flow_viz/arrows_step: 7
• /optic_flow_viz/arrows_upsample_factor: 1
• /optic_flow_viz/publish_rate: 100
• /rosdistro: kinetic
• /rosversion: 1.12.14

NODES
/
dvs_renderer (dvs_renderer/dvs_renderer)
esim_node (esim_ros/esim_node)
hector_trajectory_server (hector_trajectory_server/hector_trajectory_server)
optic_flow_viz (esim_visualization/optic_flow_converter.py)

auto-starting new master
process[master]: started with pid [4723]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 99a41850-b74d-11e9-a932-ac1f6bd0c91d
process[rosout-1]: started with pid [4736]
started core service [/rosout]
process[esim_node-2]: started with pid [4747]
process[dvs_renderer-3]: started with pid [4754]
process[optic_flow_viz-4]: started with pid [4755]
process[hector_trajectory_server-5]: started with pid [4756]
I0805 09:52:33.680042 4747 imp_multi_objects_2d_renderer.cpp:44] width = 346, height = 260
I0805 09:52:33.680193 4747 imp_multi_objects_2d_renderer.cpp:45] tmax = 1
I0805 09:52:33.680214 4747 object.cpp:12] Loading texture file from file: /home/wangy0k/sim_ws/src/rpg_esim/event_camera_simulator/imp/imp_planar_renderer/textures/office.jpg.
[ INFO] [1564987953.684704422]: Waiting for tf transform data between frames map and cam0 to become available
I0805 09:52:33.791405 4747 object.cpp:24] Pre-filtering the texture with median filter of size: 15.
[INFO] [1564987954.093177]: Started flow converter node
[INFO] [1564987954.093433]: Step size between arrows: 7
[INFO] [1564987954.093572]: Scale factor: 0.07
[INFO] [1564987954.093796]: Upsample factor: x1
[INFO] [1564987954.093885]: Publish rate: 100 Hz
I0805 09:52:35.000685 4747 object.cpp:31] Pre-filtering the texture with gaussian filter of size: 1.
I0805 09:52:35.241488 4747 object.cpp:12] Loading texture file from file: /home/wangy0k/sim_ws/src/rpg_esim/event_camera_simulator/imp/imp_multi_objects_2d/textures/rugby_ball.png.
I0805 09:52:35.260526 4747 object.cpp:24] Pre-filtering the texture with median filter of size: 11.
I0805 09:52:35.379128 4747 object.cpp:31] Pre-filtering the texture with gaussian filter of size: 1.
I0805 09:52:35.405050 4747 ros_publisher.cpp:37] Initializing ROS
I0805 09:52:35.511757 4747 rosbag_writer.cpp:39] Will write to bag: /tmp/out.bag
W0805 09:52:35.511781 4747 adaptive_sampling_benchmark_publisher.cpp:57] Empty benchmark folder string: will not write benchmark files
W0805 09:52:35.511885 4747 synthetic_optic_flow_publisher.cpp:41] Empty output folder string: will not write synthetic optic flow files
I0805 09:52:35.518921 4747 event_simulator.cpp:27] Converting the image to log image with eps = 0.001.
I0805 09:52:35.519232 4747 event_simulator.cpp:11] Initialized event camera simulator with sensor size: [346 x 260]
I0805 09:52:35.519243 4747 event_simulator.cpp:12] and contrast thresholds: C+ = 0.5 , C- = 0.5
W0805 09:52:35.520195 4747 camera_simulator.cpp:43] The images do not cover a time span long enough to simulate the exposure time accurately.
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
publish image from callback
I0805 09:52:48.169306 4747 rosbag_writer.cpp:62] Finalizing the bag...
I0805 09:52:48.172581 4747 rosbag_writer.cpp:64] Finished writing to bag: /tmp/out.bag
[esim_node-2] process has finished cleanly
log file: /home/wangy0k/.ros/log/99a41850-b74d-11e9-a932-ac1f6bd0c91d/esim_node-2*.log
[ WARN] [1564987973.686946297]: No transform between frames map and cam0 available after 20.002202 seconds of waiting. This warning only prints once.

And after that, I run the rviz
roscd esim_visualization
rviz -d cfg/esim.rviz

It shows no image. It seems the bag file is not read.

library called `ze` keeps giving error.

Hi, what is this library called ze? Does it have similar functions as eigen?
It keeps giving me error.

Sample simulated sequences

It would be useful if you can provide sample sequences of the simulator, specifically the Photorealistic Rendering part.

Detail information of argument

I want to know the detail information of argument in the *.conf, *yaml files.
In addition, I want to know which argument should be in each file.

Do you have any instruction?

Thanks.

Object trajectory differ from the csv file

Hi,
The object trajectory seem to differ from the input in the csv file. I try to increase the parameter --trajectory_num_spline_segments as suggest in #14 and and also try to set --trajectory_lambda (#33). Both parameters didn't fix the problem.

From my experimentations, the problem seems to occur when one of the quaternion value cross 0. It might be related to singularities as you mentioned in #14.

I include an experimentation where the problem occur:
trajectory.zip

In this case, the problem occur when qx cross 0.

Do you have any idea how to resolve this ?
Thanks for your work !

How to save data

Thanks for your sharing! But how could i save the data from the simulator?

build error for dvs_renderer

I am trying to 'catkin build esim_ros' but only managed to build 13 packages.
The error happens at dvs_renderer and the log is below:

CMakeFiles/dvs_renderer.dir/src/image_tracking.cpp.o: In function dvs_renderer::ImageTracking::ImageTracking(ros::NodeHandle&)': image_tracking.cpp:(.text+0x444): undefined reference to image_transport::ImageTransport::ImageTransport(ros::NodeHandle const&)'
image_tracking.cpp:(.text+0x4ba): undefined reference to image_transport::ImageTransport::advertise(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, bool)' image_tracking.cpp:(.text+0x58c): undefined reference to image_transport::ImageTransport::advertise(std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, unsigned int, bool)'
image_tracking.cpp:(.text+0x5e6): undefined reference to image_transport::ImageTransport::~ImageTransport()' image_tracking.cpp:(.text+0x72b): undefined reference to image_transport::ImageTransport::~ImageTransport()'
CMakeFiles/dvs_renderer.dir/src/image_tracking.cpp.o: In function dvs_renderer::ImageTracking::eventsCallback(boost::shared_ptr<dvs_msgs::EventArray_<std::allocator<void> > const> const&)': image_tracking.cpp:(.text+0x7a8): undefined reference to image_transport::Publisher::getNumSubscribers() const'
image_tracking.cpp:(.text+0x828): undefined reference to image_transport::Publisher::getNumSubscribers() const' CMakeFiles/dvs_renderer.dir/src/image_tracking.cpp.o: In function dvs_renderer::ImageTracking::render()':
image_tracking.cpp:(.text+0x11b2): undefined reference to image_transport::Publisher::publish(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&) const' image_tracking.cpp:(.text+0x15e7): undefined reference to image_transport::Publisher::publish(boost::shared_ptr<sensor_msgs::Image<std::allocator > const> const&) const'
CMakeFiles/dvs_renderer.dir/src/image_tracking.cpp.o: In function dvs_renderer::ImageTracking::imageCallback(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&)': image_tracking.cpp:(.text+0x2071): undefined reference to image_transport::Publisher::getNumSubscribers() const'
image_tracking.cpp:(.text+0x2082): undefined reference to image_transport::Publisher::getNumSubscribers() const' CMakeFiles/dvs_renderer.dir/src/renderer.cpp.o: In function dvs_renderer::Renderer::~Renderer()':
renderer.cpp:(.text+0x203): undefined reference to image_transport::Publisher::shutdown()' renderer.cpp:(.text+0x20f): undefined reference to image_transport::Publisher::shutdown()'
CMakeFiles/dvs_renderer.dir/src/renderer.cpp.o: In function dvs_renderer::Renderer::imageCallback(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&)': renderer.cpp:(.text+0x1389): undefined reference to image_transport::Publisher::publish(boost::shared_ptr<sensor_msgs::Image<std::allocator > const> const&) const'
CMakeFiles/dvs_renderer.dir/src/renderer.cpp.o: In function dvs_renderer::Renderer::eventsCallback(boost::shared_ptr<dvs_msgs::EventArray_<std::allocator<void> > const> const&)': renderer.cpp:(.text+0x19b0): undefined reference to image_transport::Publisher::getNumSubscribers() const'
renderer.cpp:(.text+0x214d): undefined reference to image_transport::Publisher::publish(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&) const' renderer.cpp:(.text+0x2189): undefined reference to image_transport::Publisher::getNumSubscribers() const'
renderer.cpp:(.text+0x27b4): undefined reference to image_transport::Publisher::publish(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&) const' CMakeFiles/dvs_renderer.dir/src/renderer.cpp.o: In function dvs_renderer::Renderer::Renderer(ros::NodeHandle&, ros::NodeHandle)':
renderer.cpp:(.text+0x36c3): undefined reference to image_transport::ImageTransport::ImageTransport(ros::NodeHandle const&)' renderer.cpp:(.text+0x3b6a): undefined reference to image_transport::ImageTransport::subscribe(std::_cxx11::basic_string<char, std::char_traits, std::allocator > const&, unsigned int, boost::function<void (boost::shared_ptr<sensor_msgs::Image<std::allocator > const> const&)> const&, boost::shared_ptr const&, image_transport::TransportHints const&)'
renderer.cpp:(.text+0x3d8d): undefined reference to image_transport::ImageTransport::advertise(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, bool)' renderer.cpp:(.text+0x3e19): undefined reference to image_transport::ImageTransport::advertise(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, unsigned int, bool)'
renderer.cpp:(.text+0x4111): undefined reference to image_transport::ImageTransport::~ImageTransport()' renderer.cpp:(.text+0x434d): undefined reference to image_transport::ImageTransport::~ImageTransport()'
collect2: error: ld returned 1 exit status
make[2]: *** [/home/winston/esim_ws/devel/.private/dvs_renderer/lib/dvs_renderer/dvs_renderer] Error 1
make[1]: *** [CMakeFiles/dvs_renderer.dir/all] Error 2
make: *** [all] Error 2

Multi Objects 2D renderer not able to detect office.jpg and hence doesn't work

I have followed the instructions at this link (https://github.com/uzh-rpg/rpg_esim/wiki/Multi-Objects-2D-renderer) to render a simulation of the multi objects 2D renderer. I have changed the absolute paths as instructed in the wiki. Below are the screenshots of that

and

This is the error message I get

I can see that the files referenced by all the paths given in the config file exist. However the error message says that the package has some problem with opening office.jpg
I am able to get the Planar rendering running without any problems by only changing the absolute paths.

Output full projection matrix in rosbags

Currently the output projection matrix in the rosbags do not include the translation of the cameras.

Using rosbag file

Hello,

I am wondering if there is a way of using a rosbag file with a video as input. I saw something related to it in the data_provider_rosbag.cpp file. However, it's not clear to me how to use use the simulator with this configuration.

Could you please give me any hint?

The differences between 'image_raw' and 'image_corrupted'

The output rosbag contain two topics named 'image_raw' and 'image_corrupted'. What is the differences between the two topics?

Stereo Camera Setup

Is there an example of how to setup a stereo simulation?

I am trying to create a stereo setup but both cameras give the same information and I cannot seem to offset one from the other.

I have made a .yaml file for two cameras (see below) and tried to separate them using the T_B_C matrices, but when the simulation has finished both sets of data in the ros bag look identical.

the .yaml file is

label: "simulated_camera"
id: 433585ebda8d1431223927a14788a127
cameras:
- camera:
    label: dvs0
    id: a0fba5412e961934d842d3a2a78e5cba
    line-delay-nanoseconds: 0
    image_height: 180
    image_width: 240
    type: pinhole
    intrinsics:
      cols: 1
      rows: 4
      data: [198.245521288, 198.277025706, 142.064861206, 100.903484508]
    distortion:
      type: equidistant
      parameters:
        cols: 1
        rows: 4
        data: [-0.0506755889541, 0.0456313630037, -0.0825742639337, 0.0557104403236]
  T_B_C:
    cols: 4
    rows: 4
    data: [1, 0, 0,  0,
           0, -1, 0, 0,
           0,  0, -1, 0,
           0, 0, 0, 1]
- camera:
    label: dvs1
    id: a0fba5412e961934d842d3a2a78e5cbb
    line-delay-nanoseconds: 0
    image_height: 180
    image_width: 240
    type: pinhole
    intrinsics:
      cols: 1
      rows: 4
      data: [198.245521288, 198.277025706, 142.064861206, 100.903484508]
    distortion:
      type: equidistant
      parameters:
        cols: 1
        rows: 4
        data: [-0.0506755889541, 0.0456313630037, -0.0825742639337, 0.0557104403236]
  T_B_C:
      cols: 4
      rows: 4
      data: [1, 0, 0,  100.0,
             0, -1, 0, 0.0,
             0,  0, -1, 0.0,
             0, 0, 0, 1]

I am using a modified version of the example.conf with this .yaml file. Do I need to add any other variables to get stereo to work?

Also in the T_B_C matrix what are the units for displacement?

In the simulation output it recognises that it is stereo

I0502 13:54:33.050158  8920 camera_rig.cpp:197] Camera 0 and 1: Overlap = 0.996667, Baseline = 100.0 -> Stereo Rig.

but the over lap is very high.

If i look at the camera info for the two cameras in the output ros bag they are the same when I would expect the projection matrix to have a shift.

cam 0

header: 
  seq: 0
  stamp: 
    secs: 0
    nsecs:   1000000
  frame_id: ''
height: 180
width: 240
distortion_model: "equidistant"
D: [-0.0506755889541, 0.0456313630037, -0.0825742639337, 0.0557104403236]
K: [198.245521288, 0.0, 142.064861206, 0.0, 198.277025706, 100.903484508, 0.0, 0.0, 1.0]
R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
P: [198.245521288, 0.0, 142.064861206, 0.0, 0.0, 198.277025706, 100.903484508, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False

cam 1

header: 
  seq: 0
  stamp: 
    secs: 0
    nsecs:   1000000
  frame_id: ''
height: 180
width: 240
distortion_model: "equidistant"
D: [-0.0506755889541, 0.0456313630037, -0.0825742639337, 0.0557104403236]
K: [198.245521288, 0.0, 142.064861206, 0.0, 198.277025706, 100.903484508, 0.0, 0.0, 1.0]
R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
P: [198.245521288, 0.0, 142.064861206, 0.0, 0.0, 198.277025706, 100.903484508, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False

Thanks.

Failed to generate to OpenGL window

Hi there, I am trying this new simulator on a virtual machine. After installation, I always get this error. Can you see what is the reason? Thanks in advance! (I tried with other glfw3 examples and I can successfully generate windows.)

the error shows as:F1205 00:37:45.703743 3456 opengl_renderer.cpp:58] Failed to create GLFW window_

as whole:
... logging to /home/parallels/.ros/log/fb11ed2a-f825-11e8-97e8-001c42bb2f7d/roslaunch-parallels-vm-3416.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://parallels-vm:32995/

SUMMARY

PARAMETERS

• /hector_trajectory_server/source_frame_name: cam0
• /hector_trajectory_server/target_frame_name: map
• /hector_trajectory_server/trajectory_publish_rate: 15
• /hector_trajectory_server/trajectory_update_rate: 15
• /optic_flow_viz/arrows_scale: 0.07
• /optic_flow_viz/arrows_step: 7
• /optic_flow_viz/arrows_upsample_factor: 1
• /optic_flow_viz/publish_rate: 100
• /rosdistro: kinetic
• /rosversion: 1.12.14

NODES
/
dvs_renderer (dvs_renderer/dvs_renderer)
esim_node (esim_ros/esim_node)
hector_trajectory_server (hector_trajectory_server/hector_trajectory_server)
optic_flow_viz (esim_visualization/optic_flow_converter.py)

auto-starting new master
process[master]: started with pid [3429]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to fb11ed2a-f825-11e8-97e8-001c42bb2f7d
process[rosout-1]: started with pid [3445]
started core service [/rosout]
process[esim_node-2]: started with pid [3456]
process[dvs_renderer-3]: started with pid [3467]
process[optic_flow_viz-4]: started with pid [3468]
process[hector_trajectory_server-5]: started with pid [3469]
[ INFO] [1543970265.542087462]: Waiting for tf transform data between frames map and cam0 to become available
I1205 00:37:45.645692 3456 data_provider_online_render.cpp:63] Horizontal FOV: 61.7164 deg
I1205 00:37:45.646080 3456 data_provider_online_render.cpp:64] Vertical FOV: 48.2168 deg
I1205 00:37:45.646138 3456 data_provider_online_render.cpp:65] Diagonal FOV: 73.4823 deg

F1205 00:37:45.703743 3456 opengl_renderer.cpp:58] Failed to create GLFW window_

*** Check failure stack trace: ***
@ 0x7f7b46e2943d google::LogMessage::Fail()
@ 0x7f7b46e2b253 google::LogMessage::SendToLog()
@ 0x7f7b46e28fcb google::LogMessage::Flush()
@ 0x7f7b46e2bc3e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f7b42e6b64a event_camera_simulator::OpenGLRenderer::init()
@ 0x7f7b42e6bd9a event_camera_simulator::OpenGLRenderer::setCamera()
@ 0x7f7b484087c6 event_camera_simulator::DataProviderOnlineMoving3DCameraRig::DataProviderOnlineMoving3DCameraRig()
@ 0x7f7b48400ad7 event_camera_simulator::loadDataProviderFromGflags()
@ 0x404e9e main
@ 0x7f7b462b9830 __libc_start_main
@ 0x405819 start
**[esim_node-2] process has died [pid 3456, exit code -6, cmd /home/parallels/sim_ws/devel/lib/esim_ros/esim_node --v=1 --vmodule=data_provider_from_folder=10 --flagfile=/home/parallels/sim_ws/src/rpg_esim/event_camera_simulator/esim_ros/cfg/opengl.conf __name:=esim_node __log:=/home/parallels/.ros/log/fb11ed2a-f825-11e8-97e8-001c42bb2f7d/esim_node-2.log]._**
log file: /home/parallels/.ros/log/fb11ed2a-f825-11e8-97e8-001c42bb2f7d/esim_node-2*.log
[INFO] [1543970266.207507]: Started flow converter node
[INFO] [1543970266.208061]: Step size between arrows: 7
[INFO] [1543970266.208381]: Scale factor: 0.07
[INFO] [1543970266.208666]: Upsample factor: x1
[INFO] [1543970266.209042]: Publish rate: 100 Hz
[ WARN] [1543970285.548950092]: No transform between frames map and cam0 available after 20.006891 seconds of waiting. This warning only prints once.

how to change the camera resolution and turn on color option

Can someone point me to the specific location where I can change the camera parameter? Or the relevant tutorial/demo having the option to do that?
I'm primarily playing with the esim_ros for the image plane demo in the wiki.

I'm also wondering how to turn on the color option to output color frames.

Much appreciated!

"Tried to insert a message with time less than ros::MIN_TIME"

Hi,
The following line throws a BagException "Tried to insert a message with time less than ros::MIN_TIME" if time happens to be equal to 0.5 * exposure_time_, since the RosbagWriter tries to insert a message with timestamp exactly 0 (https://github.com/uzh-rpg/rpg_esim/blob/1241667c3358c90ba207fc388fb85b42bb8af5da/event_camera_simulator/esim_visualization/src/rosbag_writer.cpp#L128), which is less than 1 ns (ros::MIN_TIME). Simply replacing >= with > should solve the issue.

rpg_esim/event_camera_simulator/esim/src/simulator.cpp

Line 110 in 1241667

const Time mid_exposure_time = (time >= 0.5 * exposure_time_) ? time - 0.5 * exposure_time_ : time;

Generally speaking, isn't it the case that, with the line above, mid_exposure_time - and therefore the timestamps of the corrupted images - can decrease even if time increases? In particular, if at a certain call of publishData time is equal to a < 0.5 * exposure_time_ and at a later call time is 0.5 * exposure_time_ + b, with 0 < b < a, then the mid_exposure_time at the first call will be a, whereas the one of the second call will be b < a.

Thank you

cloning dependencies fatal error. Permission denied (publickey).

When calling:

vcs-import < rpg_esim/dependencies.yaml

for each repository i get the following error like:

=== ./assimp_catkin (git) ===
Could not clone repository '[email protected]:uzh-rpg/assimp_catkin.git': Cloning into '.'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I think this is due to using the git@ style syntax. If the command is changed to the https style everything is cloned as expected.
e.g.

[email protected]:uzh-rpg/assimp_catkin.git'

changed to

https://github.com/uzh-rpg/assimp_catkin.git'

Can this be changed in the dependencies.yaml in the repo? or is there another work around?

Cheers.

is it possible to simulate multiple images at the same time ?

Hi, I just tried the simulator and it is nice.
but I am wondering whether it is possible to automatically read image folder (multiple images)?
Changing the image path command in the config to read single image is really cumbersome.
Thank you so much for your help.

Example for 3D OpenGL Rendering Engine

Hi, thanks for the code! I have noticed that the examples provided for 3D OpenGL Rendering Engine
are only using very simple scene. I am wondering whether an example could be provided for a more realistic scene, such as chairs in a room, so that we can get a sense of how to use Blender? Thanks!

bug when contrast threshold < 0

It seems there will be a bug when contrast threshold smaller than zero (e.g CHECK_GE(t, last_stamp_at_xy) will failed). That is not rare. For example, if i set constrast_threshold to 0.10 and contrast_sigma to 0.05, it will be triggered often.

glGetError： opengl_renderer.cpp:438] something went wrong while reading pixels: 1282

when I run roslaunch esim_ros esim.launch config:=cfg/opengl.conf, I have get these errors:

I0907 11:57:29.156168 10852 data_provider_online_render.cpp:63] Horizontal FOV: 61.7164 deg
I0907 11:57:29.156272 10852 data_provider_online_render.cpp:64] Vertical FOV: 48.2168 deg
I0907 11:57:29.156277 10852 data_provider_online_render.cpp:65] Diagonal FOV: 73.4823 deg
I0907 11:57:29.444448 10852 model.h:54] will load model at: /home/ian/catkin_ws/src/rpg_esim/event_camera_simulator/imp/imp_opengl_renderer/resources/objects/sponza/sponza.obj
I0907 11:57:29.579762 10852 opengl_renderer.cpp:118] Successfully set up multisampled color and depth framebuffers
I0907 11:57:29.580186 10852 opengl_renderer.cpp:141] Successfully set up color and depth framebuffers
I0907 11:57:29.580565 10852 opengl_renderer.cpp:164] Successfully set up optic flow framebuffer
ERROR::SHADER::FILE_NOT_SUCCESFULLY_READ
ERROR::PROGRAM_LINKING_ERROR of type: PROGRAM
Vertex info

(0) : error C5145: must write to gl_Position

ERROR::SHADER::FILE_NOT_SUCCESFULLY_READ
ERROR::PROGRAM_LINKING_ERROR of type: PROGRAM
Vertex info

(0) : error C5145: must write to gl_Position

something went wrong while reading pixels: 502
E0907 11:57:29.595811 10852 opengl_renderer.cpp:438] something went wrong while reading pixels: 1282

Projection matrix P for /cam0/camera_info is unset

Only intrinsic matrix K, not the projection matrix P is set atm. image_geometry::PinholeCameraModel has method fx(), fy(), cx(), cy(), which reads from projection matrix P, not from K. http://docs.ros.org/diamondback/api/image_geometry/html/c++/classimage__geometry_1_1PinholeCameraModel.html

Complete Workflow for Converting RGB Video to Event Data

Hi supitalp,

Thanks a lot for your work! I finally installed your package successfully and got the 'out.bag' output. But I am not sure what is this output and how to use and visualize it.

I am wondering if it is possible to give us a complete explanation of the workflow for how to convert a RGB video to the event data using your simulator.

Looking forward to hearing from you.

Thanks,
Jeremy

ERROR: failed to set up multisampled framebuffer

When I launch the OpenGL renderer, I get a crash with the following log:

NODES
 /
   dvs_renderer (dvs_renderer_advanced/dvs_renderer_advanced)
   esim_node (esim_ros/esim_node)
   hector_trajectory_server (hector_trajectory_server/hector_trajectory_server)
   optic_flow_viz (esim_visualization/optic_flow_converter.py)
   sync (topic_tools/transform)

ROS_MASTER_URI=http://127.0.0.1:11311

process[esim_node-1]: started with pid [11771]
process[dvs_renderer-2]: started with pid [11772]
process[sync-3]: started with pid [11773]
process[optic_flow_viz-4]: started with pid [11774]
process[hector_trajectory_server-5]: started with pid [11802]
[ INFO] [1534428879.606363020]: Waiting for tf transform data between frames map and cam0 to become available
[ INFO] [1534428879.693591802]: Changing framerate to 30 Hz
[INFO] [1534428880.726421]: Started flow converter node
[INFO] [1534428880.726993]: Step size between arrows: 7
[INFO] [1534428880.727366]: Scale factor: 0.07
[INFO] [1534428880.727705]: Upsample factor: x1
[INFO] [1534428880.728048]: Publish rate: 100 Hz
I0816 16:14:41.142905 11771 data_provider_online_render.cpp:63] Horizontal FOV: 61.7164 deg
I0816 16:14:41.143277 11771 data_provider_online_render.cpp:64] Vertical FOV: 48.2168 deg
I0816 16:14:41.143301 11771 data_provider_online_render.cpp:65] Diagonal FOV: 73.4823 deg
I0816 16:14:42.228034 11771 model.h:54] will load model at: /home/ggb/ros/catkin_ws/src/rpg_event_camera_simulator/event_camera_simulator/imp/imp_opengl_renderer/resources/objects/flying_room/flying_room.obj
F0816 16:14:42.913523 11771 opengl_renderer.cpp:115] ERROR: failed to set up multisampled framebuffer
*** Check failure stack trace: ***
   @     0x7f770901b03d  google::LogMessage::Fail()
   @     0x7f770901cea3  google::LogMessage::SendToLog()
   @     0x7f770901abcb  google::LogMessage::Flush()
   @     0x7f770901d88e  google::LogMessageFatal::~LogMessageFatal()
   @     0x7f77050587f0  event_camera_simulator::OpenGLRenderer::init()
   @     0x7f7705059558  event_camera_simulator::OpenGLRenderer::setCamera()
   @     0x7f770a986667  event_camera_simulator::DataProviderOnlineMoving3DCameraRig::DataProviderOnlineMoving3DCameraRig()
   @     0x7f770a984af9  event_camera_simulator::loadDataProviderFromGflags()
   @           0x421e17  main
   @     0x7f77086cd830  __libc_start_main
   @           0x421b89  _start
[esim_node-1] process has died [pid 11771, exit code -6, cmd /home/ggb/ros/catkin_ws/devel/lib/esim_ros/esim_node --v=1 --vmodule=data_provider_from_folder=10 --flagfile=/home/ggb/ros/catkin_ws/src/rpg_event_camera_simulator/event_camera_simulator/esim_ros/cfg/opengl.conf __name:=esim_node __log:=/home/ggb/.ros/log/b282cc46-a159-11e8-a753-3c970e1b96f5/esim_node-1.log].
log file: /home/ggb/.ros/log/b282cc46-a159-11e8-a753-3c970e1b96f5/esim_node-1*.log
^C[hector_trajectory_server-5] killing on exit
[optic_flow_viz-4] killing on exit
[sync-3] killing on exit
[dvs_renderer-2] killing on exit
shutting down processing monitor...

Image scale in planar renderer

In the planar renderer how is the extend/size of the image set? i.e. In the conf file is there a way to set the size of the image bounds?. e.g. if I want to set the image bounds to be a 10 m square how can this be done?

Suggestion to change in the installation

Hi, I just compiled the ESIM on my Ubuntu 16.04 machine with ROS Kinetic. I just recently setup my machine so the OS is rather clean. I noticed two problems during the installation:

First, In this line: https://github.com/uzh-rpg/ze_oss/blob/master/ze_cmake/cmake/modules/ze_setup.cmake#L4

The option Werror is turned on so that some warnings in the pyconfig.h will be flagged as errors. With this, I couldn't compile ze_matplotlib

Second, while I was compiling ze_splines, it gives me the following errors:

/usr/lib/x86_64-linux-gnu/libapr-1.so.0: undefined reference to `uuid_generate@UUID_1.0'
collect2: error: ld returned 1 exit status
make[2]: *** [/home/dgywork/catkin_ws/devel/lib/ze_splines/viz_splines_test] Error 1
make[1]: *** [CMakeFiles/viz_splines_test.dir/all] Error 2
make: *** [all] Error 2

Turns out the version of libuuid supported by the Ubuntu 16.04 is too old. Since I have a conda on my machine, so I installed libuuid through conda to fix it:

conda install -c conda-forge libuuid 

Yuhuang.

Change position of 3D model

Hello! I am trying to use 3D opengl render of ESIM, but I did not find out how to change the position of input 3D model. My task required another another 3D scene for testing and I already had the 3D model. But when I just changed the file path parameter in config file, my 3D model was not fixed in a proper place. Therefore I want to figure out whether there is any way to set the position of 3D model?
Thank you in advance for your help!

Simulation goes wrong for the test case

When I launch :
roslaunch esim_ros esim.launch config:=cfg/example.conf

The package did run but pop out some error. And I am not able to get streams when launching the visualisation node.

F0713 01:10:49.247979 4073 camera_rig.cpp:106] Check failed: fileExists(FLAGS_calib_filename) Camera file does not exist.

[esim_node-2] process has died [pid 4073, exit code -6, cmd /home/engn6528/sim_ws/devel/lib/esim_ros/esim_node --v=1 --vmodule=data_provider_from_folder=10 --flagfile=/home/engn6528/sim_ws/src/rpg_esim/event_camera_simulator/esim_ros/cfg/example.conf __name:=esim_node __log:=/home/engn6528/.ros/log/3ad2778e-a4b7-11e9-b3f2-503eaaae86b5/esim_node-2.log].
log file: /home/engn6528/.ros/log/3ad2778e-a4b7-11e9-b3f2-503eaaae86b5/esim_node-2*.log

How to get the trajectory's moving function?

Eigen Assertion failure on OpenGL and Panoramic Renderers

After a successful catkin build esim_ros, I try to run roslaunch esim_ros esim.launch config:=cfg/opengl.conf, as per the wiki. I get the following output

`nacorti@nacorti-VirtualBox:~/sim_ws/src/rpg_esim/event_camera_simulator/esim_ros$ roslaunch esim_ros esim.launch config:=cfg/opengl.conf
... logging to /home/nacorti/.ros/log/f613546c-752f-11e9-a031-080027fdbf14/roslaunch-nacorti-VirtualBox-30451.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://nacorti-VirtualBox:34771/

SUMMARY

PARAMETERS

• /hector_trajectory_server/source_frame_name: cam0
• /hector_trajectory_server/target_frame_name: map
• /hector_trajectory_server/trajectory_publish_rate: 15
• /hector_trajectory_server/trajectory_update_rate: 15
• /optic_flow_viz/arrows_scale: 0.07
• /optic_flow_viz/arrows_step: 7
• /optic_flow_viz/arrows_upsample_factor: 1
• /optic_flow_viz/publish_rate: 100
• /rosdistro: kinetic
• /rosversion: 1.12.14

NODES
/
dvs_renderer (dvs_renderer/dvs_renderer)
esim_node (esim_ros/esim_node)
hector_trajectory_server (hector_trajectory_server/hector_trajectory_server)
optic_flow_viz (esim_visualization/optic_flow_converter.py)

auto-starting new master
process[master]: started with pid [30461]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to f613546c-752f-11e9-a031-080027fdbf14
process[rosout-1]: started with pid [30474]
started core service [/rosout]
process[esim_node-2]: started with pid [30489]
process[dvs_renderer-3]: started with pid [30492]
process[optic_flow_viz-4]: started with pid [30493]
process[hector_trajectory_server-5]: started with pid [30499]

esim_node: /usr/include/eigen3/Eigen/src/Core/DenseStorage.h:109: Eigen::internal::plain_array<T, Size, MatrixOrArrayOptions, 16>::plain_array() [with T = double; int Size = 6; int MatrixOrArrayOptions = 0]: Assertion (reinterpret_cast<size_t>(eigen_unaligned_array_assert_workaround_gcc47(array)) & (15)) == 0 && "this assertion is explained here: " "http://eigen.tuxfamily.org/dox-devel/group__TopicUnalignedArrayAssert.html" " **** READ THIS WEB PAGE !!! ****"' failed.

*** Aborted at 1557718448 (unix time) try "date -d @1557718448" if you are using GNU date ***
PC: @ 0xb7f16d05 ([vdso]+0xd04) (emphasis mine)
*** SIGABRT (@0x7719) received by PID 30489 (TID 0xb35baa80) from PID 30489; stack trace: *****
[ INFO] [1557718448.506023514]: Waiting for tf transform data between frames map and cam0 to become available
[esim_node-2] process has died [pid 30489, exit code -6, cmd /home/nacorti/sim_ws/devel/lib/esim_ros/esim_node --v=1 --vmodule=data_provider_from_folder=10 --flagfile=/home/nacorti/sim_ws/src/rpg_esim/event_camera_simulator/esim_ros/cfg/opengl.conf __name:=esim_node __log:=/home/nacorti/.ros/log/f613546c-752f-11e9-a031-080027fdbf14/esim_node-2.log].
log file: /home/nacorti/.ros/log/f613546c-752f-11e9-a031-080027fdbf14/esim_node-2*.log
[INFO] [1557718449.725550]: Started flow converter node
[INFO] [1557718449.733144]: Step size between arrows: 7
[INFO] [1557718449.733319]: Scale factor: 0.07
[INFO] [1557718449.733478]: Upsample factor: x1
[INFO] [1557718449.733651]: Publish rate: 100 Hz
^C[hector_trajectory_server-5] killing on exit
[optic_flow_viz-4] killing on exit
[dvs_renderer-3] killing on exit
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done
`

The same thing happens when I try and run the panoramic renderer. However using the planar and 2d_moving_object renderers works just fine. Just not sure what's causing that Eigen Assertion error. I read the attached web page but it seems to imply there's a code problem with the underlying package, which I doubt. My gut tells me it's the version of something that's off, but I can't tell what.

For reference I'm using Ubuntu 16.04 32-bit on a Virtualbox host.

OpenGL version: 3.0 Mesa 18.0.5
Python 2.7.12
GCC: version 5.4.0

I've reinstalled and gotten the same result. Is this a known issue, and is there any way to fix this? Can provide more system info on request

how to simulate motion blur?

Thanks for your amazing work!!
I am not sure that how do esim simulate the motion blur？Does it use some exposure simulation then when camera move fast, it would bring motion blur while camera move slowly it would not. Or does it just use some blur kernel like guassian kernel to process the image?

Check failed: fileExists(FLAGS_calib_filename) Camera file does not exist

I am trying to run the Photorealistic Rendering Engine that can be find at https://github.com/uzh-rpg/rpg_esim/wiki/Photorealistic-Rendering-Engine-based-on-Unreal-Engine
I have followed all steps explained , when I launch

roslaunch` esim_ros esim.launch config:=cfg/unrealcv.conf

I get the following error on ze_oss/ze_cameras/src/camera_rig.cpp line 106

F1212 12:19:18.864552 14026 camera_rig.cpp:106] Check failed: fileExists(FLAGS_calib_filename) Camera file does not exist.

here is the full output

`NODES
/
dvs_renderer (dvs_renderer/dvs_renderer)
esim_node (esim_ros/esim_node)
hector_trajectory_server (hector_trajectory_server/hector_trajectory_server)
optic_flow_viz (esim_visualization/optic_flow_converter.py)

auto-starting new master
process[master]: started with pid [14000]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to c381b0c2-fdff-11e8-a56e-34e6d7284e42
process[rosout-1]: started with pid [14013]
started core service [/rosout]
process[esim_node-2]: started with pid [14026]
process[dvs_renderer-3]: started with pid [14031]
process[optic_flow_viz-4]: started with pid [14032]
process[hector_trajectory_server-5]: started with pid [14033]
[ INFO] [1544613558.281283420]: Waiting for tf transform data between frames map and cam0 to become available
I1212 12:19:18.846436 14026 imu_factory.cpp:20] Initialize bias ...
I1212 12:19:18.864478 14026 imu_factory.cpp:27] done.
I1212 12:19:18.864506 14026 imu_factory.cpp:35] Initialize IMU ...
I1212 12:19:18.864513 14026 imu_factory.cpp:44] done.

F1212 12:19:18.864552 14026 camera_rig.cpp:106] Check failed: fileExists(FLAGS_calib_filename) Camera file does not exist.
*** Check failure stack trace: ***
@ 0x7f9bb2d9d43d google::LogMessage::Fail()
@ 0x7f9bb2d9f253 google::LogMessage::SendToLog()
@ 0x7f9bb2d9cfcb google::LogMessage::Flush()
@ 0x7f9bb2d9fc3e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f9bae56cf57 ze::cameraRigFromGflags()
@ 0x7f9bb437c16b event_camera_simulator::DataProviderOnlineMoving3DCameraRig::DataProviderOnlineMoving3DCameraRig()
@ 0x7f9bb4374a97 event_camera_simulator::loadDataProviderFromGflags()
@ 0x404e4e main
@ 0x7f9bb222d830 __libc_start_main
@ 0x4057c9 _start
[INFO] [1544613559.038977]: Started flow converter node
[INFO] [1544613559.039281]: Step size between arrows: 7
[INFO] [1544613559.039456]: Scale factor: 0.07
[INFO] [1544613559.039638]: Upsample factor: x1
[INFO] [1544613559.039797]: Publish rate: 100 Hz

[esim_node-2] process has died [pid 14026, exit code -6, cmd /home/oaiepc/sim_ws/devel/lib/esim_ros/esim_node --v=1 --vmodule=data_provider_from_folder=10 --flagfile=/home/oaiepc/sim_ws/src/rpg_esim/event_camera_simulator/esim_ros/cfg/unrealcv.conf __name:=esim_node __log:=/home/oaiepc/.ros/log/c381b0c2-fdff-11e8-a56e-34e6d7284e42/esim_node-2.log].
log file: /home/oaiepc/.ros/log/c381b0c2-fdff-11e8-a56e-34e6d7284e42/esim_node-2*.log
[ WARN] [1544613578.283323455]: No transform between frames map and cam0 available after 20.002056 seconds of waiting. This warning only prints once.`

Any help would be really appreciated
thanks

Get simulated events

Hello,

The compilation worked fine and the simulation is running.
I can visualize the simulation results properly but is there a way to get the simulated events created (x, y, timestamp, polarity) ?

ERROR: cannot launch node of type [esim_ros/esim_node]: can't locate node [esim_node] in package [esim_ros]

Hello, All
I have completed installation of rpg_esim (https://github.com/uzh-rpg/rpg_esim/wiki/Installation)
I am trying to install Planar Renderer (https://github.com/uzh-rpg/rpg_esim/wiki/Planar-Renderer).

1. Run two commands as follows

himanshu@himanshu-pc:~/sim_ws/src/rpg_esim/event_camera_simulator$ roscd esim_ros/
himanshu@himanshu-pc:~/sim_ws/src/rpg_esim/event_camera_simulator/esim_ros$ gedit cfg/example.conf

2. Then i changed the absolute path
   

3. Run below code
   roslaunch esim_ros esim.launch config:=cfg/example.conf

But, I got below error.

How to disable image replication on the borders? How to change the size of the simulated images?

When I try to use various images to your ESIM, Especially, when the width of image is much bigger than the height of the image, your ESIM cropped the width of image and duplicate image on the top and bottom of the image.
I think your ESIM try to make the image specific size.
Can I change of that?

I attached the input image and visualization result image.

which version of opencv did you use?

I got an error:

rpg_dvs_ros/dvs_renderer/src/renderer.cpp:193:11: error: ‘undistort’ is not a member of ‘cv’

when build the esim_ros. So which version of opencv did you use? Or how to resolve it? I am using opencv 4.0.

Rendering of frame and events in Multi Objects 2D renderer are not in sync

In the Multi Objects 2D renderer, there seems to be an offset between the position of the object(in this case rugby ball) to the corresponding events that must be generated. This effect happens both while running the code on my system and in the demo video given in the wiki (https://www.youtube.com/watch?v=Yre8iS2LqPc&feature=youtu.be). Shouldn't the frame and events generated be in sync?

How can I simulate a shaking camera motion?

I want to make the camera shaking to locate the object in the similar position.
I mean I want to move the camera fast and small amount.

How can I make the camera trajectory?

Add automated rendering from youtube input?

I am a hobbiest researcher with limited time; i.e I have a full-time non-research job :-(.

Could you enhance you simulator so that it could render continuously automatically from an online URL such as a youtube video or a video file / video stream?

I want to work on creating visual / augmented modelling of word understanding based on SLAM cotextual recognition of a small set of every important words.

See https://hemseye.org/wp/2018/08/26/hemseye-project-phase-1-shorter-presentation/

Further enhancments would be adding a sync / timestamp to integrate the video sound output.

Many thanks

regards

James Kitching

esim_ros was abandoned during catkin build esim_ros

Hi, I was not able to build esim_ros. My output after $catkin build esim_ros was:

Errors << glog_catkin:make /mnt/f/Winston/sim_ws/logs/glog_catkin/build.make.001.log
/bin/sh: 1: autoreconf: not found
make[2]: *** [glog_src-prefix/src/glog_src-stamp/glog_src-configure] Error 127
make[1]: *** [CMakeFiles/glog_src.dir/all] Error 2
make: *** [all] Error 2
cd /mnt/f/Winston/sim_ws/build/glog_catkin; catkin build --get-env glog_catkin | catkin env -si /usr/bin/make --jobserver-fds=6,7 -j; cd -
.......................................................................................................................
Failed << glog_catkin:make [ Exited with code 2 ]
Failed <<< glog_catkin [ 0.5 seconds ]
Abandoned <<< eigen_checks [ Unrelated job failed ]
Abandoned <<< esim_unrealcv_bridge [ Unrelated job failed ]
Abandoned <<< minkindr [ Unrelated job failed ]
Abandoned <<< minkindr_conversions [ Unrelated job failed ]
Abandoned <<< ze_common [ Unrelated job failed ]
Abandoned <<< imp_core [ Unrelated job failed ]
Abandoned <<< imp_bridge_opencv [ Unrelated job failed ]
Abandoned <<< imp_bridge_ros [ Unrelated job failed ]
Abandoned <<< ze_cameras [ Unrelated job failed ]
Abandoned <<< esim_common [ Unrelated job failed ]
Abandoned <<< esim_rendering [ Unrelated job failed ]
Abandoned <<< esim_visualization [ Unrelated job failed ]
Abandoned <<< imp_multi_objects_2d [ Unrelated job failed ]
Abandoned <<< imp_opengl_renderer [ Unrelated job failed ]
Abandoned <<< imp_panorama_renderer [ Unrelated job failed ]
Abandoned <<< imp_planar_renderer [ Unrelated job failed ]
Abandoned <<< imp_unrealcv_renderer [ Unrelated job failed ]
Abandoned <<< ze_matplotlib [ Unrelated job failed ]
Abandoned <<< ze_ros [ Unrelated job failed ]
Abandoned <<< ze_visualization [ Unrelated job failed ]
Abandoned <<< ze_splines [ Unrelated job failed ]
Abandoned <<< ze_vi_simulation [ Unrelated job failed ]
Abandoned <<< esim_trajectory [ Unrelated job failed ]
Abandoned <<< esim_data_provider [ Unrelated job failed ]
Abandoned <<< esim [ Unrelated job failed ]
Abandoned <<< esim_ros [ Unrelated job failed ]
Finished <<< dvs_renderer [ 1.1 seconds ]
[build] Summary: 9 of 36 packages succeeded.
[build] Ignored: 8 packages were skipped or are blacklisted.
[build] Warnings: None.
[build] Abandoned: 26 packages were abandoned.
[build] Failed: 1 packages failed.
[build] Runtime: 3.7 seconds total.

I was using the Ubuntu 16.04 app on Windows 10. Did anybody encounter this?

build error for assimp_catkin

I met some problem while compiling the assimp library. The terrible campus network turn out to be blame. At the beginning, I have access to the Internet and Github, so I didn't notice it was the problem of network. After sending an email to Henri Rebecq, I found one important line which I ignored due to its light gray color on the window:

cd /home/xxc/sim_ws/build/assimp_catkin; catkin build --get-env assimp_catkin | catkin env -sl /usr/bin/make --jobserver-fds=6,7 -j; cd -

When I run the above command, the output showed that the problem had been caused by the downloading failure of "assimp-4.0.1.zip" from "https://github.com/assimp/assimp/archive/v4.0.1.zip".

About the same time, Henri replied my email and reminded me of the network problem. Finally, I changed the network and complied the ESIM successfully. I really appreciate the work and advice of Henri. I write it down here in case someone else got stuck in the same problem.

Camera trajectory different to the trajectory in CSV file

Hello,

I checked that among the options of the configuration file there is the possibility to define the trajectory of the camera using a CSV file. I tried one of your examples, but the camera does not follow exactly the trajectory defined in the CSV file. In the code, the trajectory is read, but then a spline is computed as the trajectory to follow.

I am wondering if in your code there is a way in which the camera follows exactly the trajectory defined in the CSV file. Is this possible?

using ESIM for object detection

Hi
Could be possible to use ESIM to process realtime video and use the events to make object recognition from drones performing trajectories at high speed ?
Thanks !
Germal

Is there a docker file available?

Hi, is there a docker image for it? Thanks!

Different trajectory in output rosbag to the input csv file

Possibile bug/typo in data_provider_online_render.cpp

Good evening,

Should the following line in data_provider_online_render.cpp be else if(next_t_flow_ < next_t_imu && next_t_flow_ < next_t_frame) instead?

rpg_esim/event_camera_simulator/esim_data_provider/src/data_provider_online_render.cpp

Line 368 in 1241667

else if(next_t_flow_ < next_t_imu && next_t_flow_ < next_t_imu)

(Sampling due to adaptive sampling, rather than due to minimum framerate/IMU rate).
With the current version, if I am not wrong, the else if above could in principle also capture the case in which next_t_frame < next_t_flow_ < next_t_imu, which should instead be handled by the later else if, at

rpg_esim/event_camera_simulator/esim_data_provider/src/data_provider_online_render.cpp

Line 375 in 1241667

else if(next_t_frame < next_t_imu && next_t_frame < next_t_flow_)

In practice, I believe that there should be no difference, since next_t_flow_ is always smaller than next_t_frame, as long as the delta_t of the adaptive sampling is smaller than dt_frame_^*, but I thought that it could be worth asking for confirmation.

Thank you

---

^*This by looking at the following lines:

rpg_esim/event_camera_simulator/esim_data_provider/src/data_provider_online_render.cpp

Line 293 in 1241667

next_t_flow_ = t_ + delta_t;

rpg_esim/event_camera_simulator/esim_data_provider/src/data_provider_online_render.cpp

Line 296 in 1241667

last_t_frame_ = t_;

rpg_esim/event_camera_simulator/esim_data_provider/src/data_provider_online_render.cpp

Line 354 in 1241667

const ze::real_t next_t_frame = last_t_frame_ + dt_frame_;

unable to install (error happened with catkin build esim_ros)

H, I am installing ESIM, but no matter how I tried, there always comes with error when I start 'catkin build esim_ros'. Could you give some help on why this happened?
Besides, I wonder what are the requirements of python version, cmake and gcc versions ? Aso is it possible to use conda python instead of pip ? it seems esim is contradicting to conda.
Thanks four your help.

The UnrealCV socket connection does not terminate properly on Ctrl+C program termination

when I Ctrl+C the program, the socket connections do not seem to close cleanly
which results in unrealcv complaining about a bad socket close. it then refuses to accept new connections and I have to launch playground.sh again.

Logs:
I0605 10:40:23.718904 28423 unrealcv_client.cpp:188] GET:51 ok I0605 10:40:23.749017 28423 unrealcv_client.cpp:205] SET:52 vget /camera/0/lit png I0605 10:40:23.974321 28423 unrealcv_client.cpp:205] SET:53 vget /camera/0/depth npy I0605 10:40:24.065693 28423 unrealcv_client.cpp:169] SET:54 vset /camera/0/pose -78.14488 -145.26709 128.48589 -5.03869 -36.67653 3.41916 I0605 10:40:24.095824 28423 unrealcv_client.cpp:188] GET:54 ok I0605 10:40:24.125928 28423 unrealcv_client.cpp:205] SET:55 vget /camera/0/lit png I0605 10:40:24.361157 28423 unrealcv_client.cpp:205] SET:56 vget /camera/0/depth npy I0605 10:40:24.474545 28423 unrealcv_client.cpp:169] SET:57 vset /camera/0/pose -78.20212 -145.05523 128.44524 -5.03921 -36.62993 3.41086 I0605 10:40:24.504691 28423 unrealcv_client.cpp:188] GET:57 ok I0605 10:40:24.534803 28423 unrealcv_client.cpp:205] SET:58 vget /camera/0/lit png ^CI0605 10:40:24.765202 28423 unrealcv_client.cpp:205] SET:59 vget /camera/0/depth npy [hector_trajectory_server-4] killing on exit [optic_flow_viz-3] killing on exit [dvs_renderer-2] killing on exit [esim_node-1] killing on exit W0605 10:40:24.826046 28423 signal_handler.cpp:46] Signal handler was called with signal 2 terminate called after throwing an instance of 'boost::system::system_error' what(): Interrupted system call *** Aborted at 1528188024 (unix time) try "date -d @1528188024" if you are using GNU date *** PC: @ 0x7fd002949428 gsignal *** SIGABRT (@0x3e800006f07) received by PID 28423 (TID 0x7fd005030780) from PID 28423; stack trace: *** @ 0x7fd0029494b0 (unknown) @ 0x7fd002949428 gsignal @ 0x7fd00294b02a abort @ 0x7fd002f8384d __gnu_cxx::__verbose_terminate_handler() @ 0x7fd002f816b6 (unknown) @ 0x7fd002f81701 std::terminate() @ 0x7fd002f81919 __cxa_throw @ 0x7fcfff2f1bfc event_camera_simulator::UnrealCvClient::handleError() @ 0x7fcfff302001 event_camera_simulator::UnrealCvClient::receive<>() @ 0x7fcfff3025eb event_camera_simulator::UnrealCvClient::sendCommand<>() @ 0x7fcfff2f3f3e event_camera_simulator::UnrealCvClient::getDepth() @ 0x7fd001b36c77 event_camera_simulator::UnrealCvRenderer::render() @ 0x7fd004a5a83e event_camera_simulator::DataProviderOnlineRender::sampleFrame() @ 0x7fd004a5dfee event_camera_simulator::DataProviderOnlineRender::spinOnce() @ 0x7fd004a57ac9 event_camera_simulator::DataProviderBase::spin() @ 0x4055f5 main @ 0x7fd002934830 __libc_start_main @ 0x405cc9 _start shutting down processing monitor... ... shutting down processing monitor complete done

ra/0/pose -78.20212 -145.05523 128.44524 -5.03921 -36.62993 3.41086 [2018.06.05-08.40.24:492][849]LogUnrealCV:Warning: Response: ok [2018.06.05-08.40.24:536][851]LogUnrealCV:Warning: Request: 58:vget /camera/0/lit png [2018.06.05-08.40.24:536][851]LogUnrealCV:Warning: Receive message 58:vget /camera/0/lit png [2018.06.05-08.40.24:716][851]LogUnrealCV:Warning: Response: ok [2018.06.05-08.40.24:768][855]LogUnrealCV:Warning: Request: 59:vget /camera/0/depth npy [2018.06.05-08.40.24:768][855]LogUnrealCV:Warning: Receive message 59:vget /camera/0/depth npy [2018.06.05-08.40.24:846][855]LogUnrealCV:Warning: Response: ok [2018.06.05-08.40.24:958][861]LogUnrealCV:Error: Unexpected error of socket happend, error SE_ECONNRESET [2018.06.05-08.40.24:958][861]LogUnrealCV: Client disconnected.

Failed << assimp_catkin:make [ Exited with code 2 ]

There was a error "Failed << assimp_catkin:make [ Exited with code 2 ]" when I run "catkin build esim_ros"

make[2]: *** [assimp_src-prefix/src/assimp_src-stamp/assimp_src-download] Error 1
make[1]: *** [CMakeFiles/assimp_src.dir/all] Error 2
make: *** [all] Error 2
cd /home/sss/sim_ws/build/assimp_catkin; catkin build --get-env assimp_catkin | catkin env -si /usr/bin/make --jobserver-fds=6,7 -j; cd -
...........................................................................................................................................................................................................
Failed << assimp_catkin:make [ Exited with code 2 ]
Failed <<< assimp_catkin [ 27 minutes and 6.6 seconds ]
Abandoned <<< imp_opengl_renderer [ Unrelated job failed ]
Abandoned <<< esim_data_provider [ Unrelated job failed ]
Abandoned <<< esim [ Unrelated job failed ]
Abandoned <<< esim_ros [ Unrelated job failed ]
[build] Summary: 32 of 37 packages succeeded.
[build] Ignored: 8 packages were skipped or are blacklisted.
[build] Warnings: 3 packages succeeded with warnings.
[build] Abandoned: 4 packages were abandoned.
[build] Failed: 1 packages failed.
[build] Runtime: 27 minutes and 13.5 seconds total.
[build] Note: Workspace packages have changed, please re-source setup files to use them.

Different trajectory in output rosbag to the input csv file

Hello! I tried to use "csv input" mode of ESIM to input specific trajectories, while I found sometimes the actual trajectories didn't fit well to the input. Then I carried out several simple experiments on it.

The trajectory I generated drove the object moving on only x-axis in constant velocity, keeping y and z in 0. It turned out that the actual path was bad. Considering the usage of spline interpolation, I generated a acceleration on x axis and the result seems quite great ( showed in the following figures)

Then I created a circular motion trajectory on x-y plane in constant angular velocity and the output fitted the expected path well too. ( traj, x, y comparison showed in the following figures)

However, the problem occured again when I attempted to input a spiral motion in 3 dimension ( see in the first following figure). The output path appeared as severe shrink comparing to the expected trajectory. And it was strange that motion on x and y axis were shrinking, while the motion on z axis seems acceptable. In this case, object followed circular motion on x-y plane, and acceleration on z axis. (details in the figures)




I couldn't figure out why it happened and how I should fixed this bug, so would you mind give me suggestions on it?
Notes:

• I got the output trajectories from the output rosbag file, whose path was defined in the configuration files.
• All of my attempts were under opengl renderers.
• I changed the spline segments number to 200 as your recommendation in the previous similar issue. Meanwhile, I also set all the "multipliers" to 1 although I didn't know their exact function in simulator. I would appreciate it very much if you could tell their roles in the simulator.

Thank you in advance for your help!