Stack smashing detected about cartographer_ros

[bochen87]

I found the error, the suitsparse library was still in the catkin workspace devel folder. I had to remove it from there as well and it works now.

[skohlbr]

Also ran into the stack smashing issue, similar to what is described in the OP. In my case it was caused by another repo containing a suitesparse package (https://github.com/ethz-asl/kalibr/tree/master/suitesparse). After removing that and doing a clean build, the problem is gone. Adding this here in the interest of people googling for stack smashing cartographer who probably end up reading this issue :)

[rohbotics]

If I disable the stack protector by adding -o overflow -fno-stack-protector to the compiler flags of ceres and cartographer_ros, it runs for 30sec - 1min before segfaulting.

(gdb) backtrace
#0  0x00007ffff7a2efcf in ceres::internal::ReorderProgramForSparseNormalCholesky(ceres::SparseLinearAlgebraLibraryType, ceres::OrderedGroups<double*> const&, ceres::internal::Program*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1
#1  0x00007ffff7a4e2dd in ceres::internal::TrustRegionPreprocessor::Preprocess(ceres::Solver::Options const&, ceres::internal::ProblemImpl*, ceres::internal::PreprocessedProblem*) ()
   from /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1
#2  0x00007ffff7a463f4 in ceres::Solver::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) () from /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1
#3  0x00007ffff7a46edf in ceres::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) () from /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1
#4  0x0000000000601505 in cartographer::mapping_2d::sparse_pose_graph::OptimizationProblem::Solve(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&, std::vector<cartographer::mapping::Submaps const*, std::allocator<cartographer::mapping::Submaps const*> > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*) ()
#5  0x00000000005f69a9 in cartographer::mapping_2d::SparsePoseGraph::RunOptimization() ()
#6  0x00000000005f752d in std::_Function_handler<void (std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&), cartographer::mapping_2d::SparsePoseGraph::HandleScanQueue()::{lambda(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&) ()
#7  0x000000000060639a in cartographer::mapping_2d::sparse_pose_graph::ConstraintBuilder::FinishComputation(int) ()
#8  0x000000000066351d in cartographer::common::ThreadPool::DoWork() ()
#9  0x00007ffff5186c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007ffff6d6a6fa in start_thread (arg=0x7fffdf7fe700) at pthread_create.c:333
#11 0x00007ffff4bf5b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[rohbotics]

I should add that the segfault also disappears while running under ASan, but not TSan or GDB.

[rohbotics]

I got the segfault to happen in valgrind

Here is the output:

==15634== Memcheck, a memory error detector
==15634== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==15634== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==15634== Command: /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node __name:=cartographer __log:=/home/rohan/.ros/log/d8b26ef8-5b38-11e6-863d-801f0286f30c/cartographer-1.log
==15634== 
[ INFO] [1470423499.181136001]: I0805 11:58:19.000000 15634 submaps.cc:175] Added submap 1
[ INFO] [1470423500.068260510]: I0805 11:58:20.000000 15634 motion_filter.cc:42] Motion filter reduced the number of scans to -nan%.
[ INFO] [1470423561.852290134]: I0805 11:59:21.000000 15634 motion_filter.cc:42] Motion filter reduced the number of scans to 5.4%.
==15634== Thread 8:
==15634== Invalid read of size 8
==15634==    at 0x4F7BFCF: ceres::internal::ReorderProgramForSparseNormalCholesky(ceres::SparseLinearAlgebraLibraryType, ceres::OrderedGroups<double*> const&, ceres::internal::Program*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F9B2DC: ceres::internal::TrustRegionPreprocessor::Preprocess(ceres::Solver::Options const&, ceres::internal::ProblemImpl*, ceres::internal::PreprocessedProblem*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F933F3: ceres::Solver::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F93EDE: ceres::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x601504: cartographer::mapping_2d::sparse_pose_graph::OptimizationProblem::Solve(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&, std::vector<cartographer::mapping::Submaps const*, std::allocator<cartographer::mapping::Submaps const*> > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x5F69A8: cartographer::mapping_2d::SparsePoseGraph::RunOptimization() (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x5F752C: std::_Function_handler<void (std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&), cartographer::mapping_2d::SparsePoseGraph::HandleScanQueue()::{lambda(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x606399: cartographer::mapping_2d::sparse_pose_graph::ConstraintBuilder::FinishComputation(int) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x66351C: cartographer::common::ThreadPool::DoWork() (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x7879C7F: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==15634==    by 0x5C986F9: start_thread (pthread_create.c:333)
==15634==  Address 0x8 is not stack'd, malloc'd or (recently) free'd
==15634== 
==15634== 
==15634== Process terminating with default action of signal 11 (SIGSEGV)
==15634==  Access not within mapped region at address 0x8
==15634==    at 0x4F7BFCF: ceres::internal::ReorderProgramForSparseNormalCholesky(ceres::SparseLinearAlgebraLibraryType, ceres::OrderedGroups<double*> const&, ceres::internal::Program*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F9B2DC: ceres::internal::TrustRegionPreprocessor::Preprocess(ceres::Solver::Options const&, ceres::internal::ProblemImpl*, ceres::internal::PreprocessedProblem*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F933F3: ceres::Solver::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F93EDE: ceres::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x601504: cartographer::mapping_2d::sparse_pose_graph::OptimizationProblem::Solve(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&, std::vector<cartographer::mapping::Submaps const*, std::allocator<cartographer::mapping::Submaps const*> > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x5F69A8: cartographer::mapping_2d::SparsePoseGraph::RunOptimization() (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x5F752C: std::_Function_handler<void (std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&), cartographer::mapping_2d::SparsePoseGraph::HandleScanQueue()::{lambda(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x606399: cartographer::mapping_2d::sparse_pose_graph::ConstraintBuilder::FinishComputation(int) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x66351C: cartographer::common::ThreadPool::DoWork() (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x7879C7F: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==15634==    by 0x5C986F9: start_thread (pthread_create.c:333)
==15634==    by 0x7E5FB5C: clone (clone.S:109)
==15634==  If you believe this happened as a result of a stack
==15634==  overflow in your program's main thread (unlikely but
==15634==  possible), you can try to increase the size of the
==15634==  main thread stack using the --main-stacksize= flag.
==15634==  The main thread stack size used in this run was 8388608.
==15634== 
==15634== HEAP SUMMARY:
==15634==     in use at exit: 131,395,694 bytes in 23,975 blocks
==15634==   total heap usage: 7,911,062 allocs, 7,887,087 frees, 1,637,650,075 bytes allocated
==15634== 
==15634== LEAK SUMMARY:
==15634==    definitely lost: 184 bytes in 1 blocks
==15634==    indirectly lost: 0 bytes in 0 blocks
==15634==      possibly lost: 4,800 bytes in 15 blocks
==15634==    still reachable: 131,390,710 bytes in 23,959 blocks
==15634==         suppressed: 0 bytes in 0 blocks
==15634== Rerun with --leak-check=full to see details of leaked memory
==15634== 
==15634== For counts of detected and suppressed errors, rerun with: -v
==15634== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

[rohbotics]

Here is a bag file that reproduces the error:

https://drive.google.com/file/d/0B2W9KeFIetG2NW9mNm5relQ0WEU/view?usp=sharing

[wohe]

Hi, I looked into it a bit and managed to reproduce the issue. I used the "Standalone Workspace" instructions for building on xenial/kinetic.

On my trusty/indigo system these instructions work without the issue. Likewise, installing as described in https://github.com/googlecartographer/cartographer, and using catkin only for cartographer and cartographer_ros (and not ceres_catkin) also works on xenial/kinetic. So this could at least be a short term workaround.

All crashes so far seem to be Ceres related, and can even be reproduced by some of the unit tests. A pretty small test which crashes on my system is https://github.com/googlecartographer/cartographer/blob/master/cartographer/mapping_2d/scan_matching/ceres_scan_matcher_test.cc.

[sandwichmaker]

@wohe could you share a smaller reproduction with me if possible? and i will work on fixing it?

[rohbotics]

This might be a bug in ceres_catkin, here it is checking out a commit (6a13e39) from the ceres repo, and saying that it is v1.11, https://github.com/ethz-asl/ceres_catkin/blob/master/CMakeLists.txt#L12.

In the github repo, v1.11 has is tagged as commit 4145556.
https://github.com/ceres-solver/ceres-solver/releases/tag/1.11.0

[wohe]

Already checked that. The hash is referring to upstream at https://ceres-solver.googlesource.com/ceres-solver/ and is indeed 1.11. Newer version and removing patches also did not help.

[wohe]

Shortest reproduction I have so far is on a Ubuntu 16.04 system with ROS kinetic installed.

sudo apt-get install \
  g++ \
  google-mock \
  libboost-all-dev \
  libgflags-dev \
  libgoogle-glog-dev \
  liblua5.2-dev \
  libprotobuf-dev \
  libsuitesparse-dev \
  libwebp-dev \
  ninja-build \
  protobuf-compiler \
  python-sphinx

mkdir -p ~/catkin_ws/src
cd ~/catkin_ws/src
source /opt/ros/kinetic/setup.bash
catkin_init_workspace

git clone https://github.com/googlecartographer/cartographer.git
git clone https://github.com/ethz-asl/ceres_catkin.git
git clone https://github.com/ethz-asl/suitesparse.git
git clone https://github.com/ethz-asl/glog_catkin.git
git clone https://github.com/ethz-asl/gflags_catkin.git
git clone https://github.com/ethz-asl/catkin_simple.git

cd ~/catkin_ws
catkin_make_isolated

Then several tests crash:

build_isolated/cartographer/devel/cartographer/mapping_2d/scan_matching/mapping_2d_scan_matching_ceres_scan_matcher_test
build_isolated/cartographer/devel/cartographer/mapping_2d/mapping_2d_sparse_pose_graph_test
build_isolated/cartographer/devel/cartographer/mapping_3d/sparse_pose_graph/mapping_3d_sparse_pose_graph_optimization_problem_test

[sandwichmaker]

is it possible for compile in debug mode and get a more detailed stack trace?

[wohe]

I investigated a bit more, and found the following:
The two functions from Ceres that lead to crashes (ComputeCovarianceValuesUsingSuiteSparseQR and OrderingForSparseNormalCholeskyUsingSuiteSparse) both have UsingSuiteSparse in their name, and it turns out that the version of catkin suitesparse and trusty's SuiteSparse are both 4.2.1, but xenial has 4.4.6. I uninstalled the xenial libsuitesparse-dev (which should be entirely unused in this build) and the stack smashing bug went away.

My conclusion: The catkin builds of Ceres and SuiteSparse are not ignoring the system's SuiteSparse as they should and compile/link together parts belonging to different versions of SuiteSparse. I don't know enough of catkin to fix this easily. Any ideas?

It seemed like it could be an ideal solution on xenial to remove the catkin build of Ceres, and just use xenial's libceres-dev. Unfortunately there is https://bugs.launchpad.net/ubuntu/+source/ceres-solver/+bug/1596296.

[rohbotics]

We can add FindCeres.cmake as part of ROS's cmake_modules (https://github.com/ros/cmake_modules) and add that as a dependency.

[wohe]

I think I found a solution that allows us to fix it on our side for now. @rohbotics, can you verify that #18 solves the issue for you? I tried it here with trusty/indigo and xenial/kinetic. Both seem fine.

[rohbotics]

#18 fixes this issue

[bochen87]

Hi, I'm having the stack smashing as well.

roslaunch cartographer_ros demo_backpack_2d.launch bag_filename:=${HOME}/Downloads/cartographer_paper_deutsches_museum.bag
... logging to /home/bo/.ros/log/aa0e2930-07d5-11e7-a596-e4a7a035a11f/roslaunch-bo-ThinkPad-T460p-10557.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://bo-ThinkPad-T460p:34053/

SUMMARY
========

PARAMETERS
 * /robot_description: <...>
 * /rosdistro: kinetic
 * /rosversion: 1.12.7
 * /use_sim_time: True

NODES
  /
    cartographer_node (cartographer_ros/cartographer_node)
    playbag (rosbag/play)
    robot_state_publisher (robot_state_publisher/robot_state_publisher)
    rviz (rviz/rviz)

auto-starting new master
process[master]: started with pid [10570]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to aa0e2930-07d5-11e7-a596-e4a7a035a11f
process[rosout-1]: started with pid [10583]
started core service [/rosout]
process[robot_state_publisher-2]: started with pid [10590]
process[cartographer_node-3]: started with pid [10602]
process[rviz-4]: started with pid [10610]
process[playbag-5]: started with pid [10618]
[ INFO] [1489399992.671926019]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer_ros/configuration_files/backpack_2d.lua' for 'backpack_2d.lua'.
[ INFO] [1489399992.672306517]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/map_builder.lua' for 'map_builder.lua'.
[ INFO] [1489399992.672354654]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/map_builder.lua' for 'map_builder.lua'.
[ INFO] [1489399992.672440033]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/trajectory_builder_2d.lua' for 'trajectory_builder_2d.lua'.
[ INFO] [1489399992.672481481]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/trajectory_builder_2d.lua' for 'trajectory_builder_2d.lua'.
[ INFO] [1489399992.672639777]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/trajectory_builder_3d.lua' for 'trajectory_builder_3d.lua'.
[ INFO] [1489399992.672681100]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/trajectory_builder_3d.lua' for 'trajectory_builder_3d.lua'.
[ INFO] [1489399992.672858258]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/sparse_pose_graph.lua' for 'sparse_pose_graph.lua'.
[ INFO] [1489399992.672896972]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/sparse_pose_graph.lua' for 'sparse_pose_graph.lua'.
[ INFO] [1489399992.792620165]: I0313 11:13:12.000000 10602 submaps.cc:176] Added submap 1
[ INFO] [1489399992.792723199]: I0313 11:13:12.000000 10602 map_builder_bridge.cc:37] Added trajectory with ID '0'.
[ INFO] [1489399993.392465301, 1432647016.509920142]: I0313 11:13:13.000000 10602 ordered_multi_queue.cc:172] All sensor data for trajectory 0 is available starting at '635682438164885689'.
*** stack smashing detected ***: /home/bo/catkin_ws/install_isolated/lib/cartographer_ros/cartographer_node terminated
[cartographer_node-3] process has died [pid 10602, exit code -6, cmd /home/bo/catkin_ws/install_isolated/lib/cartographer_ros/cartographer_node -configuration_directory /home/bo/catkin_ws/install_isolated/share/cartographer_ros/configuration_files -configuration_basename backpack_2d.lua echoes:=horizontal_laser_2d __name:=cartographer_node __log:=/home/bo/.ros/log/aa0e2930-07d5-11e7-a596-e4a7a035a11f/cartographer_node-3.log].
log file: /home/bo/.ros/log/aa0e2930-07d5-11e7-a596-e4a7a035a11f/cartographer_node-3*.log
0x1b98050 void QWindowPrivate::setTopLevelScreen(QScreen*, bool) ( QScreen(0xf95250) ): Attempt to set a screen on a child window.
0x1b98470 void QWindowPrivate::setTopLevelScreen(QScreen*, bool) ( QScreen(0xf95250) ): Attempt to set a screen on a child window.
0x1b8c830 void QWindowPrivate::setTopLevelScreen(QScreen*, bool) ( QScreen(0xf95250) ): Attempt to set a screen on a child window.
0x1b8b540 void QWindowPrivate::setTopLevelScreen(QScreen*, bool) ( QScreen(0xf95250) ): Attempt to set a screen on a child window.
^C[playbag-5] killing on exit
[rviz-4] killing on exit
[robot_state_publisher-2] killing on exit
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done

I saw that you changed to a different ceres package now. I have the OpenBLAS and Suitesparse libraries installed on my PC and the suitsparse package in my catkin workspace. I removed the suitesparse package, but it sill gives the same error. I also uninstalled the libsuitesparse-dev package and it still gives the same error.

[BrannonKing]

You can modify the CMakeLists.txt to not use SuitSparse (and use EigenSparse instead). It appears that there is a bug in the SuiteSparse (v4.4.6) that comes on Ubuntu 16.04. It wasn't obvious how to compile the newer SuiteSparse on Ubuntu and get it installed into the right folders.