personalrobotics / chimera

:snake: A CLI tool for generating Boost.Python/pybind11 bindings from C/C++

chimera

chi·me·ra

/kīˈmirə,kəˈmirə/

informal, noun

1. a thing that is hoped or wished for but in fact is illusory or impossible to achieve.
2. a utility to generate Boost.Python bindings for C++ code.

Chimera is a tool for generating Boost.Python/Pybind11 bindings from C/C++ header files. It uses the Clang/LLVM toolchain, making it capable of automatically handling fairly complex source files.

Usage

$ ./chimera -c <yaml_config_file> -o <output_path> my_cpp_header1.h my_cpp_header2.h -- [compiler args]

Installation

On Ubuntu using apt

Chimera provides Ubuntu packages for Xenial (16.04 LTS), Bionic (18.04 LTS), Eoan (19.10), and Focal (20.04).

Xenial and greater

$ sudo add-apt-repository ppa:personalrobotics/ppa
$ sudo apt update
$ sudo apt install chimera

On macOS using Homebrew

# Install the Homebrew package manager
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
# Add Homebrew tap for Personal Robotics Lab software
$ brew tap personalrobotics/tap
# Install Chimera
$ brew install chimera

Build from Source

Requirements

• libclang 6.0, 7, 8, 9
• llvm 6.0, 7, 8, 9 (+ tools)
• libedit
• yaml-cpp
• boost

On Ubuntu from Source

Install Dependencies

$ sudo apt-get install llvm-6.0-dev llvm-6.0-tools libclang-6.0-dev libedit-dev \
$   libyaml-cpp-dev libboost-dev lib32z1-dev

Build Chimera

$ git clone https://github.com/personalrobotics/chimera.git
$ cd chimera
$ mkdir build && cd build
$ cmake -DCMAKE_BUILD_TYPE=Release ..
$ make
$ sudo make install

On macOS from Source

$ brew install boost llvm yaml-cpp
$ brew install eigen  # for examples
$ git clone https://github.com/personalrobotics/chimera.git
$ cd chimera
$ mkdir build && cd build
$ PKG_CONFIG_PATH=$(brew --prefix yaml-cpp)/lib/pkgconfig cmake -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_DIR=$(brew --prefix llvm)/lib/cmake/llvm ..
$ make
$ sudo make install

Example

Let's try running chimera on itself!

$ cd [PATH TO CHIMERA]
$ rm -rf build && mkdir -p build && cd build
$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
$ make
$ chimera -p . -o chimera_py_binding.cpp ../src/chimera.cpp

Configuration

# The C++ namespaces that will be extracted by Chimera
namespaces:
  - dart::dynamics
  - dart::math

# Selected types that should have special handling.
# (Not implemented yet.)
types:
  'class BodyNode':
    convert_to: 'class BodyNodePtr'

# Selected function and class declarations that need custom parameters.
functions:
  'const Eigen::Vector4d & ::dart::dynamics::Shape::getRGBA() const':
    return_value_policy: copy_const_reference
  'bool ::dart::dynamics::Skeleton::isImpulseApplied() const':
    source: 'test.cpp.in'
  'const Eigen::Vector3d & ::dart::dynamics::Shape::getBoundingBoxDim() const':
    content: '/* Instead of implementing this function, insert this comment! */'
  'Eigen::VectorXd & ::dart::optimizer::GradientDescentSolver::getEqConstraintWeights()': null
    # This declaration will be suppressed.

classes:
  '::dart::dynamics::Shape':
    name: Shape
    bases: []
    noncopyable: true

Troubleshooting

Is there a length limit for the keys in the configuration file of Chimera?

Yes. yaml-cpp does not support more than 1024 characters for a single line key. If you want to use a longer key, then you should use multi-line key.

License

Chimera is released under the 3-clause BSD license. See LICENSE for more information.

Authors

Chimera is developed by Michael Koval (@mkoval) and Pras Velagapudi (@psigen), and it has received major contributions from Jeongseok Lee (@jslee02).

chimera's Issues

Generated class declaration is not fully qualified

::boost::python::class_<dart::common::Signal<void (std::shared_ptr<const dart::dynamics::MetaSkeleton>, const std::basic_string<char> &, const std::basic_string<char> &), signal::detail::DefaultCombiner>, ::boost::noncopyable >("Signal", boost::python::no_init)

Note that signal::detail::DefaultCombiner should be dart::common::signal::detail::DefaultCombiner.

Generating lambdas inside properties causes... issues.

Generates wrappers for templates that are not fully instantiated

In _ZN4dart6common6SignalIFT_DpT0_ET1_EE.cpp we generated:

::boost::python::class_<dart::common::Signal<type-parameter-0-0 (type-parameter-0-1...), Combiner>, ::boost::noncopyable >("Signal", boost::python::no_init)

Mark classes with no copy assignment operator as noncopyable

/home/mkoval/storage/dart-ws/devel/include/dart/common/Signal.h:240:27: note: copy assignment operator of 'SlotRegister<dart::common::Signal<void (const dart::dynamics::BodyNode *), signal::detail::DefaultCombiner> >' is implicitly deleted because field 'mSignal' is of reference type 'typename Signal<void (const BodyNode *), signal::detail::DefaultCombiner>::SignalType &' (aka 'Signal<void (const dart::dynamics::BodyNode *)> &')
  typename T::SignalType& mSignal;

Wrap overloaded operators

These are currently suppressed.

Generates invalid code for arguments that default to infinity

It should output something like std::numeric_limits<double>::infinity(). This will also likely break on special values like NaN.

Attempt to wrap incomplete types

This applies to types that are forward declared in the header file, but the full definition is not available. For example:

/usr/include/boost/python/type_id.hpp:85:9: error: 'typeid' of incomplete type 'Ipopt::IpoptData'
        typeid(T)

in _ZN4dart9optimizer8DartTNLPE.cpp.

Cling does not fully qualify array types

In _ZN4dart3gui6JitterE.cpp we generated:

.add_static_property("j2", []() { return dart::gui::Jitter::j2; }, [](jitter_point [2] value) { dart::gui::Jitter::j2 = value; })

Abstract classes are not marked as `noncopyable`.

Generate Python packages to match C++ namespaces

Invalid reference to add_static_property constexpr's

ImportError: ./dartpy.so: undefined symbol: _ZN4dart8dynamics38AddonWithProtectedPropertiesInSkeletonINS0_6detail15ScrewJointAddonENS2_26ScrewJointUniquePropertiesENS0_10ScrewJointEXadL_ZNS2_19JointPropertyUpdateIS3_EEvPT_EELb0EE8OptionalE

This expands to:

dart::dynamics::AddonWithProtectedPropertiesInSkeleton<dart::dynamics::detail::ScrewJointAddon, dart::dynamics::detail::ScrewJointUniqueProperties, dart::dynamics::ScrewJoint, &(void dart::dynamics::detail::JointPropertyUpdate<dart::dynamics::detail::ScrewJointAddon>(dart::dynamics::detail::ScrewJointAddon*)), false>::Optional

Missing typeinfo for some classes

E.g.:

$ python -c 'import dartpy'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: ./dartpy.so: undefined symbol: _ZTIN4dart9optimizer8DartTNLPE

_ZTIN4dart9optimizer8DartTNLPE demangles to typeinfo for dart::optimizer::DartTNLP.

Read compilation flags from the YAML configuration file

Wrap array member objects as readonly

In _ZN4dart5utils10c3d_head_tE.cpp we get the error:

/usr/include/boost/python/data_members.hpp:64:22: error: array type 'short [244]' is not assignable
          c.*m_which = d;
          ~~~~~~~~~~ ^

Add command-line flag to switch between C++ and C.

Generate setters for const static member variables

E.g.:

/tmp/dartpy/_ZN4dart9optimizer21GradientDescentSolverE.cpp:35:168: error: no viable overloaded '='
.add_static_property("Type", []() { return dart::optimizer::GradientDescentSolver::Type; }, [](const std::string value) { dart::optimizer::GradientDescentSolver::Type = value; })

Explicitly suppressing classes produces a YAML error

terminate called after throwing an instance of 'YAML::BadSubscript'
  what():  yaml-cpp: error at line 0, column 0: operator[] call on a scalar
Aborted (core dumped)

Suppress classes inside an uninstantiated template class

For example:

dart::dynamics::MultiDofJoint::Properties

Classes are not enumerated in the correct order

The main .cpp file must register the class_ declarations in an order that matches the inheritance hierarchy; i.e. all parent classes must be emitted before a child class.

Inner classes are not nested correctly

This is a special case of #60.

Unable to set properties on AddonWithProtectedPropertiesInSkeleton

I copied this type out of the Chimera output, but I get this error message if I try to use it in the YAML configuration file:

chimera.util.resolveDeclaration:1:184: error: expected ')'
dart::dynamics::AddonWithProtectedPropertiesInSkeleton<dart::dynamics::detail::EulerJointAddon, dart::dynamics::detail::EulerJointUniqueProperties, dart::dynamics::EulerJoint, &(void dart::dynamics::detail::JointPropertyUpdate<dart::dynamics::detail::EulerJointAddon>(dart::dynamics::detail::EulerJointAddon*)), false>;
                                                                                                                                                                                       ^

I'll break that up a bit:

dart::dynamics::AddonWithProtectedPropertiesInSkeleton<
  dart::dynamics::detail::EulerJointAddon,
  dart::dynamics::detail::EulerJointUniqueProperties,
  dart::dynamics::EulerJoint,
  &(void dart::dynamics::detail::JointPropertyUpdate<   // ERROR
    dart::dynamics::detail::EulerJointAddon>(           // ERROR
      dart::dynamics::detail::EulerJointAddon*)),       // ERROR
  false>;

I don't even think the syntax on those three lines is valid. Any idea what is going on here?

Missing argument names on constructors

The constructors we generate look like this:

.def(::boost::python::init<double, double, double, double, double, bool>()

We should be passing an args() structure to name them, just like we do for methods.

Require held_type and noncopyable for pure virtual classes

Template functions generate invalid code

typename Derived::PlainObject AdTJacFixed(const Eigen::Isometry3d& _T,
                                          const Eigen::MatrixBase<Derived>& _J)

produces this code:

boost::python::def("AdTJacFixed", static_cast<typename type-parameter-0-0::PlainObject (*)(const Eigen::Transform<double, 3, 1, 0> &, const MatrixBase<type-parameter-0-0> &)>(&dart::math::AdTJacFixed), (::boost::python::arg("_T"), ::boost::python::arg("_J")))

Generated code #include's `.cpp` files

Attempt to generate class_ wrapper for protected classes

/homes/mkoval/dartpy-ws/devel/include/dart/common/AddonManager.h:170:29: error: ‘template<class T> struct dart::common::AddonManager::type’ is protected
   template <class T> struct type { };

Attempt to generate add_static_property for function pointer type

.add_static_property("UpdateProperties", ::boost::python::make_getter(dart::dynamics::AddonWithProtectedPropertiesInSkeleton<dart::dynamics::detail::PlanarJointAddon, dart::dynamics::detail::PlanarJointUniqueProperties, dart::dynamics::PlanarJoint, &dart::dynamics::detail::JointPropertyUpdate, false>::UpdateProperties))

Impossible to inject extra code into a class

I need this to add additional functions to Chimera classes.

Attempt to add global variables for constexpr's inside template classes

/home/mkoval/storage/dartpy-ws/src/dartpy/src_generated/_ZN4dart8dynamics46AddonWithProtectedStateAndPropertiesInSkeleton8OptionalE.cpp:11:61: error: 
      'AddonWithProtectedStateAndPropertiesInSkeleton' is not a class, namespace, or enumeration
::boost::python::scope().attr("Optional") = dart::dynamics::AddonWithProtectedStateAndPropertiesInSkeleton::Optional;

Allow overrides for non-trivial default function parameters

This would have to be a straight string substitution. It's up to the user to make sure that the substitution is valid and has no side effects.

Attempt to generate code for anonymous structs

For example:

/tmp/chimera_output/_ZN4dart3gui12jitter_pointE.cpp:11:33: error: unexpected parenthesis after '::'
.def_readwrite("x", &dart::gui::(anonymous struct)::x)

Add support for a custom 'return_value_policy' on properties

This requires integration with the YAML configuration file.

Require a user-specified name for template class instantiations

This was generated with no entry in the YAML file:

::boost::python::class_<dart::common::Signal<void (std::shared_ptr<const dart::dynamics::MetaSkeleton>, const std::basic_string<char> &, const std::basic_string<char> &), signal::detail::DefaultCombiner>, ::boost::noncopyable >("Signal", boost::python::no_init)

Note that the class is named Signal, even though there are likely multiple instantiations of this class. This will cause a name conflict in Python.

Inject code into the top and bottom of BOOST_PYTHON_MODULE

This is necessary to register some custom converters, e.g. Boost.NumPy_Eigen.

Pass a namespace from the command line.

It would be useful to be able to pass a namespace from the command line instead of having to create a configuration YAML file.

SEGFAULTs on DART

chimera: /usr/lib/llvm-3.7/include/clang/AST/DeclCXX.h:592: struct DefinitionData &clang::CXXRecordDecl::data() const: Assertion `DD && "queried property of class with no definition"' failed.

Suppress copy constructors

Don't generate constructors for pure-virtual classes

Automatically flag abstract classes as noncopyable

Type lookup fails on template specializations

This requires replacing the using Foo; generated code with a typedef Foo Foo_t (preferable) or using Foo_t = Foo statement.

Generates unqualified type names for template template parameters

See my post on StackOverflow.

Generates def_readwrites for objects that are not assignable

For example:

.def_readwrite("onNameChanged", &dart::simulation::World::onNameChanged)

Mark classes with noncopyable members as noncopyable

/home/mkoval/storage/dart-ws/devel/include/dart/dynamics/detail/BodyNodePtr.h:52:14: note: copy constructor of 'MutexedWeakSkeletonPtr' is implicitly deleted because field 'mMutex' has a deleted copy constructor
  std::mutex mMutex;
             ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/mutex:128:5: note: 'mutex' has been explicitly marked deleted here
    mutex(const mutex&) = delete;
    ^

Ignores the -o flag and always puts output in the build directory

Suppress functions that take rvalue references

Generates def_readwrite on arrays

E.g.:

.def_readwrite("stuff", &dart::utils::c3d_head_t::stuff)

std::cout is not fully qualified as a default parameter

In _ZN4dart9optimizer6Solver10PropertiesE.cpp we generate:

/tmp/dartpy/_ZN4dart9optimizer6Solver10PropertiesE.cpp:11:368: error: use of undeclared identifier 'cout'; did you mean 'std::cout'?
  ...= 500, ::boost::python::arg("_iterationsPerPrint") = 0, ::boost::python::arg("_ostream") = &cout, ::boost::python::arg("_printFinalRes...
                                                                                                 ^~~~
                                                                                                 std::cout

Suppress non-public inner classes

E.g. dynamics::HierarchicalIK::Constraint

Naming

@mkoval and I agree that this repo should be renamed chimera. I mean, look at this badass thing:

Constructor arguments are not fully qualified

Chimera just generated this code:

.def(::boost::python::init<const MultiDofJoint<3>::Properties &, const PlanarJoint::UniqueProperties &>())

These constructor arguments should be fully qualified, e.g. dart::dynamics::MultiDofJoint<3>::Properties.