sdilts / cl-bindgen

A command line tool and library for generating FFI wrappers for Common Lisp

cl-bindgen

A command line tool and library for creating Common Lisp language bindings from C header files.

Features:

• Generates CFFI bindings for function declarations, enums, variables, unions, and structures.
• Handles nested and anonymous structures, unions, and enums.
• Warns when it cannot produce a correct binding.
• Documentation comments from the C source files are lispified and included with the generated bindings when available.
• Provides a powerful way to customize how names are translated into lisp symbols.

Installation

cl-bindgen requires libclang, which is not installed with the other Python dependencies and not available on PyPi. It is recommended to install it first before installing cl-bindgen. Use your distribution's package manager to install it.

Once libclang is installed, you can then install cl-bindgen from source or from PyPI.

From PyPI:

pip install cl-bindgen

From source:

git clone --depth=1 https://github.com/sdilts/cl-bindgen
cd cl-bindgen
pip install --user .

Processing individual files

To process individual files, use the f command and specify one or more files to process. By default, output will be printed to stdout, but the output file can be specified with the -o option. To see a full list of options, run cl-bindgen f -h.

# Process test.h and print the results to stdout:
cl-bindgen f test.h
# Process the files test1.h, test2.h, and place the output in output.lisp:
cl-bindgen f -o output.lisp test1.h test2.h

Batch file processing

cl-bindgen can use a yaml file to process many header files with a single invocation. Use the b command to specify one or more batch files to process:

cl-bindgen b my_library.yaml

Batch file format

Batch files use the YAML format. Multiple documents can be contained in each input file.

Required Fields:

• output : where to place the generated code
• files : a list of files to process

Optional Fields:

• package : The name of the Common Lisp package of the generated file
• arguments : Arguments to pass to clang
• force : Ignore errors while parsing. Valid values are True or False
• pkg-config: A list of package names needed by the library. Adds the flags needed to compile the given header files as told by pkg-config --cflags
• pointer_expansion (experimental): Used to provide either a regex or a list of pointer types to expand or not expand in the output.

To see example batch files, look in the examples directory.

Handling Include Directories and Clang Arguments

If you need to specify additional command line arguments to the clang processor, you can use the -a option, and list any clang arguments after.

cl-bindgen b batch_file.yaml -a -I include_dir1 -I include_dir2
# Use -- to stop collecting clang arguments:
# Note that instead of directly calling pkg-config when using a batch
# file, you can use the pkg-config option instead.
cl-bindgen f -a `pkg-config --cflags mylibrary` -- header.h

If a header file isn't found while processing the input files, cl-bindgen will halt and produce no output. This is to avoid producing incorrect bindings: while bindings can still be produced when header files are missing, they are likely to be incorrect. To ignore missing header files and other fatal errors, the -f flag can be used:

cl-bindgen b -f batch_file.yaml
cl-bindgen f -f header.c

Customizing the behavior of cl-bindgen

cl-bindgen attempts to provide a reasonable interface that is usable in most cases. However, if you need to customize how C names are converted into lisp names or embed cl-bindgen into another application, cl-bindgen is available as a library.

The cl_bindgen package is broken up into modules: the processfile, mangler, util and macro_util modules. The processfile module provides the functions to generate the lisp bindings, the mangler module provides functions to convert C names into lisp names, and the util module provides functions to use batch files and cl-bingen's command line interface.

The processfile Module

This module exports two functions: process_file and process_files, which work on a single header file or many, respectively. Both functions take two arguments: the file(s) to be processed and an ProcessOptions object.

The ProcessOptionsclass is the way to specify how the processing functions generate their output. It has the following fields:

• typedef_mangers, enum_manglers, type_manglers, name_manglers and constant_manglers : See the mangler module section for what these do.
• output : The path of the file where the output is placed. ":stdout" or ":stderr" can be specified to use standard out or standard error.
• package : If not None, this specifies the package the the generated output should be placed in.
• arguments : The command line arguments that should be given to the clang processor.
• force : If true, then ignore errors while parsing the input files.
• macro_detector: The macro detctor function used to detect header macros
• expand_pointer_p: A function that takes a typename and returns whether or not pointers of this type should be fully expanded or left as :pointer.

The mangler Module

cl-bindgen uses a set of classes called manglers to translate C names so that they follow lisp naming conventions. Each mangler class provides one or more transformations to a symbol. For example, the UnderscoreMangler class converts underscores (_) into dashes (-). A series of manglers are applied to each C name to make it follow lisp naming conventions.

To maximize customization, a list of manglers is associated with each type of name that can be converted. enums, variable names, typedefs, constants, and record types all use a different set of manglers.

Built-in manglers:

• UnderscoreMangler : Converts underscores to dashes.
• ConstantMangler : Converts a string to follow Common Lisp's constant style recomendation.
• KeywordMangler : Adds a : to the beginning of a string to make it a symbol. Doesn't perform any action if the string has a package prefix.
• RegexSubMangler : Substitutes the substring matched by a regex with the given string.

Mangler Interface

Mangler classes follow a simple interface:

• can_mangle(string): returns true if the mangler can perform its operations on the given string
• mangle(string): returns a string with the desired transformations applied.

The util Module

The util module provides two functions: process_batch_file and dispatch_from_arguments.

• process_batch_file(batch_file, options) : Processes the given batch file using options as the default options.
• dispatch_from_arguments(arguments, options) : Uses the provided command line arguments to perform the actions of cl-bindgen using options as the default options.

The macro_util Module

This module provides the macro_matches_file_path function that is used by default to check if a macro is a header guard, and the convert_literal_token that converts literal tokens into CL literals.

The macro_matches_file_path is a macro detector function. Macro detector functions are used to determine if a C macro is a header guard. They take two arguments: the location of the file and the name of the file as a string.

Examples

The best example of how to use cl-bindgen as a library is to look at its main function found in cl_bindgen/__main__.py. In it, cl-bindgen's default options are set, then passed to dispatch_from_arguments to run the utility.

cl-bindgen's Issues

Use `clang -print-resource-dir` to derive the system include directory

clang -print-resource-dir prints out the resource directory of clang, which contains the include directory needed to find standard header files. Use this command to derive this at run-time instead of making the user do it themselves.

Quotes in docstrings are not escapted

Thanks for this awesome library.

I discovered a minor inconvenience: when parsing a file like cuda.h quotes are not escaped in docstrings. So forms like the following are produced:

(cffi:defcfun "custreamgetcaptureinfo" CUresult
  "\brief Query capture status of a stream
 
  Query the capture status of a stream and and get an id for
  the capture sequence, which is unique over the lifetime of the process.
 
  If called on ::CU_STREAM_LEGACY (the "null stream") while a stream not created
  with ::CU_STREAM_NON_BLOCKING is capturing, returns ::CUDA_ERROR_STREAM_CAPTURE_IMPLICIT.
 
  A valid id is returned only if both of the following are true:
  - the call returns CUDA_SUCCESS
  - captureStatus is set to ::CU_STREAM_CAPTURE_STATUS_ACTIVE
 
  \return
  ::CUDA_SUCCESS,
  ::CUDA_ERROR_STREAM_CAPTURE_IMPLICIT
  \notefnerr
 
  \sa
  ::cuStreamBeginCapture,
  ::cuStreamIsCapturing"
  (hstream CUstream)
  (capturestatus (:pointer CUstreamCaptureStatus))
  (id (:pointer cuuint64-t)))

The problem is "null stream" where the quotes around it are not escaped.

Seems to have trouble with stdint.h types

Here's a simple header:

#include <stdint.h>

void somefunc(int8_t x);

cl-bindgen produces an error while processing it:

WARNING: errors occured while parsing somec.c
This may cause bindings to be generated incorrectly.
somec.c:3:15: error: unknown type name 'int8_t'

;; next section imported from file somec.c

(cffi:defcfun "somefunc" :void
  (x :int))

I didn't do any digging as to why it doesn't work, but it seems like c2ffi can correctly process it. Any tips?

Arrays in structs are not converted properly

For the struct

struct s {
    char[4] array;
};

The following is being generated;

(cffi:destruct s
  (array (:pointer :char :count 4)))

It should be generating:

(cffi:destruct s
  (array :char :count 4))

header guards?

Hi there, I'm just trying out your library, I wonder if this is expected behaviour with respect to the include guard at the top of the file? Many thanks!

(venv) anticrisis@Au14-1:/tmp/test$ cat cpp.h
#ifndef CPP_H
#define CPP_H

#ifdef __cplusplus
extern "C" {
#endif

  int hello();

#ifdef __cplusplus
}
#endif
#endif
(venv) anticrisis@Au14-1:/tmp/test$ cl-bindgen f cpp.h
Found macro +cpp-h+ definition in: cpp.h:2:9

;; next section imported from file cpp.h

#| MACRO_DEFINITION
(defconstant +cpp-h+ ACTUAL_VALUE_HERE)
|#

(cffi:defcfun "hello" :int)
(venv) anticrisis@Au14-1:/tmp/test$

u_managler

cl-bindgen/cl_bindgen/__main__.py

Line 8 in dc19add

u_managler = mangler.UnderscoreMangler()

Is there any reason why this is named managler and not mangler?

Nested anonymous records cause cl-bindgen to crash

Structs like

struct foo {
    enum type flag;
    union {
        int a;
        double b;
    };
};

Cause an assertion error originating here:

cl-bindgen/cl_bindgen/processfile.py

Lines 210 to 213 in 920df4c

	if field.is_anonymous():
	assert(field.type.kind == clang.TypeKind.ELABORATED)
	inner_name = name + '-' + field_name
	actual_elaborated_type = _determine_elaborated_type(field.type)

Here's what the C spec says on nested unions:
https://en.cppreference.com/w/c/language/union

Add ability to not fully expand pointer types in structure and function declarations

cl-bindgen always expands pointer types to the the type's full definition, i.e.

struct foo {
    struct other_thing *foo
};

always gets expanded to:

(defcstruct foo
  (foo (:pointer (:struct other-thing))))

However, this can be problematic if the other_thing type doesn't yet have (or won't ever) bindings generated for it. There should be a way to define a regex or some other matching function to tell cl-bindgen that some pointer types should not be expanded. The desired output would then be something like:

(defcstruct foo
    (foo :pointer #| (:struct other-thing) |#))