:claw honing12 Jun 2020
Free time too play with CL is quite a luxury item for me as of late, but
finally! - I’ve got a whole week of me-time and I’m planning to pour all of it
:claw - the precious jewel at the
heart of most of my projects.
:claw is a library for automatically creating Common Lisp bindings to foreign
libraries that provide C interface.
:claw already works well, but having
access only to libraries with C-compatible interface is quite limiting. I’m a
game developer both at work and at home and a lot of gamedev libraries are
either C++ only and don’t export any C interface at all or such interface is
out-of-date or heavily underdocumented. Unfortunately, I don’t have enough
man-hours to reimplement all of those in pure Common Lisp myself and bindings
seem like a huge time saver. That’s right, I’m working on bringing C++ support
In this post, I’ll try to summarize how I envision this support being
:claw. If you think something can be done better, faster or
maybe I’m plain wrong somewhere - I appreciate if you drop a comment with your
ideas here or via email/IRC.
:claw started as a fork of
:cl-autowrap to create cleaner bindings
(exclude rule by default) with less overhead (no wrapper structs, direct
pointers, no autoconversion) and autogenerated C wrapper instead of
calling struct-by-value functions (passing and returning strucs as values).
Recently, I removed autowrap’s
sffi and refactored
:claw to also generate
But once I decided to add
C++ support things got pretty
c2ffi C++ handling is pretty basic and
many things aren’t implemented. Adding them means going through
C++ internals which are not stable across releases (and, to be fair, are not
guaranteed to in the first place) and writing a bunch of
C++ to bring required
c2ffi. I tried. I didn’t feel so much pain programming in quite
c2ffi is also a pain to build - you need specific
version and its development libraries, which sometimes are missing from binary
distributions, so you need to compile toolchain yourself. I quickly abandoned
that idea. I decided to go with
libclang is guaranteed to have a stable C interface and very simple to build
against - likely you already have it somewhere in your system if you are a
programmer. So I built a little wrapper around it to use it from Common Lisp -
libclang uses structs as
values in API, but
:cffi doesn’t like passing struct as values into foreign
libffi which is another can of worms I once tried to open - each OS
has it’s own flavor of it - none is tasty. It wasn’t a happy experience as you
might have guessed, so I just went with writing
libresect that can be
trivially called from Common Lisp. It was really a breeze compared with going
c2ffi/llvm/clang C++ codebase.
At this moment, it is safe to say
C++ branch is a complete rewrite
cl-autowrap. Most work is happening in
cxx branch where code
is flying around unpredictably while I explore the possibilities.
Programming at home, I like to explore - I overengineer for fun.
:claw to be a bit less chaotic and much simpler to understand than
cl-autowrap. I’ve split
:claw into several subsystems to better separate
functionality: library specification model, C/C++ header parser, bindings
generator and an umbrella facade over these to make
:claw simple to use.
Library specification model subsystem (
:claw/spec) is just a bunch of classes
and functions to represent and handle C/C++ foreign entities (functions,
structs, classes, variables, etc) in Common Lisp - to generate bindings from,
C/C++ header parser subystem (
libresect as a backend and
convert headers into library specification model.
:claw has two bindings generators:
:claw/cffi for C bindings that generate
:cffi code, and
:claw/iffi to generate tricky C++ bindings that ofc
:cffi under the hood, but exports different API to access C++
native objects (unfortunately or not,
:cffi doesn’t include any API for
:claw/wrapper subsystem is an umbrella interface that combines all these
subsystems into a single whole for the smooth user experience.
How to C++ with :claw
Here is how I’m planning to deal with C++. I want to use
:claw ability to
generate C wrappers (tiny and portable C libraries that reexport some native
aspect of foreign code that can’t be directly used via
:cffi, e.g. passing
structs by value to functions), but apply it to C++. E.g. class constructor will
be generated as a C function that accepts certain arguments and return a pointer
to an object of a class.
Two immediate problems arise: templates and function overloading.
I’m not planning to deal with uninstantiated templates in any way. In
there would be a way for a user to instantiate a template somehow before library
is wrapped. Maybe in a utility C++ header. There’s no templates in the compiled
code - a function or a class instantiated from a template can be accessed like
anything else. So no, you won’t be able to metaprogram with C++ templates in
Common Lisp using
:claw or instantiate templates on the fly in runtime. But
acessing instantiated templates shouldn’t be a problem.
Function overloading is tricky. C doesn’t have anything for that, but we don’t
need to improvise too much here.
libclang provides mangled function names for
C++ entities that we can use to name our C wrapper functions and they won’t
overlap by design. Now the other problem is that Common Lisp doesn’t have
function overloading either. That’s why
iffi - intricate foreign
function interface - a simple way to call overloaded functions. The plan so far
is to pass argument type alongside argument value during a call to C++ function
;; syntax is a subject to change (iffi:intricate-funcall 'operator+ 'awesome-class this :int that) (iffi:intricate-funcall 'operator+ 'awesome-class this :float that)
The general idea is that
iffi will keep a map between function signature
(name + arg types) and actual mangled C wrapper counterpart and would inline
call to this certain C function at call site. Types most likely will be
iffi might also keep type hierarchy to canonicalize types
for the user.
Next week goal is to bring Filament into
Common Lisp via
:claw. It doesn’t need to be a full bindings, but with it I
should be able to bring something onto a screen using
Filament from Common
Lisp. For this I need to:
- Rework spec model to incorporate required C++ features
- Implement C wrapper generator for C++ code
iffiforeign function interface
- Integrate C++ options for
- Autogenerate bidings to
Filamentand write a test system using those