:claw honing

Free time too play with CL is quite a luxury item for me as of late, but finally! - I’ve got a whole week of me-time and I’m planning to pour all of it into :claw - the precious jewel at the heart of most of my projects.

:claw is a library for automatically creating Common Lisp bindings to foreign libraries that provide C interface. :claw already works well, but having access only to libraries with C-compatible interface is quite limiting. I’m a game developer both at work and at home and a lot of gamedev libraries are either C++ only and don’t export any C interface at all or such interface is out-of-date or heavily underdocumented. Unfortunately, I don’t have enough man-hours to reimplement all of those in pure Common Lisp myself and bindings seem like a huge time saver. That’s right, I’m working on bringing C++ support into :claw.

In this post, I’ll try to summarize how I envision this support being implemented in :claw. If you think something can be done better, faster or maybe I’m plain wrong somewhere - I appreciate if you drop a comment with your ideas here or via email/IRC.

Origins

:claw started as a fork of :cl-autowrap to create cleaner bindings (exclude rule by default) with less overhead (no wrapper structs, direct pointers, no autoconversion) and autogenerated C wrapper instead of libffi for calling struct-by-value functions (passing and returning strucs as values). Recently, I removed autowrap’s sffi and refactored :claw to also generate pure :cffi constructs.

But once I decided to add C++ support things got pretty ugly. c2ffi C++ handling is pretty basic and many things aren’t implemented. Adding them means going through llvm/clang C++ internals which are not stable across releases (and, to be fair, are not guaranteed to in the first place) and writing a bunch of C++ to bring required features into c2ffi. I tried. I didn’t feel so much pain programming in quite a while. c2ffi is also a pain to build - you need specific llvm/clang version and its development libraries, which sometimes are missing from binary distributions, so you need to compile toolchain yourself. I quickly abandoned that idea. I decided to go with libclang.

libclang is guaranteed to have a stable C interface and very simple to build against - likely you already have it somewhere in your system if you are a programmer. So I built a little wrapper around it to use it from Common Lisp - libresect. libclang uses structs as values in API, but :cffi doesn’t like passing struct as values into foreign code w/o libffi which is another can of worms I once tried to open - each OS has it’s own flavor of it - none is tasty. It wasn’t a happy experience as you might have guessed, so I just went with writing libresect that can be trivially called from Common Lisp. It was really a breeze compared with going through c2ffi/llvm/clang C++ codebase.

At this moment, it is safe to say :claw C++ branch is a complete rewrite from cl-autowrap. Most work is happening in :claw’s cxx branch where code is flying around unpredictably while I explore the possibilities.

Architecture

Programming at home, I like to explore - I overengineer for fun.

I want :claw to be a bit less chaotic and much simpler to understand than cl-autowrap. I’ve split :claw into several subsystems to better separate functionality: library specification model, C/C++ header parser, bindings generator and an umbrella facade over these to make :claw simple to use.

Library specification model subsystem (:claw/spec) is just a bunch of classes and functions to represent and handle C/C++ foreign entities (functions, structs, classes, variables, etc) in Common Lisp - to generate bindings from, basically.

C/C++ header parser subystem (:claw/resect) uses libresect as a backend and convert headers into library specification model.

:claw has two bindings generators: :claw/cffi for C bindings that generate pure :cffi code, and :claw/iffi to generate tricky C++ bindings that ofc still uses :cffi under the hood, but exports different API to access C++ native objects (unfortunately or not, :cffi doesn’t include any API for handling C++).

:claw/wrapper subsystem is an umbrella interface that combines all these subsystems into a single whole for the smooth user experience.

How to C++ with :claw

Here is how I’m planning to deal with C++. I want to use :claw ability to generate C wrappers (tiny and portable C libraries that reexport some native aspect of foreign code that can’t be directly used via :cffi, e.g. passing structs by value to functions), but apply it to C++. E.g. class constructor will be generated as a C function that accepts certain arguments and return a pointer to an object of a class.

Two immediate problems arise: templates and function overloading.

I’m not planning to deal with uninstantiated templates in any way. In :claw there would be a way for a user to instantiate a template somehow before library is wrapped. Maybe in a utility C++ header. There’s no templates in the compiled code - a function or a class instantiated from a template can be accessed like anything else. So no, you won’t be able to metaprogram with C++ templates in Common Lisp using :claw or instantiate templates on the fly in runtime. But acessing instantiated templates shouldn’t be a problem.

Function overloading is tricky. C doesn’t have anything for that, but we don’t need to improvise too much here. libclang provides mangled function names for C++ entities that we can use to name our C wrapper functions and they won’t overlap by design. Now the other problem is that Common Lisp doesn’t have function overloading either. That’s why :claw needs iffi - intricate foreign function interface - a simple way to call overloaded functions. The plan so far is to pass argument type alongside argument value during a call to C++ function or method:

;; syntax is a subject to change
(iffi:intricate-funcall 'operator+ 'awesome-class this :int that)
(iffi:intricate-funcall 'operator+ 'awesome-class this :float that)

The general idea is that iffi will keep a map between function signature (name + arg types) and actual mangled C wrapper counterpart and would inline call to this certain C function at call site. Types most likely will be canonicalized, so iffi might also keep type hierarchy to canonicalize types for the user.

Plans

Next week goal is to bring Filament into Common Lisp via :claw. It doesn’t need to be a full bindings, but with it I should be able to bring something onto a screen using Filament from Common Lisp. For this I need to:

• Rework spec model to incorporate required C++ features
• Implement C wrapper generator for C++ code
• Implement iffi foreign function interface
• Integrate C++ options for resect and iffi into :claw interface
• Autogenerate bidings to Filament and write a test system using those