Why zero-copy is missing in CL-ZMQ

ZeroMQ can utilize zero-copy technique, which may save a lot of cpu ticks when using large messages. Unfortunately, it's not possible to get this feature to work with supported Common Lisp implementations due to a couple of reasons:

• ZeroMQ uses threads for asynchronous I/O operations on messages. When it finishes all the work with message's data, it calls back Lisp to let it know that data can be safely ripped out.

Unfortunately, not all Lisps can distinguish and handle callback from non-lisp thread. For example, Clozure CL feels good here, but SBCL crashes and LispWorks just hangs.

• Garbage collector may move lisp objects, which are currently used in I/O threads. This leads to image corruption or segmentation fault.

Widely used with-pinned-object technique is not working here, because ZeroMQ operates asynchronously and pinned body can't protect data from GC when it needs to. It's possible to turn off garbage collecting completely or design system in such way, that it won't do gc'ing when it shouldn't. But that's a hard way.

Other possibility is to allocate objects in static area, where garbage collector doesn't move live objects. Some Lisps support this feature, for example AllegroCL, LispWorks and GCL.

If both features (callback from non-lisp thread, objects in static area) are supported by certain Lisp implementation, it's possible then to use zero-copy in CL-ZMQ. Unfortunately, all four Lisps that I use for development and testing (SBCL, Clozure, CLISP and LispWorks) miss this combination.

CL-ZMQ and LispWorks 6

Since yesterday I'm happy user of commercial version of LW.

LW complains about unshadowed symbol `identity'. Yeah, that's strange it wasn't yet catched by 3 other lisps I tested with (sbcl, clisp, ccl). Other minor problem was with `trivial-garbage' library: it uses `hcl:mark-and-sweep', but this function exists only on 32-bit LWs. 64-bit uses `system:marking-gc'. Also `system::hash-table-weak-kind' has a bit different name: `system::hash-table-%weak-kind'.

Otherwise everything works good. LW is officially supported by CL-ZMQ now :)