32.2. External Modules
Prev | Chapter 32. Platform Specific Extensions | Next

## 32.2. External Modules

### Platform Dependent: UNIX, Win32 platforms only.

### Modules on Win32

Everything described in the section will work verbatim on Win32 when using Cygwin or MinGW, except for one thing - you will need to replace the `run` extension in `lisp.run` with the Win32 executable extension `exe`.

For historical reasons, all examples appear to assume UNIX and use the `run` file type (“extension”) for the CLISP run-time. This does not mean that they will not work on Win32.

### 32.2.1. Overview

CLISP has a facility for adding external modules (written in C, for example). It is invoked through clisp-link.

A module is a piece of external code which defines extra Lisp objects, symbols and functions. A module name must consist of the characters `A`-`Z`, `a`-`z`, `_`, `0`-`9`. The module name “clisp” is reserved. Normally a module name is derived from the corresponding file name.

clisp-link needs a directory containing:

- `"modules.c"`
- `"clisp.h"`

clisp-link expects to find these files in a subdirectory `linkkit/` of the current directory. This can be overridden by the environment variable `CLISP_LINKKIT`.

clisp-link operates on CLISP linking sets and on module sets.

#### linking set

A linking set is a directory containing:

`makevars`

  some /bin/sh commands, setting the variables

  `CC` | the C compiler
  `CPPFLAGS` | flags for the C compiler, when preprocessing or compiling
  `CFLAGS` | flags for the C compiler, when compiling or linking
  `CLFLAGS` | flags for the C compiler, when linking
  `LIBS` | libraries to use when linking (either present in the linking set directory, or system-wide)
  `X_LIBS` | additional X Window System libraries to use
  `RANLIB` | the ranlib command
  `FILES` | the list of files needed when linking

`modules.h`
  the list of modules contained in this linking set
`modules.o`
  the compiled list of modules contained in this linking set
all the `FILES`
  listed in `makevars`
`lisp.run`
  the executable
`lispinit.mem`
  the memory image

To run a CLISP contained in some linking set `directory`, call

    $ directory/lisp.run -M directory/lispinit.mem

or

    $ clisp -K directory

(recommended, since it also passes `-B` to the run-time).

#### module set

A module set is a directory containing:

link.sh
  some /bin/sh commands, which prepare the directory before linking, and set the variables `NEW_FILES`, `NEW_LIBS`, `NEW_MODULES`, `TO_LOAD` and optionally `TO_PRELOAD`
and any other files
  needed by link.sh

In link.sh the module set directory is referred to as `$modulename/`.

#### Module set variables

The following variables should be defined in link.sh.

`NEW_FILES`
  the space-separated list of files that belong to the module set and will belong to every new linking set.
`NEW_LIBS`
  the space-separated list of files or C compiler switches that need to be passed to the C compiler when linking the `lisp.run` belonging to a new linking set.
`NEW_MODULES`
  the space-separated list of the module names belonging to the module set. Normally, every `#P".c"` file in the module set defines a module of its own. The module name is derived from the file name.
`TO_LOAD`
  the space-separated list of Lisp files to load before building the `lispinit.mem` belonging to a new linking set.
`TO_PRELOAD` (optional)

  the space-separated list of Lisp files to load into an intermediate `lispinit.mem` file, before building the `lispinit.mem` belonging to a new linking set. This variable is usually used to create (or unlock) the Lisp `PACKAGE`s which must be present when the new `#P".c"` files are initialized. E.g., the `FFI:DEF-CALL-IN` functions must reside in already defined packages; see Example 32.6, “Calling Lisp from C”. You can find a live example in `modules/syscalls/preload.lisp` and `modules/syscalls/link.sh.in`.

  ### Warning

  If you are unlocking a package, you must also `DELETE` it from `CUSTOM:*SYSTEM-PACKAGE-LIST*` (see Section 31.2, “Saving an Image”) here and re-add it to `CUSTOM:*SYSTEM-PACKAGE-LIST*` in one of the `TO_LOAD` files. See, e.g., `modules/i18n/preload.lisp` and `modules/i18n/link.sh.in`.

#### Creating linking sets

The command

    $ clisp-link create-module-set module file1.c ...

creates a module set in `module` directory which refers (via symbolic links) to `file1.c` etc. The files are expected to be modules of their own.

The command

    $ clisp-link add-module-set module source destination

combines a linking set in directory `source` and a module in directory `module` to a new linking set, in the directory `destination` which is newly created.

The command

    $ clisp-link run source module ...

runs the linking set in directory `source`, with the module in directory `module` loaded. More than one module can be specified. If CLISP has been built with the configuration option `--with-dynamic-modules`, the loading will be performed dynamically. Otherwise - this is much slower - a temporary linking set will be created and deleted afterwards.

### 32.2.2. Module initialization

Each module has two initialization functions:

void `module__`name`__init_function_1` (struct module_t* module)

  called only once when CLISP discovers while loading a memory image that there is a module present in the executable (`lisp.run`) which was not present at the time the image was saved. It can be used to create Lisp objects, e.g. functions or keywords, and is indeed used for that purpose by modprep.

  You do not have to define this function yourself; modprep and “FFI” will do that for you.

  If you use “FFI”, `(`FFI:C-LINES` :init-once ...)` will add code to this function.

  ### Warning

  The `PACKAGE`s must already exist and be unlocked, cf. `TO_PRELOAD`.

  ### Warning

  If you are using modprep and defining your own “init-once” function, it must call the `module__`name`__init_function_1__modprep` function!

void `module__`name`__init_function_2` (struct module_t* module)

  called every time CLISP starts. It can be used to bind names to foreign addresses, since the address will be different in each invocation of CLISP, and is indeed used for that purpose by “FFI” (e.g., by `FFI:DEF-CALL-OUT`). It can also be used to set parameters of the libraries to which the module interfaces, e.g., the `pcre` module sets `pcre_malloc` and `pcre_free`.

  You do not have to define this function yourself; modprep and “FFI” will do that for you.

  If you use “FFI”, `(`FFI:C-LINES` :init-always ...)` will add code to this function.

`name` is the module name.

See also Section 31.1, “Customizing CLISP Process Initialization and Termination”.

### 32.2.3. Module finalization

Each module has a finalization function

void `module__`name`__fini_function` (struct module_t* module)

  called before exiting CLISP.

  You do not have to define this function yourself; modprep and “FFI” will do that for you.

  If you use “FFI”, `(`FFI:C-LINES` :fini ...)` will add code to this function.

`name` is the module name.

See also Section 31.1, “Customizing CLISP Process Initialization and Termination”.

### 32.2.4. Function `EXT:MODULE-INFO`

Function `(`EXT:MODULE-INFO` `&OPTIONAL` `name` `verbose`)` allows one to inquire about what modules are available in the currently running image. When called without arguments, it returns the list of module names, starting with “clisp”. When `name` is supplied and names a module, 3 values are returned - `name`, `subr-count`, `object-count`. When `verbose` is non-`NIL`, the full list of module lisp function names written in C (Subrs) and the full list of internal lisp objects available in C code are additionally returned for the total of 5 values.

When `name` is `:FFI`, returns the list of shared libraries opened using `:LIBRARY`. When `verbose` is non-`NIL`, return the association list of DLL names and all foreign objects associated with it.

### 32.2.5. Function `SYS::DYNLOAD-MODULES`

#### Platform Dependent: Only when compiled with configure flag `--with-dynamic-modules`.

### Note

Dynamic loading does not work on all operating systems (`dlopen` or equivalent is required).

### Note

`--with-dynamic-modules` precludes some optimizations which are enabled by default.

Function `(`SYS::DYNLOAD-MODULES` `filename` ({`name`}+))` loads a shared object file or library containing a number of named external CLISP modules.

### Note

This facility cannot be used to access arbitrary shared libraries. To do that, use the `:LIBRARY` argument to `FFI:DEF-CALL-OUT` and `FFI:DEF-C-VAR` instead.

External modules for CLISP are shared objects (dynamic libraries) that contain the `module__`name`__subr_tab` variable, among others. This serves to register external functions which operate on Lisp-level structures with CLISP.

To use `dlopen` with modules, you should add `-fPIC` to the module's compilation options. Something like cc -shared -o `name`.so `name`.o may be needed to produce the shared object file.

### 32.2.6. Example

To link in the “FFI” bindings for the GNU/Linux operating system, the following steps are needed. (Step 1 and step 2 need not be executed in this order.)

1. Create a new module set

       $ clisp-link create-module-set linux /somewhere/bindings/linux.c

2. Modify the newly created `linux/link.sh`

   1. add `-lm` to the libraries

      replace

          NEW_LIBS="$file_list"

      with

          NEW_LIBS="$file_list -lm"

   2. load `linux.fas` before saving the memory image

      replace

          TO_LOAD=''

      with

          TO_LOAD='/somewhere/bindings/linux.fas'

3. Compile `linux.lisp`, creating `linux.c`

       $ clisp -c /somewhere/bindings/linux.lisp

4. Create a new linking set

       $ clisp-link add-module-set linux base base+linux

5. Run and try it

       $ base+linux/lisp.run -M base+linux/lispinit.mem -x '(linux:stat "/tmp")'

### 32.2.7. Module tools

There are some tools to facilitate easy module writing.

#### 32.2.7.1. Modprep

If your module is written in C, you can pre-process your sources with modprep in the CLISP distribution and define lisp functions with the `DEFUN` macro:

    DEFUN(MY-PACKAGE:MY-FUNCTION-NAME, arg1 arg2 &KEY FOO BAR) {
      if (!boundp(STACK_0)) STACK_0 = fixnum(0); /* BAR */
      if (!boundp(STACK_1)) STACK_1 = fixnum(1); /* FOO */
      pushSTACK(`MY-PACKAGE::SOME-SYMBOL`); /* create a symbol in the package */
      pushSTACK(`#(:THIS :IS :A :VECTOR)`); /* some vector, created once */
      pushSTACK(``MY-PACKAGE::MY-FUNCTION-NAME``); /* double `` means FUNCTION */
      VALUES1(listof(7)); /* cons up a new list and clean up the STACK */
    }

Then `(MY-PACKAGE:MY-FUNCTION-NAME 'A 12 :FOO T)` will return (A 12 T 0 MY-PACKAGE::SOME-SYMBOL #(:THIS :IS :A :VECTOR) #<ADD-ON-SYSTEM-FUNCTION MY-PACKAGE:MY-FUNCTION-NAME>) (assuming you `EXPORT`ed `MY-FUNCTION-NAME` from “MY-PACKAGE”).

Another useful macros are:

`DEFVAR`
  create a GC-visible private object
`DEFFLAGSET`
  define a C function which will remove several flag arguments from the `STACK` and return the combined flag value
`DEFCHECKER`
  define a map from cpp constants to lisp symbols and functions that map between them, checking that the argument is appropriate

See `modules/syscalls/calls.c` and other included modules for more examples and file modprep for full documentation.

### Warning

If you manipulate Lisp objects, you need to watch out for GC-safety.

#### 32.2.7.2. clisp.h

If your module is written in C, you will probably want to `#include "clisp.h"` to access CLISP objects. You will certainly need to read `"clisp.h"` and some code in included modules, but here are some important hints that you will need to keep in mind:

- Lisp objects have type object.
- Variables of this type are invalidated by lisp memory allocation (`allocate_*()` functions) - but not C allocations (`malloc` et al) - and must be saved on the `STACK` using cpp macros `pushSTACK()`, `popSTACK()` and `skipSTACK()`.
- Access object slots using the appropriate `TheFoo()` macro, e.g., `TheCons(my_cons)->Car`, but first check the type with `consp()`.
- Arguments are passed on the `STACK`, as illustrated in the above example.
- Wrap your system calls in `begin_system_call()`/`end_system_call()` pairs. These macros, defined in `"clisp.h"`, save and restore registers used by CLISP which could be clobbered by a system call.

#### 32.2.7.3. Exporting

If your module uses “FFI” to interface to a C library, you might want to make your module package case-sensitive and use `exporting.lisp` in the CLISP distribution to make “FFI” forms and `DEFUN`, `DEFMACRO` at al export the symbols they define. See `modules/netica/`, `modules/matlab/` and `modules/bindings/` for examples.

### 32.2.8. Trade-offs: “FFI” vs. C modules

When deciding how to write a module: whether to use “FFI” or to stick with C and modprep, one has to take into account several issues:

Speed: C wins

  “FFI” has a noticeable overhead: compare `RAWSOCK:HTONS` (defined in `modules/rawsock/rawsock.c`) with

      (FFI:DEF-CALL-OUT htons (:name "htons") (:library :default)
        (:arguments (s ffi:short)) (:return-type ffi:short) (:language :stdc))

  and observe that `RAWSOCK:HTONS` is almost 3 times as fast (this really does compare the “FFI” overhead to the normal lisp function call because `htons` is computationally trivial). This difference will matter only if you call a simple function very many times, in which case it would make sense to put the loop itself into C.

Portability: C wins

  First of all, “FFI” is not as widely ported as CLISP, so it is possible that you will face a platform where CLISP runs but “FFI” is not present.

  Second, it is much easier to handle portability in C: observe the alternative implementations of `htonl` et al in `modules/rawsock/rawsock.c`.

  Third, certain C structures have different layout on different platforms, and functions may take 64-bit arguments on some platforms and 32-bit arguments on others; so the “FFI” code has to track those differences, while C will mostly take care of these things for you.

Code size: “FFI” wins
  You need to type much fewer characters with “FFI”, and, if you use the `:LIBRARY` argument to `FFI:DEF-CALL-OUT` and `FFI:DEF-C-VAR`, you do not need to leave your CLISP session to try out your code. This is a huge advantage for rapid prototyping.
UI: C wins
  To produce a nice lispy UI (using `&OPTIONAL` and `&KEY`word arguments etc), you will need to write wrappers to your `FFI:FOREIGN-FUNCTION`s, while in C you can do that directly. The same goes for “polymorphism”: accepting different argument types (like, e.g., `POSIX:RESOLVE-HOST-IPADDR` does) would require a lisp wrapper for `FFI:FOREIGN-FUNCTION`s.
Learning curve: unclear

  If you are comfortable with C, you might find the CLISP C module facilities (e.g., modprep) very easy to use.

  CLISP “FFI”, on the other hand, is quite high-level, so, if you are more comfortable with high-level languages, you might find it easier to write “FFI” forms than C code.

Safety: unclear
  One can get a segfault either way: if your `FFI:DEF-CALL-OUT` form does not describe the function's expectations with respect to the arguments and return values (including `ALLOCATION`), you will probably learn that the hard way. If the module is written in C, all the opportunities to shoot oneself in the foot (and other body parts) are wide open (although well known to most C users). However, with C, one has to watch for GC-safety too.

### Note

The granularity of the choice is per function: the same module can use both modprep and “FFI”.

### Note

It is not a good idea to have both `foo.lisp` and `foo.c` files in a module, because if you ever add an “FFI” form to the former, `COMPILE-FILE` will overwrite the latter.

### 32.2.9. Modules included in the source distribution

A few modules come with the source distribution of CLISP (but are not necessarily built in a particular binary distribution).

To use modules, read `unix/INSTALL` and build CLISP in directory `build-dir` with, e.g.,

    $ ./configure --with-module=pcre --with-module=clx/new-clx --cbc build-dir

then run it with

    $ ./build-dir/clisp -K full

This will create a base linking set with modules `i18n`, `regexp` and `syscalls` (and maybe `readline`); and a full linking set with modules `clx/new-clx` and `pcre` in addition to the 3 (or 4) base modules.

Here we list the included modules by their general theme. See Chapter 33, Extensions Implemented as Modules for individual module documentation.

#### 32.2.9.1. Base Modules

The default build process includes the following modules in both base and full linking sets:

`i18n`
  Internationalization of User Programs.
`regexp`
  The POSIX Regular Expressions matching, compiling, executing.
`syscalls`
  Use some system calls in a platform-independent way.
`readline` (only when both GNU readline and “FFI” are available)
  When GNU readline and “FFI” are available, some advanced readline and history features are exported using this module.

The composition of the full linking set depends on the platform and on the vendor preferences.

#### 32.2.9.2. Database, Directory et al

`gdbm`
  Interface to GNU DataBase Manager by Masayuki Onjo.
`berkeley-db`
  Berkeley DB from Sleepycat Software interface.
`dirkey`
  Directory Access (LDAP, Win32 registry etc).
`postgresql`
  Access PostgreSQL from CLISP.
`oracle`
  Access Oracle from CLISP; by John Hinsdale.

#### 32.2.9.3. Mathematics, Data Mining et al

`libsvm`
  Build Support Vector Machine models using LibSVM inside CLISP.
`pari`
  Interface to the computer algebra system PARI.
`matlab`
  Do matrix computations via MATLAB.
`netica`
  Work with Bayesian belief networks and influence diagrams using Netica C API.

#### 32.2.9.4. Matching, File Processing et al

`pcre`
  The Perl Compatible Regular Expressions matching, compiling, executing.
`wildcard`
  Shell (/bin/sh) globbing (Pathname Matching).
`zlib`
  Compress `VECTOR`s using ZLIB.

#### 32.2.9.5. Networking

`rawsock`
  Raw socket access.
`fastcgi`
  Access FastCGI from CLISP; by John Hinsdale.

#### 32.2.9.6. Graphics

CLX

  Call Xlib functions from CLISP. Two implementations are supplied:

  `clx/mit-clx`, from MIT ftp://ftp.x.org/R5contrib/CLX.R5.02.tar.Z
    the standard implementation
  `clx/new-clx`, by Gilbert Baumann

    faster, with additional features, but not quite complete yet. Please try it first and use `clx/mit-clx` only if `clx/new-clx` does not work for you. `clx/new-clx` comes with several demos, please try them using

        $ clisp -K full -i modules/clx/new-clx/demos/clx-demos.lisp -x '(clx-demos:run-all-demos)'

    and follow the intructions.

  This functionality is documented in the manual http://www.stud.uni-karlsruhe.de/~unk6/clxman/, also available in the CLISP source distribution as `modules/clx/clx-manual.tar.gz`.

`gtk2`
  Use GTK+ and Glade to create GUI by James Bailey.

#### 32.2.9.7. Bindings

Call the operating system functions from CLISP. The following platforms are supported:

`bindings/glibc`
  Linux/GNU libc
`bindings/win32`
  Win32

#### 32.2.9.8. Toys and Games

`queens`
  Compute the number of solutions to the `n`-queens problem on a `n`×`n` chessboard (a toy example for the users to explore the CLISP module system).
`modules/clx/new-clx/demos/sokoban.lisp`
  a demo which comes with `clx/new-clx`.

| These notes document CLISP version 2.45 | Last modified: 2008-05-14

Enter your search termsSubmit search form
Webclisp.cons.orgclisp.podval.orgwww.lisp.org

Prev | Up | Next
32.1. Random Screen Access | Home | 32.3. The Foreign Function Call Facility