Re: Foreign Function Interface

[Gary Byers <gb@cambridge.apple.com>]

> We are trying to use the Foreign Function Interface (FFI), together with MPW
> PASCAL, to simulate the [Not In ROM] system tools (e.g., the RAM Serial
> Driver).  

If I remember correctly, the "RAM serial Driver" has been in ROM since the
introduction of the Mac Plus; the Lisp doesn't run on earlier ROM versions.
I -think- that the serial driver chapter of IM IV tries to explain this,
but don't have a copy handy at the moment.  Anyway ...

> 
> This is the problem:
> (1) The Inside MAC PASCAL calls require VAR argument calls to types other
> than FLOAT or PSTRING.  For example, 
> 	SerGetBuf (refNum: INTEGER; VAR count: LONGINT) : OSErr
> requires a VAR LONGINT type argument.  However, MACL doesn't support these
> types with the :by-reference flag.  
> (2) In fact, many types, such as RECORDs, ARRAYs, and the hundreds of
> MACINTOSH defined PASCAL types, do not appear to be supported by the FFI
> mechanism.  
> 

(1) I don't remember the rationale for calling ":by-reference" by that name,
but it seems to have been an unfortunate choice which causes many people to 
conclude that types other than STRINGs and FLOATs can't be passed by reference.
The Awful Truth is that no Lisp object is -ever- passed "by reference" - e.g.,
by address - to a foreign function: doing so would potentially violate
storage management conventions imposed by the GC and, since Lisp and
foreign functions generally represent data objects differently, wouldn't
be incredibly useful anyway.

In fact, Lisp objects are never passed directly to foreign functions (whether
"by address" or "by value"); in all cases, the Lisp arguments are coerced
to a representation that (should) match that expected by the foreign function -
Lisp strings are coerced to C or Pascal strings, etc. - and either this
"foreign" value or a pointer to it, as appropriate, is passed to the foreign
function.  (Those "foreign" values that require storage - such as floats
and strings - are typically stack-allocated via the %STACK-BLOCK mechanism.)

In some cases, where these "foreign" parameters are passed by address and the
foreign function side-effects them, the FFI can generate code which makes
parallel side-effects on the Lisp objects which were used to initialize the
"foreign" parameters.  Note that it is well-defined to "make the Lisp string
used to initialize the foreign parameter contain the same sequence of
characters as the returned Pascal/C string" and well-defined (although
somewhat more dangerous) to cause a Lisp float to represent the same floating-
point value, but that it's not so well-defined to "make the lisp integer 3 
become 4."  This "parallel side-effecting" behavior is what the :BY-REFERENCE
parameter specifier is all about, and (hopefully) explains why this parameter
specifier can only be used with those primitive Lisp data types for which
side-effecting via destructive modification is possible.

In cases where we can't talk about modifying a Lisp object so that it
represents the same value as a returned "foreign" parameter and where we're
interested in that value, we have to do some of the work that the FFI is
otherwise able to do ourselves: we have to perform the coercion step
"manually" and deal explicitly with the fact that the parameter(s) in
question is/are passed by address.  We also have to arrange that the
address(es) that are passed are non-relocatable and otherwise acceptable
to the GC (on the stack or in the Mac heap somewhere.)  So ...

(deffpfun (SerGetBuf "SerGetBuf") 
  ((fixnum :word) ;; As one would expext
   (t :ptr))      ;; Make the -caller- deal with coercion & allocation
  :word)

(defun test-sergetbuf (refnum)
  (let ((error nil))
   (%stack-block ((countp 4)) ; "count" is an output parameter,
                              ; "countp" is a safe place to put it.
     (setq error (SerGetBuf refnum countp))
     (if (< error 0)
       (error "Bad Things happened in SerGetBuf call - error = ~d" error)
       (%get-long countp))))) ; return SerGetBuf's output parameter.

Okay; that's a bit on the low-level side.  But it's not -that- bad, is it?

(2) The general idea is that one uses DEFRECORD to define the record types
you need to use, some combination of RLET/MAKE-RECORD to allocate instances
of them, and passes them around (as arguments of type :ptr) to foreign 
functions and System/Toolbox traps.

The hard(est) part of all this would seem to be the first part, both because
of the large amount of (tedious) work involved in hand-translating the
Pascal interface files and because of limitations in the DEFRECORD mechanism
(notably in dealing with embedded arrays within record definitions.)  We've
taken some steps in addressing both of these concerns and have discussed
the possibility of making Lisp versions of the interfaces files (freely, I
-think-) available through various Apple Developer channels.

> Our questions are: 
> (1) Short of rewriting every [Not In ROM] call to conform to the few supported
> FFI types, how can we more easily achieve this functionality?
> (2) In general, how can we use the FFI with both standard and user-defined
> types?  
> 
> Thank you. - Mark
> 
> 

Hopefully this made -some- sense.