foreign function calls

[John Alan McDonald <jam@stat.washington.edu>]

I have some questions about the foreign function interface. I'm not sure
of the best place to send this kind of question, so I'm cc-ing both
bugs@franz.com and allegro-cl@berkeley.edu. 

We are trying to create an interface to a fortran package for constrained
optimization called NPSOL.  NPSOL normally takes user supplied
(fortran) subroutines to compute the objective function to be minimized,
its gradient, etc. We want the user to be able to supply lisp functions
rather than fortran subroutines, which means we need to call back to lisp
from fortran (or from C called from fortran).

Before I invest a lot of time, I'd like to clear up a few points that
I find confusing in ch. 10 of the User Guide.
I'll appreciate any help or suggestions.

The recommended scenario seems to be something like this:

1) Lisp registers the data (several NxN arrays of double-floats,
	where N may be as much as 1000) to be passed to NPSOL using
	ff:register-value and registers the lisp objective and gradient
	functions (eg. objfun-l) using ff:register-function.

2) Lisp calls a C wrapper for NPSOL (say npsol_c), passing the indexes of the
	registered data and functions.
	
3) npsol_c saves the indexes in global variables.
	It retrieves pointers to the lisp arrays using lisp_value.
	It calls NPSOL with the pointers to lisp arrays and with
	fixed C procedures as the objective and gradient subroutines
	(eg. objfun_c).

4) NPSOL, in the course of minimization calls the C procedures (eg. objfun_c),
	with arguments that are pointers to lisp arrays.

5) objfun_c looks in the appropriate global variable to find the index
	of objfun-l and calls it (using lisp_call) with arguments that
	are pointers to lisp arrays that it got from NPSOL.
	
6) A garbage collection may occur. (Section 10.8 seems to imply that gc
	cannot happen during the execution of foreign code, except during
	the call back to the lisp function. Is this true---even with
	multiprocessing? If it's not true, how can any foreign function
	call work reliably?)

7) At this point, any pointers to lisp arrays in foreign code may be invalid,
	so objfun_c should refetch the pointers	to lisp arrays, using
	lisp_value, before returning control to NPSOL.

The problem with this scenario is that there are pointers to lisp arrays
scattered throughout NPSOL that objfun_c can't possibly fix. I can imagine
several ways to deal with the problem:
	
a) Suspend all scavenging during the execution of foreign code.
	As far as I can tell, there is no mechanism provided to do this.
	
b) Arrange to tenure all the lisp objects to be passed to foreign code
	and prevent global gc by (setf excl:*global-gc-behavior* nil).
	The problem with this is that there doesn't seem to be any way to 
	guarantee that the lisp objects are tenured. Sections 3.1.2 and
	3.1.4 are contradictory on this point. Item (3) in 3.1.4 claims that
	(excl:gc :tenure) "will cause ALL live objects to be tenured,"
	but the description of excl:gc in 3.1.2 warns that
	"You should not assume that all objects will be tenured..."

c) Have the wrapper npsol_c copy all the lisp arrays to foreign arrays,
	except for those that are going to be passed back to lisp from
	objfun_c. This is undesirable because, for large problems,
	it means duplicating tens of megabytes of data. This raises a
	question---can malloc be used in code loaded into lisp, or do I
	have to predeclare large global arrays to be used to hold the copies?



System description: 1990-8-10 15:24:19

Site: belgica.stat.washington.edu.
Machine: Sun4 id: 587210204 

Lisp: Allegro CL 
3.1.13.1 [Sun4] (0/0/0).
Software: SunOS.
Features: (LOOP LOOP COMPOSER FAKE-ACTIVE-REGIONS PCL CLOS BIG-ENDIAN GSGC
	 ALLEGRO-V3.1 FRANZ-INC EXCL ALLEGRO COMMON-LISP CONFORMING-IEEE IEEE
	FLAVORS SUNOS4.0 SUN SPARC SUN4 UNIX MULTIPROCESSING CLX XLIB
	CLX-MIT-R4 CLX-CL-ERROR CW-X).

John Alan McDonald, Statistics, GN-22, U of W, Seattle, Wa. 98195
(206) 545 7438, 543 2517
jam@stat.washington.edu