[smh@Franz.COM: Re: [spr7955] named pipes question]

[oe@dfki.uni-sb.de (Stephan Oepen)]

i am forwarding this to the `allegro-cl' mailing list to get comments
from other common-lisp users.  many thanx to Steve Haflic for the
detailed explanation.  however, i come to think that the topic deserves
further discussion in the light of the forthcoming ansi `standard'.

i still do not fully understand why common-lisp implementations on
systems with fifos as `ordinary' file system objects shall in principle
be unable to handle the synchronization problems on named pipes (and
similar `communication lines', e.g. ttys)

in general one could think of two possible setups:

(1) open() (from inside lisp) on fifos blocks until both sides of the
    pipe are connected to some process (in the appropriate direction).

(2) open() immediately returns but calls to read() or write() will
    block until the associated stream is `fully connected'.

from the Allegro CL (on Unix) implementation point of view i see the
point that in a multi-processing image blocking system calls _are_ a
serious problem.  moreover, reading the open(2) man page gives the
impression that actually there is no way to call open(2) non-blocking
for writing on fifos without getting an error (instead of a file
descriptor |:-) as long as nobody is reading from the other end.
--- hence, option (2) from above seems to be out (and, anyhow, i think
(1) should be preferred).

so, if blocking system calls in a multi-processing image are a hassle
(which i understand), yes, option (1) definitely requires active
polling.  only, why do i have to do that work myself?  i would probably
expect either a decent common-lisp standard to know about fifo-like
objects (and define (1) as the appropriate behaviour) or at least want
my Unix common-lisp implementation to provide that functionality as an
(extra free bonus |:-) implementation-specific add-on.

                                            other opinions?  -  oe

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;; Stephan Oepen, Duerkheimer Str. 7, 66oo Saarbruecken, (o681) - 3o2 52 84
;;;      - oe@dfki.uni-sb.de | oe@cs.tu-berlin.de | oe@gnu.ai.mit.edu -
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;


[--- start of forwarded message ---]

Return-Path: <smh@Franz.COM>
Date: Thu, 1 Apr 93 15:09:37 PST
From: smh@Franz.COM (Steve Haflich)
To: oe@dfki.uni-sb.de
Class: bh
Bh-Id: spr7955
Bh: append spr7955 expire
Subject: Re: [spr7955] named pipes question
Cc: bugs@Franz.COM

[This matter has been assigned the tracking identifier "spr7955".
Please refer to it in any followup communication.  Also, be sure
to cc bugs@franz.com so that if I am unavailable someone else will
be able to respond.]

   Date: Tue, 23 Mar 93 17:38:16 +0100
   Message-Id: <9303231638.AA01997@disco-elc5>
    ...

   but as i write it, here is a `real' bug report concerning Allegro CL
   4.1 itself: i tried to read from and write to named pipes on sparcs
   (SunOS 4.1.1 and 4.1.3) and run into trouble as soon as i call open()
   (from lisp) with whatever arguments to `:if-exists' one can think of.

   what i get is an `interrupted system call' error because the open(2)
   blocks until both sides of the fifo are connected to some process;
   additionally some of the `:if-exists' arguments force a call to
   lseek(2) on the file descriptor which again results in an error.

   i figure that handling of named pipes as ordinary file system objects
   is quite tricky if you want to stick to the CLtL2 arguments to open().
   one would probably have to stat(2) the file first, for fifos add the
   `O_NDELAY' flag to the open(2) call and presumably mark the resulting
   stream as (possibly) incomplete open so that following calls to read()
   and write() would block.

Yes.  The simple truth is that the only portable way to open a stream
(ignoring indirect streams and string streams) is to call open with a
pathname in the file system.  The language definition considers file
names to name regular files, and in fact, the Allegro language
implementation make no special consideration of filesystem nodes that
might name real devices, pseudo devices, or sockets.  Any manipulation
of such objects are outside the standard, and therefore (at least some
of them) must be done with non-portable code.

Unix has two kinds of system calls, "slow" calls that are potentially
interruptible (returning EINTR) and "fast" calls which return
immediately.  open() on a filesystem file is "fast" and is supposed to
return immediately, but open on a named pipe is slow and may hang for
an arbitrarily large time.  The Common Lisp OPEN function is not
appropriate for such files, at least not in a multiprocessing image.
I don't think this is particularly a Lisp problem, in that I can't
think of any way even in C to wait for a named pipe to be in a state
that wouldn't block, other than using polling.

   our solution finally was to run-shell-command() cat(1) to read and
   write to named pipes but i honestly dislike that.

What this does, in effect, is to form off a separate Unix process (the
cat) which doesn't matter if it blocks for a long time in the open.
Polling is one alternative to this design, but that's also ugly in
that it imposes runtime load on your machine.  To use it, you would
have to defforeign _open (obviously with a different Lisp name, since
CL:OPEN is already taken) and then call it with the pathname and the
O_NDELAY flag, looping with a process-sleep until open returns a
nonnegative value.  Once you have obtained a file descriptor, you can
use this to wrap a stream around it:

   (make-instance 'excl::bidirectional-terminal-stream
		  :fn-in  fd
		  :fn-out fd)

This is taken from the function ipc:make-ipc-terminal-stream in the
library file ipc.cl.  You might want to examine this file for some
other examples of networking, although I don't think anything there
addresses the problem of waiting indefinitely for a the other end of a
socket to become connected.

I hope this information is of some use.  It's a hard problem.

[--- end of forwarded message ---]