Pipelining streams

["Robert W. Kerns" <RWK@scrc-yukon.arpa>]

    Date: Wed, 21 Oct 87 10:48 EDT
    From: J. Scott Penberthy  <JSP@IBM.COM>

There are both coroutine streams and pipes implemented in the system.
See page 10 of the reference guide to Streams, Files, and I/O.

You'll need the following if you're going to do anything but
(unsigned-byte 8) IO in 7.1 (fixed in 7.2.)


Contrary to the documentation, pipes exist.  See the functions
cli::make-unidirectional-pipe and cli::make-bidirectional-pipe.

cli::
(defun open-coroutine-stream (function &key (buffer-size 1000) (element-type 'character)
			      (direction :input))
  (ecase direction
    (:input
      (make-coroutine-unidirectional-stream-internal
	'coroutine-input-stream 'coroutine-output-stream function nil
	buffer-size element-type))
    (:output
      (make-coroutine-unidirectional-stream-internal
	'coroutine-output-stream 'coroutine-input-stream function nil
	buffer-size element-type))
    (:bidirectional
      (make-coroutine-bidirectional-stream-internal function nil buffer-size element-type))))

    Has anyone had any luck trying to 1pipeline0 streams?  Suppose that you
    had a function 2foo0 that takes an input stream as an argument.  Further,
    suppose you had another function 2bar0 that takes an output stream as an
    argument.  For example,

    (defun 2foo0 (input-stream)
      (read-and-process-stream input-stream))

    (defun 2bar0 (output-stream)
      (create-garbage-and-write-to output-stream))

    I'd like to be able to call a third function, 2pipeline0, that would take the
    output generated by 2bar0 and send it as input to the function 2foo0.  For
    example,

    (pipeline bar foo)

    should do the right thing.  For now, I'm using those fake "string
    streams," using macros like

    (defmacro with-output-to-stream ((s) &body body)
      `(make-string-input-stream
	 (with-output-to-string (,s)
	   ,@body)))

    and

    (defun bad-pipeline-implementation (f1 f2)
      (funcall
	f2
	(with-output-to-stream (s)
	  (funcall f1 s))))

    However, this hack creates huge temporary strings.  In addition, it
    needlessly gathers all of the output from the first function before
    sending it as input to the second function.  Does anyone know of a
    better way to solve this problem?  In particular, does anyone have a
    solution that permits concurrent execution of the pipelined functions?

    Thanks!

    Scott