[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
poor make-pipe-stream performance
- To: email@example.com
- Subject: poor make-pipe-stream performance
- From: firstname.lastname@example.org (Marcus Daniels)
- Date: Tue, 7 Dec 93 05:14:50 PST
I'm archiving big records (up to 1.6 meg) as structures, using the
#S() syntax. It is generated by a simple database interface program
that communicates through standard I/O. The records consists
primarily of double-vectors (6 sets, of 300-30000 elements).
My plan was to use make-pipe-input-stream to `read' from this program.
I've found that pipe I/O performance is not as good as AKCL. Reading
directly from a file yields equivalent performance to AKCL, so I
don't believe it is a O.S. thing (and there is plenty of resident
memory free in all cases).
CLISP and AKCL are both compiled with gcc-2.5.6 -O6 on Linux. CLISP
doesn't use shared memory due to the 4MB limitation (in the following
example the memory use is around 7.5 MB).
Here were the results for the largest record (27951 * 6 * double-vector +
3 identifier fields).
CLISP Straight (local) file: 37.6 35.1
CLISP MKFIFO file: 258.3 151.04
CLISP MAKE-PIPE-INPUT-STREAM 232.9 145.27
AKCL Straight (local) file: 33.0 N/A
AKCL RUN-PROCESS 58.0 N/A
The CLISP MKFIFO/MAKE-PIPE exec'ed way the shell... So the fact that AKCL
uses execvp with no shell wouldn't seem to matter.
Also, I played with using dd to reblock stuff for CLISP, but that didn't
seem to accomplish anything..
I'd be willing to pay the 25 second penalty for such records, if
I could use pipe, but not a 3 minute penalty. Are there any buffering
parameters I might play with? What else could account for this discrepancy?
The subshell approach was intended to be tolerable, hopefully temporary,
substitute for a socket based or Postgres backend (perhaps something like
Picasso or CLING). Has anyone ever worked out a simple socket oriented
(i.e. database) interface for CLISP?
I noticed that the only socket support CLISP includes is apparently for
the sake of CLX -- not for general use. Is there any (technical) reason
not to provide such an interface (other than it could be difficult or
impossible to provide GPL'ed on all platforms). But a DJGPP network
capable CLISP would come in _very_ handy for all the DOS bound PC crud