Re: LispM's crummy I/O performance (was RE: LispM Market Share)

[sobeck%russian.spa.symbolics.com@RELAY.CS.NET]

    Date: Tue, 16 Jan 90 19:35:21 EST
    From: alanr@media-lab.media.mit.edu


	  Now, anyone who has touched UNIX knows that reading in a 5meg file and
	  even allocating memory for 1 million tokens (which are represented as
	  integers) takes on the order of a few minutes (at most).  The LispM
	  took almost over 5 hours to read in the data set!  This is on a 3650
	  under 7.2.  What's more, the same LispM took almost 48 hours to dump
	  the data (via NFS to a disk on a SUN4)!
	  The ironic part of this is the CM took only about 15 seconds to
	  actually process the data.

	  I solved the same problem using C without the connection machine.  In
	  fact, on a HP Series 800, the serial solution takes only a few hours,
	  and it would take under an hour if not for paging problems at runtime.

	  I think that 1meg/hr is not an acceptable I/O rate.

	  -- David Magerman (magerman@linc.cis.upenn.edu)
	  University of Pennsylvania LINC Laboratory

    Given this horror story, I suppose that I should chip in with an io story
    that is somewhat more reasonable.

    I also use the connection machine, and needed to get access to ~ 1megabyte tiff
    image files from a networked server. These needed to be converted to
    symbolics rasters (so I could display them) and then downloaded to the cm.

    While my initial implementation was slow (order of several minutes), I was
    able to improve it substantially, to the point that I was getting a
    throughput of around 50k bytes per second over ethernet, comparable to ftp
    between the unix hosts.

If you are willing to work with local FEP files, you can achieve performence of 
about 250K Bytes/sec(reading or writing), not counting the time required to construct 
the data structures.