[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

File System Performance Loss in Genera 8.0 (16X slower!)



I think this may be related to the problem discussed in May.  See below for
the description and solution.

    Date: 6 Jul 90 15:34:17 GMT
    From: cogen@XN.LL.MIT.EDU

    What has happened to the network file system in going from Genera 7.2 to 8.0?
    By one measure, it is 16 TIMES SLOWER THAN BEFORE! I would like to know if
    anyone else has experienced this, or is it a problem with our local
    installation?


;;; -*- Mode: LISP; Syntax: Common-lisp; Package: FILE-SYSTEM; Base: 10; Patch-File: Yes -*-

;;; Patches to the TCP-FTP client module of the IP-TCP system.

;;; *********************** CHANGE LOG *******************
;;; 
;;; 12/28/88 17:27:19 Barmar:  Created.  Contents taken from
;;; TMC:SYSTEM;GENERA-PATCHES.LISP.
;;; 
#| *************** END OF CHANGE LOG ***************|#
;Date: Thu, 10 May 90 14:18 CDT
;From: Stephen L. Nicoud <snicoud@atc.boeing>
;Subject: FS:DIRECTORY-LIST & IP-TCP/FTP
;To: dmitchell@backus
;In-Reply-To: <19900510185459.2.DON@trc.amoco.com>
;
;
;    Date: Thu, 10 May 90 13:54 CDT
;    From: Donald H. Mitchell <dmitchell@backus.trc.amoco.com>
;
;    Stephen L. Nicoud comments on TCP-IP doing a directory listing for the whole
;    directory (rather than specific file) when saving a new file on a Unix host.
;    On the UX400S running NFS, I see the same behavior everytime I save a file.
;    It is disturbing (especially for large directories).  Could it be related to
;    trying to figure out the version numbers (even for versionless directories, it
;    saves version numbers sometimes (I think only when the old file is held by a
;    process and the Symbolics doesn't want to delete it even if the Symbolics is
;    holding the handle due to slow release)).
;
;Here're some replies I got which, I think, answer at least some your
;questions.  There's even a patch from barmar at the end.
;
;Enjoy,
;
;Steve
;
;------- Forwarded Message -------
;Date: Thu, 10 May 90 10:45 PDT
;From: Eric S. Crawley <ESC@STONY-BROOK.SCRC.Symbolics.COM>
;Subject: FS:DIRECTORY-LIST & IP-TCP/FTP
;To: Stephen L. Nicoud <snicoud@atc.boeing.com>
;cc: Slug@WARBUCKS.AI.SRI.COM.ARPANET
;In-Reply-To: <19900510160622.5.SLN@SKAGIT.atc.boeing.com>
;
;    Date: Thu, 10 May 90 09:06 PDT
;    From: snicoud@atc.boeing.com (Stephen L. Nicoud)
;
;    When I try to read my unix mail inbox "/usr/spool/mail/snicoud" I
;    sometimes have to wait five or more minutes while my Symbolics waits for
;    a directory listing of our Ultrix host's /usr/spool/mail directory,
;    which is very large.
;
;    So, I think I've traced it down to a call to FS:DIRECTORY-LIST which is
;    passed "bcsaic:/usr/spool/mail/snicoud.zmail" (the .zmail file being the
;    renamed file for my inbox).  The Ultrix host has FTP as its only FILE
;    service.  Well my Symbolics (7.2 ip-tcp 67.8 (eco 3)) asks Ultrix for
;    listing of "/usr/spool/mail".  Why should it do that?  Why not the whole
;    spec:  "/usr/spool/mail/snicoud.zmail"?  Is this a bug in IP-TCP or is
;    this a bug at all?
;
;It is a bug that Unix does not have a way to get file property information on
;a directory by using the ls command and most, if not all, Unix FTP servers
;don't support the STAT command.  While it seems perfectly correct to get the
;information for your mail file using the specific file name, if the pathname
;you asked for was a directory, you would get information on all the files in
;the directory and no properties on the directory itself.  So, the FTP code
;assumes that the parent directory exists for the pathname you have asked for
;and does a directory list of the the parent directory and extracts the
;information on the file.  
;
;I agree that it is a real pain and I have thought about speeding it up.  I
;even came up with a patch to make it faster but ran into the behavior
;mentioned above.  I suppose one gross kludge would be to try listing the
;specific file and if a long list gets returned, to try the parent directory
;but that is pretty gross.
;
;    Ideas, suggestions?
;
;Use NFS?  If you are willing to risk problems with getting properites on a
;directory, I might be able to send you the patch that I made.
;
;BTW, In 8.0, I fixed all the timing bugs with FTP connections being blown away
;at inappropriate times.
;------- End of Forwarded Message -------
;
;------- Forwarded Message -------
;Date: Thu, 10 May 90 11:55 PDT
;From: Barry Margolin <barmar@Think.COM>
;Subject: FS:DIRECTORY-LIST & IP-TCP/FTP
;To: Stephen L. Nicoud <snicoud@atc.boeing.com>
;cc: slug@Warbucks.AI.SRI.COM.ARPANET
;In-Reply-To: <19900510160622.5.SLN@SKAGIT.atc.boeing.com>
;
;    Date: Thu, 10 May 90 09:06 PDT
;    From: snicoud@atc.boeing.com (Stephen L. Nicoud)
;
;    When I try to read my unix mail inbox "/usr/spool/mail/snicoud" I
;    sometimes have to wait five or more minutes while my Symbolics waits for
;    a directory listing of our Ultrix host's /usr/spool/mail directory,
;    which is very large.
;
;    So, I think I've traced it down to a call to FS:DIRECTORY-LIST which is
;    passed "bcsaic:/usr/spool/mail/snicoud.zmail" (the .zmail file being the
;    renamed file for my inbox).  The Ultrix host has FTP as its only FILE
;    service.  Well my Symbolics (7.2 ip-tcp 67.8 (eco 3)) asks Ultrix for
;    listing of "/usr/spool/mail".  Why should it do that?  Why not the whole
;    spec:  "/usr/spool/mail/snicoud.zmail"?  Is this a bug in IP-TCP or is
;    this a bug at all?
;
;    Has anyone else noticed this behavior?
;
;I noticed it and patched around it.  The reason it behaves this way is
;because the Unix FTP server's LIST command behaves differently depending
;on whether the argument you give is the name of a file or a directory.
;Thus, if you did
;
;	(fs:directory-list "bcsaic:/usr/spool/mail")
;
;this would translate to the FTP command
;
;	LIST /usr/spool/mail
;
;and Unix would then list the *contents* of the mail directory rather
;than the mail directory entry itself.  Symbolics's way of dealing with
;this is to always ask for the listing of the directory contents and then
;search the results itself.
;
;The reason the FTP server behaves this way is that it simply constructs
;a command line that calls the /bin/ls program.  This explanation led me
;to the realization that a better workaround is to prepend the "-d"
;option to the argument, since this tells /bin/ls not to list directory
;contents.  With this change in place I also found that I could implement
;wildcard lookups by sending the wildcarded name to the server rather
;than asking for the whole directory and doing the matching locally.
;
;I'll send you my patches under separate cover.  They work in both 7.2
;and 8.0.
;
;                                                barmar
;------- End of Forwarded Message -------
;
;------- Forwarded Message -------
;Date: Thu, 10 May 90 11:58 PDT
;From: Barry Margolin <barmar@Think.COM>
;Subject: FS:DIRECTORY-LIST & IP-TCP/FTP
;To: Stephen L. Nicoud <snicoud@atc.boeing.com>
;In-Reply-To: <19900510160622.5.SLN@SKAGIT.atc.boeing.com>
;
;Here are my patches to TCP-FTP to make Unix access more efficient.
;


;=====================================
(SYSTEM-INTERNALS:BEGIN-PATCH-SECTION)
(SYSTEM-INTERNALS:PATCH-SECTION-SOURCE-FILE "SYS:IP-TCP;TCP-FTP.LISP.1527")
(SYSTEM-INTERNALS:PATCH-SECTION-ATTRIBUTES
  "-*- Mode: Lisp; Syntax: Common-Lisp; Package: FILE-SYSTEM; Base: 10; Lowercase: Yes -*-")

1(defsubst ftp-command-equal (string1 string2)
  (string-equal string1 string2 0 0 4 4))

0;=====================================
(SYSTEM-INTERNALS:BEGIN-PATCH-SECTION)
(SYSTEM-INTERNALS:PATCH-SECTION-SOURCE-FILE "SYS:IP-TCP;TCP-FTP.LISP.1527")
(SYSTEM-INTERNALS:PATCH-SECTION-ATTRIBUTES
  "-*- Mode: Lisp; Syntax: Common-Lisp; Package: FILE-SYSTEM; Base: 10; Lowercase: Yes -*-")

(defun-in-flavor (tcp-ftp-directory-lister tcp-ftp-conn) (dir-cmd dir-path pathname)
  (selector dir-cmd 1ftp-command-equal
0    (("NLST" "LIST")
     (with-open-stream
       (stream (tcp-ftp-open-data-connection dir-path dir-path :input nil dir-cmd))
       (loop with line and eof
	     do (cl:multiple-value-setq (line eof) (send stream :line-in))
	     until eof
	     when tcp:*tcp-ftp-debug-p*
	       do (send trace-output :line-out line)
	     when (send file-access-path :parse-directory-response line pathname dir-cmd)
	       collect it)))
    ("STAT"
     (multiple-value-bind (line num)
	 (tcp-ftp-check-command-response '(212 213) "STAT ~A"
					 (send dir-path :string-for-host))
       (send file-access-path :parse-stat-response pathname line num t)))
    (otherwise (error "Unknown listing command ~S."1 dir-cmd0))))


;=====================================

(SYSTEM-INTERNALS:BEGIN-PATCH-SECTION)
(SYSTEM-INTERNALS:PATCH-SECTION-SOURCE-FILE "SYS:IP-TCP;TCP-FTP.LISP.137")
(SYSTEM-INTERNALS:PATCH-SECTION-ATTRIBUTES
  "-*- Mode: Lisp; Syntax: Common-Lisp; Package: FILE-SYSTEM; Base: 10; Lowercase: Yes -*-")


(defun-in-flavor (tcp-ftp-properties-internal tcp-ftp-conn) (pathname for-list)
  (cond ((send file-access-path :operation-handled-p :parse-stat-response)
	 (multiple-value-bind (line num)
	     (tcp-ftp-check-command-response '(212 213) "STAT ~A"
					     (send pathname :string-for-host))
	   (send file-access-path :parse-stat-response pathname line num for-list)))
	((not (1ftp-command-equal0 "NLST" (send file-access-path :directory-command pathname nil)))
	 (let ((list 1(tcp-ftp-directory-list-internal pathname nil)0))
	   (when (null list)
	     (error 'tcp-ftp-file-not-found :pathname pathname :response ""))
	   (first list)))
	(t
	 (let ((list 1(tcp-ftp-directory-list-internal pathname '(:fast))0))
	   (when (null list)
	     (error 'tcp-ftp-file-not-found :pathname pathname :response ""))
	   (list* pathname	 
		(let ((byte-size (tcp-ftp-default-byte-size pathname)))
		  (if byte-size
		      `(:characters nil :byte-size ,byte-size)
		      '(:characters t))))))))

;=====================================
(SYSTEM-INTERNALS:BEGIN-PATCH-SECTION)
(SYSTEM-INTERNALS:PATCH-SECTION-SOURCE-FILE "SYS:IP-TCP;TCP-FTP.LISP.137")
(SYSTEM-INTERNALS:PATCH-SECTION-ATTRIBUTES
  "-*- Mode: Lisp; Syntax: Common-Lisp; Package: FILE-SYSTEM; Base: 10; Lowercase: Yes -*-")


(defmethod (:directory-command unix-tcp-ftp-file-access-path) (pathname options)
  (let ((cmd (if (member :fast options)
		 "NLST -d"			1;-d so we can get directory properties
0		 "LIST -d")))
    (values cmd
	    1pathname
0	    1#||0(send pathname :directory-pathname-as-file)1||#
0	    1nil0)))