[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
File System Performance Loss in Genera 8.0 (16X slower!)
I think this may be related to the problem discussed in May. See below for
the description and solution.
Date: 6 Jul 90 15:34:17 GMT
From: cogen@XN.LL.MIT.EDU
What has happened to the network file system in going from Genera 7.2 to 8.0?
By one measure, it is 16 TIMES SLOWER THAN BEFORE! I would like to know if
anyone else has experienced this, or is it a problem with our local
installation?
;;; -*- Mode: LISP; Syntax: Common-lisp; Package: FILE-SYSTEM; Base: 10; Patch-File: Yes -*-
;;; Patches to the TCP-FTP client module of the IP-TCP system.
;;; *********************** CHANGE LOG *******************
;;;
;;; 12/28/88 17:27:19 Barmar: Created. Contents taken from
;;; TMC:SYSTEM;GENERA-PATCHES.LISP.
;;;
#| *************** END OF CHANGE LOG ***************|#
;Date: Thu, 10 May 90 14:18 CDT
;From: Stephen L. Nicoud <snicoud@atc.boeing>
;Subject: FS:DIRECTORY-LIST & IP-TCP/FTP
;To: dmitchell@backus
;In-Reply-To: <19900510185459.2.DON@trc.amoco.com>
;
;
; Date: Thu, 10 May 90 13:54 CDT
; From: Donald H. Mitchell <dmitchell@backus.trc.amoco.com>
;
; Stephen L. Nicoud comments on TCP-IP doing a directory listing for the whole
; directory (rather than specific file) when saving a new file on a Unix host.
; On the UX400S running NFS, I see the same behavior everytime I save a file.
; It is disturbing (especially for large directories). Could it be related to
; trying to figure out the version numbers (even for versionless directories, it
; saves version numbers sometimes (I think only when the old file is held by a
; process and the Symbolics doesn't want to delete it even if the Symbolics is
; holding the handle due to slow release)).
;
;Here're some replies I got which, I think, answer at least some your
;questions. There's even a patch from barmar at the end.
;
;Enjoy,
;
;Steve
;
;------- Forwarded Message -------
;Date: Thu, 10 May 90 10:45 PDT
;From: Eric S. Crawley <ESC@STONY-BROOK.SCRC.Symbolics.COM>
;Subject: FS:DIRECTORY-LIST & IP-TCP/FTP
;To: Stephen L. Nicoud <snicoud@atc.boeing.com>
;cc: Slug@WARBUCKS.AI.SRI.COM.ARPANET
;In-Reply-To: <19900510160622.5.SLN@SKAGIT.atc.boeing.com>
;
; Date: Thu, 10 May 90 09:06 PDT
; From: snicoud@atc.boeing.com (Stephen L. Nicoud)
;
; When I try to read my unix mail inbox "/usr/spool/mail/snicoud" I
; sometimes have to wait five or more minutes while my Symbolics waits for
; a directory listing of our Ultrix host's /usr/spool/mail directory,
; which is very large.
;
; So, I think I've traced it down to a call to FS:DIRECTORY-LIST which is
; passed "bcsaic:/usr/spool/mail/snicoud.zmail" (the .zmail file being the
; renamed file for my inbox). The Ultrix host has FTP as its only FILE
; service. Well my Symbolics (7.2 ip-tcp 67.8 (eco 3)) asks Ultrix for
; listing of "/usr/spool/mail". Why should it do that? Why not the whole
; spec: "/usr/spool/mail/snicoud.zmail"? Is this a bug in IP-TCP or is
; this a bug at all?
;
;It is a bug that Unix does not have a way to get file property information on
;a directory by using the ls command and most, if not all, Unix FTP servers
;don't support the STAT command. While it seems perfectly correct to get the
;information for your mail file using the specific file name, if the pathname
;you asked for was a directory, you would get information on all the files in
;the directory and no properties on the directory itself. So, the FTP code
;assumes that the parent directory exists for the pathname you have asked for
;and does a directory list of the the parent directory and extracts the
;information on the file.
;
;I agree that it is a real pain and I have thought about speeding it up. I
;even came up with a patch to make it faster but ran into the behavior
;mentioned above. I suppose one gross kludge would be to try listing the
;specific file and if a long list gets returned, to try the parent directory
;but that is pretty gross.
;
; Ideas, suggestions?
;
;Use NFS? If you are willing to risk problems with getting properites on a
;directory, I might be able to send you the patch that I made.
;
;BTW, In 8.0, I fixed all the timing bugs with FTP connections being blown away
;at inappropriate times.
;------- End of Forwarded Message -------
;
;------- Forwarded Message -------
;Date: Thu, 10 May 90 11:55 PDT
;From: Barry Margolin <barmar@Think.COM>
;Subject: FS:DIRECTORY-LIST & IP-TCP/FTP
;To: Stephen L. Nicoud <snicoud@atc.boeing.com>
;cc: slug@Warbucks.AI.SRI.COM.ARPANET
;In-Reply-To: <19900510160622.5.SLN@SKAGIT.atc.boeing.com>
;
; Date: Thu, 10 May 90 09:06 PDT
; From: snicoud@atc.boeing.com (Stephen L. Nicoud)
;
; When I try to read my unix mail inbox "/usr/spool/mail/snicoud" I
; sometimes have to wait five or more minutes while my Symbolics waits for
; a directory listing of our Ultrix host's /usr/spool/mail directory,
; which is very large.
;
; So, I think I've traced it down to a call to FS:DIRECTORY-LIST which is
; passed "bcsaic:/usr/spool/mail/snicoud.zmail" (the .zmail file being the
; renamed file for my inbox). The Ultrix host has FTP as its only FILE
; service. Well my Symbolics (7.2 ip-tcp 67.8 (eco 3)) asks Ultrix for
; listing of "/usr/spool/mail". Why should it do that? Why not the whole
; spec: "/usr/spool/mail/snicoud.zmail"? Is this a bug in IP-TCP or is
; this a bug at all?
;
; Has anyone else noticed this behavior?
;
;I noticed it and patched around it. The reason it behaves this way is
;because the Unix FTP server's LIST command behaves differently depending
;on whether the argument you give is the name of a file or a directory.
;Thus, if you did
;
; (fs:directory-list "bcsaic:/usr/spool/mail")
;
;this would translate to the FTP command
;
; LIST /usr/spool/mail
;
;and Unix would then list the *contents* of the mail directory rather
;than the mail directory entry itself. Symbolics's way of dealing with
;this is to always ask for the listing of the directory contents and then
;search the results itself.
;
;The reason the FTP server behaves this way is that it simply constructs
;a command line that calls the /bin/ls program. This explanation led me
;to the realization that a better workaround is to prepend the "-d"
;option to the argument, since this tells /bin/ls not to list directory
;contents. With this change in place I also found that I could implement
;wildcard lookups by sending the wildcarded name to the server rather
;than asking for the whole directory and doing the matching locally.
;
;I'll send you my patches under separate cover. They work in both 7.2
;and 8.0.
;
; barmar
;------- End of Forwarded Message -------
;
;------- Forwarded Message -------
;Date: Thu, 10 May 90 11:58 PDT
;From: Barry Margolin <barmar@Think.COM>
;Subject: FS:DIRECTORY-LIST & IP-TCP/FTP
;To: Stephen L. Nicoud <snicoud@atc.boeing.com>
;In-Reply-To: <19900510160622.5.SLN@SKAGIT.atc.boeing.com>
;
;Here are my patches to TCP-FTP to make Unix access more efficient.
;
;=====================================
(SYSTEM-INTERNALS:BEGIN-PATCH-SECTION)
(SYSTEM-INTERNALS:PATCH-SECTION-SOURCE-FILE "SYS:IP-TCP;TCP-FTP.LISP.1527")
(SYSTEM-INTERNALS:PATCH-SECTION-ATTRIBUTES
"-*- Mode: Lisp; Syntax: Common-Lisp; Package: FILE-SYSTEM; Base: 10; Lowercase: Yes -*-")
1(defsubst ftp-command-equal (string1 string2)
(string-equal string1 string2 0 0 4 4))
0;=====================================
(SYSTEM-INTERNALS:BEGIN-PATCH-SECTION)
(SYSTEM-INTERNALS:PATCH-SECTION-SOURCE-FILE "SYS:IP-TCP;TCP-FTP.LISP.1527")
(SYSTEM-INTERNALS:PATCH-SECTION-ATTRIBUTES
"-*- Mode: Lisp; Syntax: Common-Lisp; Package: FILE-SYSTEM; Base: 10; Lowercase: Yes -*-")
(defun-in-flavor (tcp-ftp-directory-lister tcp-ftp-conn) (dir-cmd dir-path pathname)
(selector dir-cmd 1ftp-command-equal
0 (("NLST" "LIST")
(with-open-stream
(stream (tcp-ftp-open-data-connection dir-path dir-path :input nil dir-cmd))
(loop with line and eof
do (cl:multiple-value-setq (line eof) (send stream :line-in))
until eof
when tcp:*tcp-ftp-debug-p*
do (send trace-output :line-out line)
when (send file-access-path :parse-directory-response line pathname dir-cmd)
collect it)))
("STAT"
(multiple-value-bind (line num)
(tcp-ftp-check-command-response '(212 213) "STAT ~A"
(send dir-path :string-for-host))
(send file-access-path :parse-stat-response pathname line num t)))
(otherwise (error "Unknown listing command ~S."1 dir-cmd0))))
;=====================================
(SYSTEM-INTERNALS:BEGIN-PATCH-SECTION)
(SYSTEM-INTERNALS:PATCH-SECTION-SOURCE-FILE "SYS:IP-TCP;TCP-FTP.LISP.137")
(SYSTEM-INTERNALS:PATCH-SECTION-ATTRIBUTES
"-*- Mode: Lisp; Syntax: Common-Lisp; Package: FILE-SYSTEM; Base: 10; Lowercase: Yes -*-")
(defun-in-flavor (tcp-ftp-properties-internal tcp-ftp-conn) (pathname for-list)
(cond ((send file-access-path :operation-handled-p :parse-stat-response)
(multiple-value-bind (line num)
(tcp-ftp-check-command-response '(212 213) "STAT ~A"
(send pathname :string-for-host))
(send file-access-path :parse-stat-response pathname line num for-list)))
((not (1ftp-command-equal0 "NLST" (send file-access-path :directory-command pathname nil)))
(let ((list 1(tcp-ftp-directory-list-internal pathname nil)0))
(when (null list)
(error 'tcp-ftp-file-not-found :pathname pathname :response ""))
(first list)))
(t
(let ((list 1(tcp-ftp-directory-list-internal pathname '(:fast))0))
(when (null list)
(error 'tcp-ftp-file-not-found :pathname pathname :response ""))
(list* pathname
(let ((byte-size (tcp-ftp-default-byte-size pathname)))
(if byte-size
`(:characters nil :byte-size ,byte-size)
'(:characters t))))))))
;=====================================
(SYSTEM-INTERNALS:BEGIN-PATCH-SECTION)
(SYSTEM-INTERNALS:PATCH-SECTION-SOURCE-FILE "SYS:IP-TCP;TCP-FTP.LISP.137")
(SYSTEM-INTERNALS:PATCH-SECTION-ATTRIBUTES
"-*- Mode: Lisp; Syntax: Common-Lisp; Package: FILE-SYSTEM; Base: 10; Lowercase: Yes -*-")
(defmethod (:directory-command unix-tcp-ftp-file-access-path) (pathname options)
(let ((cmd (if (member :fast options)
"NLST -d" 1;-d so we can get directory properties
0 "LIST -d")))
(values cmd
1pathname
0 1#||0(send pathname :directory-pathname-as-file)1||#
0 1nil0)))