Issue: DIRECTORY-DOES-TOO-MUCH

[Jim McDonald <jlm@lucid.com>]

While thinking about pathnames, I was reminded of this possible small
addition:

Issue:          DIRECTORY-DOES-TOO-MUCH
Forum:	        Cleanup
References:     DIRECTORY (p427)
Related issues: NONE
Category:       ADDITION
Edit history:   14-Mar-89, Version 1 by James L. McDonald
Status:         For Internal Discussion

Problem description:

  According to CLtL, DIRECTORY returns a list of truenames, "one for
  each file in the file system that matches the given pathname".

  The problem is that sometimes one wants the truenames for just one
  or a few of the files that match, or one wants to interleave
  processing of each file as it is found, to minimize the start-up
  time when processing large directories. 

Proposal (DIRECTORY-DOES-TOO-MUCH:ADD-GENERATOR):

  Add the function DIRECTORY-GENERATOR which would accept the same
  argument spectrum as DIRECTORY and return a function which, when
  successively applied, would yield each of truenames in the list
  of truenames that DIRECTORY would have returned, and then NIL to
  indicate no more files are available. 

Examples:

  This example illustrates how wasted effort could be avoided:

  (DEFUN FIND-DEFINING-FILE (MUMBLE)
    (LET ((FN (DIRECTORY-GENERATOR "/moby-dir/*.lisp")))
      (DO ((TRUENAME (FUNCALL FN) (FUNCALL FN)))
          ((NULL TRUENAME)
           NIL)
        (WHEN (FILE-DEFINES-P TRUENAME MUMBLE)
          (RETURN TRUENAME)))))

  This example shows how a system with some distributed processing 
  ability might interleave file accessing and processing:

  (DEFUN COMPILE-WORLD ()
    (LET ((FN (DIRECTORY-GENERATOR "/moby-dir/*.lisp")))
      (DO ((TRUENAME (FUNCALL FN) (FUNCALL FN)))
          ((NULL TRUENAME)
           NIL)
        (INITIATE-DISTRIBUTED-COMPILATION TRUENAME))))

Test Cases:

  This should return true for all arguments, assuming that during
  the execution of the test files are not added to or removed from
  the file-system being accessed.

  (DEFUN FOO (X)
    (OR (NOT (PATHNAMEP X))
        (NULL (SET-EXCLUSIVE-OR ; why doesn't CL have SET-EQUAL ?
                 (DIRECTORY X)
                 (LET ((FN (DIRECTORY-GENERATOR X))
         	       (DIR '()))
                   (DO ((TRUENAME (FUNCALL FN) (FUNCALL FN)))
                       ((NULL TRUENAME)
			(REVERSE DIR))
                     (PUSH TRUENAME DIR)))))))

Rationale:

  This seems simple, useful, and uncontroversial.  For many file 
  systems, it provides a CL primitive that maps more directly to
  underlying OS primitives.  

Current practice:

  Lucid Common Lisp has always implemented DIRECTORY in much this way.

Cost to Implementors:

  Minimal.  Any port could come into compliance by defining
  DIRECTORY-GENERATOR as:

  (DEFUN DIRECTORY-GENERATOR (X)
    (LET ((DIR (DIRECTORY X)))
      #'(LAMBDA () (POP DIR))))

  Implementing it more directly is probably either a fairly small task
  or clearly impossible.  Either way, not much work.

Cost to Users:

  None.

Cost of non-adoption:

  DIRECTORY continues to be needlessly inefficient in some cases.

Performance impact:

  Some programs may run faster or reduce the maximum delay visible
  to users. 

Benefits:

  See performance impact.

Esthetics:

  Minor.

Discussion:

  The test case presumes truenames are generated in the same order
  that DIRECTORY now lists them.  This is a minor restriction but
  would fail for systems that explicitly sort their results or file
  systems that randomly reorder directories (e.g. on every access).
  Set equivalence is probably just as good a test if anyone cares.