Issue: PATHNAME-WILD (version 5)

[David A. Moon <Moon@STONY-BROOK.SCRC.Symbolics.COM>]

This issue is on the agenda for the June X3J13 meeting.  KMP and I
have prepared a revised writeup which we think is ready for release.
I'd like to distribute this to X3J13 as soon as discussion, if any,
in the cleanup subcommittee is completed.

Issue:        PATHNAME-WILD
References:   Pathnames (pp410-413)
Related issues: PATHNAME-LOGICAL
Category:     ADDITION
Edit history: 21-Jul-88, Version 1 by Pitman
              06-Oct-88, Version 2 by Pitman
               9-May-89, Version 3 by Moon (small fixes)
              10-May-89, Version 4 by Moon (add two more functions)
              13-May-89, Version 5 by Moon (minor cleanups, add clarification)

Problem Description:

  Some file systems provide more complex conventions for wildcards than
  simple component-wise wildcards (:WILD). For example,

  "F*O" might mean:
    - a normal three character name
    - a three-character name, with the middle char wild
    - at least a two-character name, with the middle 0 or more chars wild
    - a wild match spanning multiple directories

  ">foo>*>bar" might imply:
    - the middle directory is named "*"
    - the middle directory is :WILD
    - there may be zero or more :WILD middle directories
    - the middle directory name matches any one-letter name

  ">foo>**>bar" might mean
    - the middle directory is named "**"
    - the middle directory is :WILD
    - there may be zero or more :WILD middle directories
    - the middle directory name matches any two-letter name

  Some file systems support even more complex wildcards, for example
  regular expressions.

  The CL pathname model does not specify a way to represent complex
  wildcards, which means, for example, that (MAKE-PATHNAME :NAME "F*O")
  cannot be recognized by portable code as containing a wildcard.

  Common Lisp provides only the first of these four common operations
  on wildcard pathnames:
  (1) Enumerate the set of existing files that match the pathname;
  this is provided by the DIRECTORY function.
  (2) Test whether a pathname contains wildcards.
  (3) Test whether a pathname matches a wildcard pathname.
  (4) Translate one pathname into another according to a mapping specified
  by a pair of wildcard pathnames.

Proposal (PATHNAME-WILD:NEW-FUNCTIONS):

  Introduce the following three functions:

    WILD-PATHNAME-P pathname &optional field-key
  
    Tests a pathname for the presence of wildcard components.  If the first
    argument is not a pathname, string, or file stream an error of type
    TYPE-ERROR is signalled.
  
    If no <field-key> is provided, or the <field-key> is NIL, the result is
    T if <pathname> has any wildcard components, NIL if <pathname> has none.
  
    If a non-null <field-key> is provided, it must be one of :HOST, :DEVICE,
    :DIRECTORY, :NAME, :TYPE, or :VERSION.  In this case, the result is T
    if the indicated component of <pathname> is a wildcard, NIL if the
    component is not a wildcard.


    PATHNAME-MATCH-P pathname wildcard

    T if <pathname> matches <wildcard>, otherwise NIL.  The matching rules
    are implementation-dependent but should be consistent with the
    DIRECTORY function.  Missing components of <wildcard> default to :WILD.

    If either argument is not a pathname, string, or file stream an error
    of type TYPE-ERROR is signalled.  It is valid for <pathname> to be a
    wild pathname.  It is valid for <wildcard> to be a non-wild pathname.


    TRANSLATE-PATHNAME source from-wildcard to-wildcard &optional reversible

    Translate the pathname <source> according to the correspondence between
    the two wildcard pathnames.  This translation is implementation
    dependent.  The result is <to-wildcard> with each missing or wildcard
    field replaced by the portion of <source> that matches the corresponding
    field (usually a wildcard) in <from-wildcard>.  Additional translations
    of alphabetic case or file naming conventions might also occur,
    especially when from-wildcard and to-wildcard are for different hosts.

    If <reversible> is false, the translation is determined by the user
    interface conventions of the file systems involved.  If <reversible> is
    true, the translation must instead be reversible, that is, the
    following identity must hold for all cases where no error is signalled:

      (equal (translate-pathname (translate-pathname pathname from to t)
                                 to from t)
             pathname)

    In some file systems the above identity is true only when
    (member (pathname-version pathname) '(:newest :unspecific)).
    This is considered valid, as Common Lisp cannot force all the
    file systems in the world to implement versions.

    In some file systems the <reversible> argument is ignored because the
    user interface conventions are reversible anyway.

    If any of the first three arguments is not a pathname, string, or file
    stream an error of type TYPE-ERROR is signalled.  It is valid for
    <source> to be a wild pathname; in general this will produce a wild
    result.  It is valid for <from-wildcard> and/or <to-wildcard> to be
    non-wild pathnames.  (PATHNAME-MATCH-P <source> <from-wildcard>) must
    be true or an error is signalled.
    
    Implementation guideline: one typical file system performs this
    operation by examining each field of the pathnames in turn, where a
    field is a component or an element of a structured component such as a
    hierarchical directory.  Hierarchical directory elements in
    <from-wildcard> and <to-wildcard> are matched by whether they are
    wildcards, not by depth in the directory hierarchy.  If the field in
    <from-wildcard> does not match the field in <source>, an error is
    signalled.  If the field in <to-wildcard> is present and not wild, it
    is copied into the result.   If the field in <to-wildcard> is :WILD or
    NIL, and either <reversible> is false or the field in <from-wildcard>
    is not a complex wildcard, the field in <source> is copied into the
    result.  Otherwise, the field in <to-wildcard> might be a complex
    wildcard such as "foo*bar" and the field in <from-wildcard> should be
    wild; the portion of the field in <source> that matches the wildcard
    portion of the field in <from-wildcard> fills in the wildcard portion
    of the field in <to-wildcard> and the field value produced is used in
    the result.

  Clarify that the functions OPEN (and WITH-OPEN-FILE), RENAME-FILE,
  DELETE-FILE, PROBE-FILE, FILE-WRITE-DATE, FILE-AUTHOR, LOAD,
  COMPILE-FILE, and TRUENAME only accept non-wildcard pathnames and signal
  an error if given a pathname for which WILD-PATHNAME-P returns true.

Examples:

  (WILD-PATHNAME-P (MAKE-PATHNAME :NAME :WILD)) => T
  (WILD-PATHNAME-P (MAKE-PATHNAME :NAME :WILD) :NAME) => T
  (WILD-PATHNAME-P (MAKE-PATHNAME :NAME :WILD) :TYPE) => NIL
  (WILD-PATHNAME-P (PATHNAME "S:>foo>**>")) => T   ;Lispm
  (WILD-PATHNAME-P (PATHNAME :NAME "F*O")) => T    ;Most places

  ;This example assumes one particular set of wildcard conventions
  ;Not all file systems will run this example exactly as written
  (DEFUN RENAME-FILES (FROM TO)
    (DOLIST (FILE (DIRECTORY FROM))
      (RENAME-FILE FILE (TRANSLATE-PATHNAME FILE FROM TO))))
  (RENAME-FILES "/usr/me/*.lisp" "/dev/her/*.l")
    ;Renames /usr/me/init.lisp to /dev/her/init.l
  (RENAME-FILES "/usr/me/pcl*/*" "/sys/pcl/*/")
    ;Renames /usr/me/pcl-5-may/low.lisp to /sys/pcl/pcl-5-may/low.lisp
    ;In some file systems the result might be /sys/pcl/5-may/low.lisp
  (RENAME-FILES "/usr/me/pcl*/*" "/sys/library/*/")
    ;Renames /usr/me/pcl-5-may/low.lisp to /sys/library/pcl-5-may/low.lisp
    ;In some file systems the result might be /sys/library/5-may/low.lisp
  (RENAME-FILES "/usr/me/foo.bar" "/usr/me2/")
    ;Renames /usr/me/foo.bar to /usr/me2/foo.bar

  ;This example assumes one particular set of wildcard conventions and
  ;illustrates how and why reversible translation uses different rules
  (PATHNAME-NAME (TRANSLATE-PATHNAME "foobar" "foo*" "*baz" NIL)) => "barbaz"
  (PATHNAME-NAME (TRANSLATE-PATHNAME "foobar" "foo*" "*baz" T))   => "barbaz"
  (PATHNAME-NAME (TRANSLATE-PATHNAME "foobar" "foo*" "*"    NIL)) => "foobar"
  (PATHNAME-NAME (TRANSLATE-PATHNAME "foobar" "foo*" "*"    T))   => "bar"
  (PATHNAME-NAME (TRANSLATE-PATHNAME "foobar" "*"    "foo*" NIL)) => "foofoobar"
  (PATHNAME-NAME (TRANSLATE-PATHNAME "foobar" "*"    "foo*" T))   => "foofoobar"
  (PATHNAME-NAME (TRANSLATE-PATHNAME "bar"    "*"    "foo*" NIL)) => "foobar"
  (PATHNAME-NAME (TRANSLATE-PATHNAME "bar"    "*"    "foo*" T))   => "foobar"

  ;This example presumes background information described in PATHNAME-LOGICAL
  (DEFUN TRANSLATE-LOGICAL-PATHNAME-1 (PATHNAME RULES)
    (LET ((RULE (ASSOC PATHNAME RULES :TEST #'PATHNAME-MATCH-P)))
      (UNLESS RULE (ERROR "No translation rule for ~A" PATHNAME))
      (TRANSLATE-PATHNAME PATHNAME (FIRST RULE) (SECOND RULE) T)))
  (TRANSLATE-LOGICAL-PATHNAME-1 "FOO:CODE;BASIC.LISP"
                        '(("FOO:DOCUMENTATION;" "MY-UNIX:/doc/foo/")
                          ("FOO:CODE;"          "MY-UNIX:/lib/foo/")
                          ("FOO:PATCHES;*;"     "MY-UNIX:/lib/foo/patch/*/")))
   => the pathname MY-UNIX:/lib/foo/basic.l

Rationale:

  These three functions provide a standardized interface to the
  idiosyncratic wildcard functionality of each host file system.

  WILD-PATHNAME-P makes it possible to detect wild pathnames reliably and
  do something useful (give up, merge out the bothersome components, call
  DIRECTORY for a list of matching pathnames, etc.)

  TRANSLATE-PATHNAME is needed by many application programs that deal with
  wildcard pathnames.  PATHNAME-MATCH-P and TRANSLATE-PATHNAME are needed
  by logical pathnames.  The reversible feature is needed by logical
  pathnames.

Current Practice:

  Presumably no implementation supports the proposal exactly as stated.
  Symbolics Genera has had similar features under different names for many
  years:

    (SEND pathname :WILD-P) returns a value such as NIL, :NAME, :TYPE,
    etc., indicating the first wild field.

    (SEND pathname :NAME-WILD-P), (SEND pathname :DIRECTORY-WILD-P),
    etc. test individual fields.

    The :TRANSLATE-WILD-PATHNAME, :TRANSLATE-WILD-PATHNAME-REVERSIBLE, and
    :PATHNAME-MATCH messages resemble TRANSLATE-PATHNAME and
    PATHNAME-MATCH-P.

  The clarification is current practice as far as the authors are aware.
  If some implementations are found that specify a meaning for wildcard
  pathnames as arguments to these functions, this proposal should be changed
  to say that the consequences are unspecified rather than signalling an error.

Cost to Implementors:

  Many implementations probably have a substrate which is capable of this
  or something similar already. In such cases, it's a relatively small
  matter to add the proposed interface.

  Even in cases where an implementation doesn't have ready code, it's clearly
  better for the implementor to write that code once and for all than to ask
  each user of wildcards to write it.

  Since the detailed behavior is at the implementor's discretion, the cost
  is unlikely to be large.  Some file systems will do all the work and the
  implementor need only provide an interface to the file system or to a
  standard library routine.  For other file systems the implementor has to
  write the actual matching and translation algorithms.

Cost to Users:

  None. This change is upward compatible.

Cost of Non-Adoption:

  Wild pathnames would continue to be mistaken for ordinary pathnames in
  many situations.  User programs that deal with wildcard pathnames would
  have to operate on implementation-dependent representations and hence
  would not be easily portable.

  The biggest cost is that any logical pathnames proposal would be stymied.

Performance Impact:

  None.

Benefits:

  A more complete set of wildcard pathname operations.  Portable user
  programs that deal with wildcard pathnames will be more consistent
  and reliable.  A portable system construction tool can be written
  and the foundations are laid for a `logical pathname' facility
  (proposed separately in PATHNAME-LOGICAL).

Aesthetics:

  This change would make some portable code less kludgey.

Discussion:

  There was some question about the name. The name PATHNAME-WILD-P
  suggests a ``slot'' of a pathname (like PATHNAME-HOST),
  while WILD-PATHNAME-P suggests a type (like INPUT-STREAM-P).
  The committee was split on what to call it. Since it is more
  like a type than a slot, the name WILD-PATHNAME-P was chosen.