Issue: PACKAGE-CLUTTER (Version 2)

[masinter.pa@Xerox.COM]

I've made several changes to the proposal, only some of which were justified by
the discussion:
	I explicitly said that LISP might have additional internal symbols.
	I eliminated the requirement that there be a SYSTEM package. 
		(This is out of the blue. Exactly what belonged in SYSTEM vs STORAGE-ALLOCATOR
			or FILE-SYSTEM always baffled me. )
	I made explict that USER might contain implementation-specific symbols, 
		as well as using implementation-specific packages. 

 	I mentioned that CLOS and the condition system might have their
	own packages. 
I think we might be able to pass a package-clutter issue and deal with a new
issue  MAKE-PACKAGE-USE-DEFAULT separately, or possibly as an amendment to this
one.

!
Issue:        PACKAGE-CLUTTER
References:   LISP, USER, SYSTEM packages (p181)
		   Issue: DEFPACKAGE
		  Issue: IN-PACKAGE-FUNCTIONALITY
Category:     CHANGE/CLARIFICATION
Edit history: 07-Jul-88, Version 1 by Pitman
		23-Sep-88, Version 2 by Masinter

Problem Description:

  CLtL specifies that

   ``The package named LISP contains the primitives of
     the Common Lisp system. Its external symbols include
     all of the user-visible functions and global variables
     that are present in the Common Lisp system, such as
     CAR, CDR, *PACKAGE*, etc. Almost all other packages will
     want to use LISP so that these symbosl will be accessible
     without qualification.''

  It specifies "all" but not "all and only".

  Some implementations place their extensions in the Lisp package.
  Nothing in CLtL explicitly prohibits this, but it leads to problems 
  in general.  For example:

  - A user defining a function by a name not mentioned in CLtL may be
    surprised to clobber a system function in some implementations

  - In one particular implementation, the variable HELP was a system
    constant, so that ((LAMBDA (HELP) ...HELP...) "Press ? for help.")
    signalled a correctable error (asking what variable to bind
    instead of HELP :-).

Proposal (PACKAGE-CLUTTER:REDUCE):

  Specify that, not only must the LISP package  contain at least all of the 
 symbols listed in the standard, it will have no other external symbols.
  (The LISP package may have additional internal symbols.)

  Those symbols with function, variable, macro, setf, constant definitions
  or uses as properties or tokens may have no other additional definitions
  other than those specified.

  Eliminate the requirement that the initial Common Lisp system 
  have a package named "SYSTEM". Specify that implementations may
  have several other packages available.

  Clarify that the "USER" package may have additional symbols interned
  within it and that it may :USE other implementation-specific packages.

 
 Examples:

  #1: The symbol HELP may not be on the LISP package because it is not
      mentioned in CLtL.

  #2: The symbol VARIABLE is specified to be on the LISP package (because
      it is a valid second argument to the DOCUMENTATION function). Since
      it is not defined as a variable, type, or function, however, it may
      not be bound, defined as a type, or defined as a function, macro or
      special form.

Rationale:

  If extra symbols are permitted in the LISP package, users may be surprised
  by relationships between the LISP package and other packages which they
  did not expect, or may be surprised by functionality that they did not
  expect. The degenerate case is:

   (DEFCONSTANT LISP:A 'YOU-LOSE)
   (DEFCONSTANT LISP:B 'YOU-LOSE)
   (DEFCONSTANT LISP:C 'YOU-LOSE)   
   ...
   (DEFCONSTANT LISP:AA 'YOU-LOSE)
   (DEFCONSTANT LISP:AB 'YOU-LOSE)
   (DEFCONSTANT LISP:AB 'YOU-LOSE)
   ...etc.

  Given such an implementation, even things like (LAMBDA (X) X) are not
  valid because they attempt to bind "system constants". It is necessary
  that the programmer be able to know for sure that an arbitrary name is
  "free for use" and best way to conveniently assure this is to require
  that the LISP package be unadulterated.

  As for the additional definitions, there are situations where additional
  definitions would cause a problem. For example, if a symbol on the Lisp
  package were declared as a special variable even though that value was
  not mentioned in the standard, that variable would behave incorrectly when
  used as a lexical variable. Similarly, if a symbol in the lisp package
  were defined as an implementation-dependent special form, problems might
  result if a user redefined or even bound (as by FLET or MACROLET) that
  name.

  The LISP package is the foothold from which portable programs establish
  their desired environment. Careful control is desirable to make sure
  everyone is starting off on the right foot.

Current Practice:

  Some implementations have been known to add additional symbols (usually
  functional and/or variable extensions) to the LISP package.

  Symbolics Genera currently has exactly the set of symbols on it which is
  prescribed by CLtL. A few symbols, such as EVALHOOK, ROOM, and APPLYHOOK
  are spuriously defined as special variables. The symbol LAMBDA is defined
  as a macro. There may be a few other exceptions like this.

Cost to Implementors:

  The actual cost of moving the symbols out of the LISP package in cases
  where they are not already gone is quite small. However, if any
  implementation really has to do this, it may have a number of suppositions
  about what is in what package, and the changes could potentially be extensive.

Cost to Users:

  This change is upward compatible with any portable program, but users
  of a particular implementation's extensions may be forced to find their
  functions in a different package, so there may be a measurable practical
  cost.

  In many cases where an extension symbol FOO is simply expected to have
  been directly available (due to :USE "LISP"), it will work to just just
  do (IMPORT 'new-home-package-for-foo:FOO) where the user's package is
  declared.

  In many cases where an extension symbol FOO is used by explicit package
  prefix, such as LISP:FOO, it should be easy to search for `LISP:FOO' or
  even `LISP:' to find the cases.

Cost of Non-Adoption:

  The potential for the LISP package to be adulterated and for supposedly
  portable programs to have difficulty getting a foothold in some
  implementations will be `noticeably non-zero'.

Benefits:

  Portability of some programs will be enhanced.

Aesthetics:

  This change probably supports the naive expectation of most programmers
  writing portable code.

Discussion:

  This issue came up a while back on Common-Lisp list. The issue
  of whether the   USER package may contain symbols other than those 
  specified in the standard was controversial.  The smart programmer
  of portable code will never rely on the contents of the
  USER package.    However, if someone wants a completely empty 
  package that uses only Lisp, it's easy and portable to create one.

  While it would improve portability slightly to disallow additional internal
  symbols in the LISP package (since it affects what DO-SYMBOLS will do)
  explicitly prohibiting a common practice didn't seem like the best way
  to discourage a possibly troublesome implementation technique. 

  Implementors should be especially careful about accidentally 
  exporting unwanted additional definitions for symbols,e.g., a variable
   definition for EVALHOOK which might show through because of
   an unintended name collision.

  It is likely that the recently included portions of the standard (CLOS and
  the signal mechanism) will reside in their own packages. These externally
  defined packages should have the same constraints as outlined for
  the LISP package here.

  There has been a suggestion that vendor-specific extensions should
  be placed in a package named like ACME-COMMON-LISP for the "Acme"
  company. 

  A registry of packages (as well as features, modules and other global
  names) would be useful, although probably not a part of the language
  standard, per se.

  There is considerable debate in the cleanup committee whether the default
  value for the :USE keyword in IN-PACKAGE, MAKE-PACKAGE (and
  the proposed DEFPACKAGE) should be changed to be implementation 
   specific. This version of the proposal leaves the default value as specified
  in CLtL, namely, that it is the "LISP" package.

  There was not consensus on this sub-issue; it could be handled as a separate
  issue, if this proposal is endorsed by X3J13.