Making KCL

[ajb@kerri.oz.au (Anthony Berglas)]

I have just moved KCL minver 257 (not the current one)
to SCO SysV Unix 3.2 and have had problems, mainly with the make files.
I think wfs did the original port some time ago.
I will now describe my experiance as it may be helpfull to anyone else
moving KCL around and then some suggestions about how the makefiles could
be changed, fairly easily, so that even a mug like me can work with them.

Specific Problems moving to Unix 386.
-------------------------------------

1) I used the rcc ("real" at&t cc, not the microsoft compiler), unoptimised
   on Unix 386, not Xenix.  The microsoft compiler blew up at merge.c because
   macro invocations had spaces between the macro name and the '(', I did not
   investigate it further.  WFS says rcc produces bad code when optimized
   and that gcc works fine.

2) add-defs mp386

3) I needed to add SHELL=/bin/sh, SCO doesn't seem to recognise the #!/bin/sh.

4) evalmacros.data, cmpspecial.data and move-if-changed
   are filenames bigger that 14 chars, producing an Error.
   AT&T may remove this restriction in the next couple of decades but I would
   suggest that they be renamed to evalmacs and cmpspec, as I have done.
   The makefiles and sys_kcl.c needed to be changed appopriately.
   There is a wierd reference to evalmacros in xbin/relink-script which renames
   them back, beware.

5) rcc does not recognise the volitile keyword.

It now successfully linked raw_kcl but fails when trying to run it
saying DEFVAR is undefined.

Probably the most O/S dependant parts are the fasl loader and the save-system
routines, especially with the .data text appended to the .o object files.
Further there seems to be strange dependancies to code compiled on the
Original KCL(!) and I am not sure exactly what the history of my version is 
and so am not sure which .c files where made by what.
Thus I wanted to try to make a cut down, simple, interpretive
version to try to then isolate the problems, and kcn appeared to be just
such a version.

6) Define unixport/makfile FOR_RAW to be just $(OBJS), added $(EMUL).
   Fix sys_kcn.  It would not make on the Mips/Ricos where the saved_kcl
   has been successfully made ("No Package Named COMPILER").  KCN
   does not seem to have been maintained, maybe we need a fossil direcrory
   for dead things.  Anyway
   I wanted to check that a least the read_fasl_data was Ok so added
   the following prints to sys_kcn after the read_fasl_data:-
	fasl_data = read_fasl_data("../lsp/defmacro.o");
printf("dm fasl read\n");
print(fasl_data, Cnil); /* AJB */
   The printf works fine but the print aborts, illegal type cannot print.
   (file.d/read_fasl_data calls read.d/read_fasl_vector returns 
   alloc_simple_vector, of type object?)  WHY?
   (This is all on the mips.)

7) Give up on KCN and make a fully interpretive version.  sys_boot.c seems to
   be such a version, normally dependant on greping ASSIGN-DOWN in cmpvar.c??
   Anyway I clobbered the making of sys_boot.c from sys_kcl.c in the makefile,
   and hacked sys_boot so that Only .lsp files were init_or_load ed and the
   symbol name (fn) was ignored, and then tryed to make it with only $(OBJS)
   (no LSP or CMP .os).  (Previously it seemed to link in .os for a compiler
   that does not work because of changes to the  representation of structures,
   then override them with .lsp files?!)
   
8) This links but then blows up during the init with '~S is undefined'.
   It turns out that '~S' is our old friend DEFVAR, and the reason is that
   the defmacro.lsp module has Defvars in it even though the Defvar macro
   (defined in evalmacs) could not be defined untill Defmacro was!  I do not
   know how this ever worked but by simply changing the Defvars to 
   si:*Make-Special s all worked fine and I produced a saved_kcl, all be it
   with a fully interpretive compiler (who cares during the boot process?).
   This succeessfully compiles and loads a simple test file. (The 386 fast
   loader is wonderfull compard to the ld stuff on the mips).
   
9) Now do a akcl/make which compiles all the .lsp files and then makes a
   real, fully compiled saved_kcl.  This took many hours (overnight) but
   that was not a problem.


A Plea for new Make Procedures
------------------------------

As an averagely competent Unix hacker with limited time to study Kcl's
internal workings I found the make procedures complicated and obscure.
I never feel confident that I have made a version properly, or that I
could do it again, or that I am starting with the right files.  This
makes me feel uneasy about relying on what seems to otherwise be an
excellent product, which is unfortunate.  I wonder how may other people
simply gave up.  My principle problems are the following:-

1) Strange dependacies on the original KCL, the new system seems to rely
   on files compiled by the old.  This seems to be an attempt to optimise
   the bootstrapping procedure.  Also the links between akcl and kcl are a
   nuisence; cpio does not copy symbolic links properly and we are short of
   disk space and most of the kcl files are merged anyway.

2) The Save-System and dynamic loading code are the two areas that are
   highly system dependant, appart from the relatively simple emul stuff,
   and they certainly would not work for me on an early attempt at a
   mips port.  It should be possible to build a version without either
   of these facillities being mandatory so that one can work on them
   if necessary from a base in which most things work.
   Idealy one could even make an Unknown.defs/.h which used an inefficient
   C based emul and hence make a sort-of version on any machine that had
   a C compiler.

3) I also note that the gbc seems to sweep the C Stack?  Perhaps
   a list of potentially system dependant behavior like this could be made.
   WARNING - an unpredicatble part of the C Stack \is actually in registers
   which could then escape this C Stack Sweep, especially on Risc machines
   with lots of them.  If the C Stack Sweep is unreliable then may be it
   should not be there exept as debuging option, gbc bugs can be extremely
   hard to find!

4) The kcl report, porting guide and other documentation is 
   good but many years out of date.
   For example the discussion about object declarations specifically says
   that the C Stack is not gbced.
   I realise that I am asking a lot from public domain software but
   simply placing "**CHANGED**" markers at appropriate places 
   would be helpful.

5) There seems to be many fossils and unsupported code. There also seems
   to be a bit of copied code, eg. the OBJS list in make_kcn.  Dead things
   should either be left in the original kcl area or flagged in some way,
   perhaps by putting them in a fossils directory.


I think that the following making procedures would at least satisfy items 1 
and 2.

1) Have three kcl trees, AKCL, OKCL (orig) and KCL (the live one).
   OKCL contains only the original sources as typed in by real people,
   no genrated files, and AKCL just merge files to those sources and
   any new sources.
   akcl/make clean simply "rm -r"s kcl;  then akcl/make *copies* and merges
   the sources into the KCL tree.  It is then the KCL tree (only) that is
   normally distributed.
   (This step assumes that people realy are working on the original kcl and
   that a re-sync will eventually be needed, a big job.  If not then maybe
   OKCL could simply be put to rest, with due credit being given to those
   who developed it, of course).

2) Now the make of the KCL Sources could be done as follows:-
   - addefs your version.  (It is painfull that Make STILL does not
     have an #include.  What do AT&Ts 6,000 Unix
     programmers actually do?)
   - Make all the dpp, merge and other utilities.
   - Compile the c/* files and odd unixport .c originals.
   - Link just these to produce an Interp_Very_Raw_Kcl (IRK)
   - Load an IRK init file that loads the lsp/*.lsp (not .c!) files but not
     the compiler and ends without a save-system.
   * We now have a simple, usable KCL, with luck.  It could be saved if that
     is available, else we will simply have to init it every time it is used.

   - Now load an CMP-Load.lsp file that loads and inits the compiler,
     all interpretively.  We now have a fully working KCL, if slow, but
     have not used save-system or dynamic loading.
   - Load compile-lsp and compile-cmp files that compile all the .lsp
     files slowly (who cares, we only do this once) but does not attempt
     to load them.
     A special switch can prevent the appending of the .data files to the
     .o files, another potential source of problems, and the .data files
     themselves can be read during initialization.
   - Exit IRK and (statically) link all these files together, producing
     Compiled-Raw-KCL.
   - Now load the CRK init file, it does not load much else.
   * We now have a compiled, fast KCL, without any heavy O/S dependancies.
     This, for example, could easily ported to VMS.

   - Save-System, if available, else get it and the dynamic loading to work
     if one has time and ability knowing that everything else is basically
     ok.

3) Kcl/Make Clean-Bin deletes all .o and executables, kcl/Make Clean deletes
   everything that was made, inluding lsp/c files etc.

4) Where problems are found with ports change the original to reflect this
   eg.  rename evalmacros and other files to be less than 14 chars,
   add SHELL=/bin/sh everywhere etc.  I
   suspect that my problems relate to not fudging this properly.

These changes could not be difficult to do for someone who understands
the system, I managed to hack my way to a similar result for the 386.
New features and faster programs are great but for users like myself a
clean system that one can feel confident about is far more important.

Regards
	Anthony.