Epsilons update (long)

[MILLER@VAX.CAM.NIST.GOV]

Mabry and Barmar suggested that my mail problem is due to the headers being munged
up and Zmail doesnt see the character style info.
A couple of other people said they are having similar problems and would I please
post a solution (if I find one).  As this problem has been around for quite awhile,
I'll try to update on what I'm learning for the benefit of everyone (hardcore net-experts
please bear with my naivete!)

I've sorted out a couple of things by experimentation, detective work & attempting to
understand RFC#822 (regarding mail headers). Some background: the message comes in two parts:
the headers and the message text. The headers is a sequence of headers ended by
a blank line (!). Each header consists of a single `logical' line (it can be more 
than one line if the continuation starts with whitespace); a field-name with a `:' and
the field info. Field names can be the obvious things like Date:, From:, 
Subject:, To:, etc or can be more esoteric things like Received:, Message-Id: 
Character-Type-Mappings:, Fonts:, etc.  Apparently most fields are optional,
it is allowable for fields to be repeated (although RFC#822 doesn't specify 
what merging is implied), and the fields dont have to be in a particular order.

One thing: When you are reading a message, ZMAIL hides all of the esoteric header
lines; PROVIDING it understands the header! I never realized this because
my headers have ALWAYS been munged!  So, if you see fields like Received:, you 
already know there's a problem. [If you edit the message ZMAIL reveals all 
of the hidden headers]

Back to the point; I give my email address as my address on our VAX.
I have the vax forward to my lispm where I also read my mail. (I am NOT reading
my mail from a VAX formatted mail file).  Now comes the problem;
The (complete) header info that I am seeing in my mail looks like (my comments added):

  Recieved: from <vax> by <lispm>        ; obviously added by <lispm>
  Date: <date>				 ; Who added the next four lines?
  From: <email address>
  Subject: <subject>			 ; sometimes the real subject, sometimes not.
  To:  <me>
  <BLANK LINE>                           ;  !!!!!!
  <More header lines obviously original + description of path ...>

Sometimes, for example internet mail, the following is inserted after the blank line:
  Received: from <somewhere> by <vax>    ; obviously added by <vax>
  <BLANK LINE>				 ;  !!!!

If any font info is there, it is included in the original header lines which are
not visible to Zmail because of the blank line(s)!
Another oddity: If I edit out the blank lines within zmail, it seems to 
understand more of the header and digest the esoteric fields. However, the
character-type-mappings & fonts headers disappear!

The question is reduced to "Who added the extra four lines, and most important,
who added the Blank line?"   

The differences between the system clocks of various machines & discontinuities in
various times printed various fields, and the format of the From: address field
led me to suspect that these lines were generated on the lispm side. 
Indeed (DEFMETHOD (:SERVER-TOP-LEVEL DNA-MAIL-SERVER) in SYS:DNA;DNA-MAIL
is the culprit! It extracts to,from & subject from the input
(presumably these fields are represented specially by dna?)
and stuffs these into a `header-stream' as header date,from,subject & to and 
then a blank line!

The remaining input will be copied out to the mail file after the header lines 
are copied out. For a `simple' message, the input pointer is pointing directly
at the message body, so the newline IS needed. If the remaining input contains
additional header lines, however, the new line should NOT be output.
Actually, assumming distinction between header-stream and body-stream is important,
the remaining header lines should be copied to the header-stream, and THEN a
new line added; the body-stream would then contain only the message-body.

One problem with implementing this is that there would seem to be some ambiguity
in distinguishing whether the nest line of input really was intended to be header; 
At this point the input is NOT required to have any header lines and
you COULD have the first line of your message containing a word (w/o whitespace) 
followed by a colon!  
The DNA implementors were either unaware of the problem of additional headers,
or just took the easy way out.  In any case symbolics doesnt seem to want to fix
it soon.

There are a couple of things messy about the fix, even assumming you ignore the
potential ambiguity; you've got to peek at the input lines, you need to read
logical lines rather than simply CR terminated lines, and so forth.
Basically it needs a call to something like
      (MAYBE-COPY-ADDITIONAL-HEADER-LINES STREAM HEADER-STREAM)
inserted just after the 
      (SEND HEADER-STREAM :ADD-HEADER-LINE (STRING #\RETURN))
in the aforementioned method. 
The above functions would presumably be easier to write if we had access to some 
of the header-line parsers in zmail (why is this software off-limits?)
If anybody out there is familiar with this stuff and is willing to fix it,
please let me know, otherwise I'll give it a shot (w/o guarantees!).

Bruce
MILLER@VAX.CAM.NIST.GOV