benchmarks

[attila@flash.bellcore.com (Leslie A. Walko)]

>From:  attila@bellcore.com
>Newsgroups:     BCR.DIGEST.SLUG
>Message-ID:  <20156@bellcore.bellcore.com>

Path: slug%ai.sri.com@flash
Subject: benchmarks- what is "CPU" time?


***
QUESTION:  Does anyone know the "standard" "accepted" definition of "CPU
time", "elapsed time" "wall clock time".  Is there a publication which
defines these terms?
***

I am somewhat confused by the difference between CPU time vs Elapsed
time.  [I thought I knew the difference until I as a posting by "Jeff"
on the net.  (excerpt below below)] 

He seems to be using "CPU time" to include the
	time spent by the processor
	plus sequence breaks
	plus I/O time
	plus I/O wait.
He seems to be using "elapsed" as
	all of the above
	plus the time spent to service interrupts.  
This could include any other process such as gc, ethernet, and whatever
else that may be running.

I thought that "CPU time" was time spent by the processor on behalf of a
task, which does not include, gc, other processes, I/O wait, etc.  I
would normally measure this with "without-interrupts" and subtract the
I/O times.  

For example, in bench1:
	
1your wish?0(without-interrupts (time (bench1 100000)))
Evaluation of (BENCH1 100000) took 1.439145 seconds of elapsed time including:
  0.047 seconds processing sequence breaks,
  0.258 seconds in the storage system (including 0.000 seconds waiting for pages):
    0.000 seconds processing 0 page faults including 0 fetches,
    0.258 seconds in creating and destroying pages, and
    0.000 seconds in miscellaneous storage system tasks.
200,002 list words consed in WORKING-STORAGE-AREA.
NIL

The bench1 "CPU time" would be:
	1.439145 - (sequence breaks + storage system) = 1.134145 seconds.
Bench1 "elapsed time" would be
	1.439145 seconds.

My idea of "wall-clock time" is the end-timestamp minus begin-timestamp
of a function where it is not GUARANTEED process exclusivity.  So, I ran
"bench1 again but did NOT wrap the time function in a
without-interrupts.  Here is the result:

1your wish?0(time (bench1 100000))
Evaluation of (BENCH1 100000) took 28.943506 seconds of elapsed time including:
  1.108 seconds processing sequence breaks,
  7.773 seconds in the storage system (including 0.381 seconds waiting for pages):
    0.388 seconds processing 8 page faults including 8 fetches,
    0.617 seconds in creating and destroying pages, and
    6.768 seconds in miscellaneous storage system tasks.
The garbage collector has flipped; so no consing was measured.
NIL

Summary: my
	"CPU" time for bench1 is 1.134145 seconds,
	"elapsed time" is 1.439145 seconds
	"wall-clock time" is 28.943506 seconds. 



So what are the correct definitions? Any published references are
welcome.  

Thank you,
Leslie Walko
attila@bellcore.com

********************************************************************
***                    posted by Jef                             *** 
***  most of the message edited out except for beginning and end  ***
********************************************************************

> I have set up a directory >Qobi>Public> on Zermatt.LCS.MIT.EDU.
> In that directory are several files:
> Benchmark-Styled.lisp --- the original source of the benchmark
>                           this file contains font codes so it can only
>                           be read with Genera
> Benchmark.lisp        --- the source with the font codes stripped
> Benchmark.bin         --- the binary for 36xx series machines
> Benchmark.text        --- the results that I have compiled so far

	[...]

> My experimental procedure was as follows. For each machine/software
configuration

	[...]


> Best of three tries
> Times in seconds rounded to nearest second
> Development is default speed, safety, compilation-speed
> Production is (optimize (speed 3) (safety 0) (compilation-speed 0))
> CPU times are
>   Symbolics: WITHOUT-INTERRUPT times
>   Lucid: Total Run time
>   Franz: cpu time (total) user
> Elapsed times are
>   Symbolics: times with interrupts enabled
>   Lucid: Elapsed Real Time
>   Franz: real time
> 
>                       |            cpu            |          elapsed
> Compilation           | development |  production | development | production
> ----------------------------------------------------------------------------
> 3630                  |    108      |             |    128      |
> XL400                 |    114      |             |    130      |
> NeXT-Franz            |    118      |     98      |    120      |   103
> Sun4/260-Lucid        |             |             |             |
> Sun4/260-Franz        |             |             |             |
> SparcStation1-Lucid   |             |             |             |
> SparcStation1-Franz   |     64      |     53      |     66      |    56
> SparcStation330-Lucid |     10      |     39      |     15      |    42
> SparcStation330-Franz |     50      |     42      |     55      |    44
> 
>                       |            cpu            |          elapsed
> Execution             | development |  production | development | production
> ----------------------------------------------------------------------------
> 3630                  |    283      |             |    336      |
> XL400                 |    187      |             |    227      |
> NeXT-Franz            |   1398      |   1069      |   1448      |  1108
> Sun4/260-Lucid        |             |             |             |
> Sun4/260-Franz        |             |             |             |
> SparcStation1-Lucid   |             |             |             |
> SparcStation1-Franz   |    222      |    147      |    224      |   148
> SparcStation330-Lucid |    132      |     81      |    133      |    81
> SparcStation330-Franz |    168      |    108      |    169      |   108