benchmarks

[Qobi@zermatt.lcs.mit.edu (Jeffrey Mark Siskind)]

    Date: Thu, 22 Feb 90 19:04 PST
    From: Kalman Reti <Reti@STONY-BROOK.SCRC.Symbolics.COM>

        Date: Thu, 22 Feb 90 14:29 EST
        From: barmar@think.com (Barry Margolin)

        If you can get exclusive use of a system, this is a reasonable way to do
        benchmarks.  However, the TIME macro doesn't know that you've arranged
        this, so it reports the two numbers.

        Traditionally, "CPU time" refers to the amount of time the processor
        spends executing within a given process; in most systems this is
        implemented by the scheduler noting the time when it schedules a process
        to run, noting the time when it next deschedules that process, and
        adding the difference to a total for the process.  On timesharing
        systems this is useful since it factors out the load of the system
        somewhat; a program should take approximately the same amount of CPU
        time no matter how many other users are using the system.  Systems
        differ in whether they include time spent in the kernel in this amount;
        some systems distinguish "system time" and "user time".  Sometimes it's
        useful to disregard system time, because it is more affected by
        activities of other users (for example, if you're timing file access,
        the system time will depend on whether another user had recently
        accessed the same or a nearby file).
    An excellent summary; I'd only like to add that even this this definition
    of CPU time can produce widely varying results because of paging.  For that
    reason the KL-10 computed CPU time by counting memory references (it had
    hardware for that) which resulted in the same 'CPU time' for every execution
    of the same program whether it never encountered a page fault or if it took
    one on every single memory reference (impossible, but you get the idea).

I think that you missed my point. I know what CPU time is and I know why it is
reported and why people use it. I just reiterate my point that personally I
find elapsed wall clock time to be the most appropriate measure of system
performance. I know that it can vary depending on many factors such as system
load, network load, and paging quirks. BUT ALL OF THESE AFFECT HOW LONG I MUST
WAIT TO GET MY RESULT. And that is all that counts to me. Besides, few people
use timesharing systems anymore today. Most people use single user machines
(even if they happen to be Unix boxes which can support timesharing, they are
apportioned one per person and typically run one user task at a time) so most
of the arguments of using CPU time to get load invariant benchmark data are
overweighed by the response time argument.

        On single-user workstations such as the Lisp Machine it is less
        necessary to factor out the affects of other processes, because the user
        is believed to have more control over them than on a timesharing system.
        In Genera you can use WITHOUT-INTERRUPTS or PROCESS:WITHOUT-PREEMPTS to
        hog the processor when you want to factor out other processes, but
        ordinary users on timesharing systems generally can't get exclusive use.
        CPU time is supposed to be close to the time you'd see if yours were the
        only process on the system.  If you just want to know whether 3650 or
        Sparcstation-1 processors are faster at addition this is usually the
        right way to measure it.
    The time spent servicing network interrupts can be significant, especially
    on heavily-used networks, so in addition to WITHOUT-INTERRUPTS-style control,
    I unplug my network.

        There are, of course, times when CPU time is completely worthless.  For
        instance, if you're timing operations that use network facilities or
        back-end processors, you don't want to factor out the time spent waiting
        for the server.  Or if you want to know how well a program performs
        under various system loads you want to measure its real time.