MSPS-game (Re: PCL benchmark)

[Masayuki Ida <ida%cc.aoyama.junet@UTOKYO-RELAY.CSNet>]

Two years ago, when I was interested in the measurement of PCL,
I got an idea to try to have an index to measure the PCL performance.
I wrote a code and tested with several implementations as a coffee
break entertainment.

I called it MSPS for 'Message Send Per Second'.
The recent discussion of this mail box triggered me to remember MSPS-game
of two years ago.

Benchmarking always assumes a model and my model for MSPS was
 25% classical method with two arguments, 25% muti-method,
 50% default arguments.
and 50% inherited access, 50% non-inherited access.
This assumption is not 'scientific' but from my intuition at that
time.
If we can have more realistic model, we may have a common measure for
generic function call overhead.

Here is the source for it.

Masayuki Ida
Aoyama Gakuin Univ. Japan

----------------- cut here ----------------

;;; -*- Syntax: Common-Lisp; Package: USER; Base: 10; Mode: LISP -*-
;;;
;;; a simple test program 'yattegoran' for PCL/CLOS
;;; this program gives us a measure for generic function call performace
;;;   in MSPS (Message Send Per Second)
;;;
;;;			1986.12.10 Masayuki Ida, Aoyama Gakuin University
;;;                     1988.11.25 Masayuki Ida  updated to comform CLOS
;;;
;;;
;;;  Features tested here are
;;;    Single inheritance, discrimination with multiple arguments,
;;;    Instance creation, built-in class handling
;;;
;;; several data
;;;  PCL on KCL (Ultrix 8600) in 1986. interpreted!!!
;;;     with CommonLoops	real 24.417 sec	CPU 4.783 sec
;;;     without message send	real 0.250 sec	CPU 0.100 sec
;;;          - )____________________________________________
;;;					 -	    4.683 sec for 200 message send
;;;  ===> 200/4.683 = 42.7 Message Send Per Second.
;;;                   ----
;;;  PCL on Allegro CL (SUN-4 260) in 1988. compiled!!!
;;;     with PCL       CPU (total user time) 250 msec  real time: 440 msec
;;;     without PCL    CPU (total user time)   0 msec  real time:   0 msec
;;;          -)______________________________________
;;;                                          250 msec for 200 message send
;;;  ===> 200/0.250 = 800.0 Message Send Per Second.
;;;                   -----               (interpreted version is 632 MSPS)
;;;
;;; The above data DOES NOT show the performance comparison of different CL
implementation,
;;; since 1)machines used are quite different,
;;;   2)the source codes are slightly different(86 version uses ndefstruct
and defmeth...)
;;;   3)Allegro version is compiled and mesured, but not for KCL-case,
;;;   4)there are two year time difference to progress.
;;; ( Rather it shows the advances of PCL implementation techniques in some
extent.)

;;; --------------- The Model ----------------------------
;;; This value reflects the following assumption:
;;;   1) 25% classical method with two arguments, 50% with default
arguments
;;;   2) 25% multi-method
;;;   3) 50% non inherited access, 50% inherited access
;;; -------------------------------------------------------

(defvar *x*)

(defclass foo () (x))
(defclass bar (foo)())

(defmethod yattegoran ((x foo) (y number))
  (if (= y 0)
      1 
      (yattegoran (setq *x* (make-instance 'bar))
		  (yattegoran y nil))))
(defmethod yattegoran ((x bar) y)
  (if (= y 0)
      1
      (yattegoran (setq *x* (make-instance 'foo))
		  (yattegoran y nil))))
(defmethod yattegoran (x y)
  (1- x))

(time (yattegoran (make-instance 'foo) 100))	; ====> (1)

(defun yatte1 (x y)
  (if (= y 0)
      1
      (yatte2 x (yatte3 y nil))))
(defun yatte2 (x y)
  (if (= y 0)
      1
      (yatte1 x (yatte3 y nil))))
(defun yatte3 (x y)
  (1- x))

(time (yatte1 nil 100))			; ====> (2)

;; get the difference of the CPU times in (1) and (2). (for compiled codes,
(2) is negligible)
;; divide 200 by it. (CAUTION: exactly speaking, yattegoran-foo-number is
called 51 times,
;; yattegoran-bar-t called 50 times,yattegoran-t-t called 100 times. so the
sum itself is 201)
;;
;; PCL(St.Patrick day version) on Allegro CL 3.0.1 on SUN-4 '88 Fall has
800MSPS
;;  which TRY to mean the implementation can do 800 generic function calls
per second.
;; Comment: the reason why I inserted meaningless setq to *x* is to provide
a chance
;;        to measure slot access things in the future. If
method-dispatching is the
;;        only target to measure, dropping setqs gives slightly better
value.
;; The possible target: MSPS value should be more closer to usual function
call.
;;     Say, 10K MSPS with FULL function CLOS is possible ?