« A Farewell to Alms | Main | Fix your Mac »

November 13, 2006

New Software, For You!

We released two bits of code a little while ago, both at PLaneT:

• A new minor version of SchemeUnit that adds a check-= form for comparing numbers within a specified tolerance, and fixes a few bugs.
• A new add-on to SchemeUnit, called Benchmark that, as the name suggests, adds forms for benchmarking code.

A simple example of using the Benchmark library:

Suppose you were benchmarking some functions that worked on vectors of numbers, and you wanted to see if the SRFI-43 vector-map was faster than writing a loop by hand. You can test this assumption using the check-faster form:

(module test mzscheme

  (require
   (lib "vector-lib.ss" "srfi" "43")
   (planet "test.ss" ("schematics" "schemeunit.plt" 2))
   (planet "text-ui.ss" ("schematics" "schemeunit.plt" 2))
   (planet "benchmark.ss" ("schematics" "benchmark.plt" 1)))

  (define big-vector
    (vector-unfold
     (lambda (i x) (values i x))
     1000
     0))

  (define check-map
    (test-case
     "Check vector-map is faster than hand-written loop"
     (check-faster
      (lambda ()
        (vector-map - big-vector))
      (lambda ()
        (let loop ([vec (make-vector 1000)]
                   [idx 1000])
          (if (zero? idx)
              vec
              (begin
                (let ([idx (sub1 idx)])
                  (vector-set! vec idx (- (vector-ref big-vector idx)))
                  (loop vec idx)))))))))

  (test/text-ui check-map)
  )

On my computer the hand-written loop is a fraction faster than vector-map, so if performance is essential than the loop is to be preferred.

By formalising assumptions as tests you automatically get notified when implementation changes render them invalid. So if changes in the JIT compiler made vector-map faster this test would fail and I would know to rewrite my performance critical code.

Often it isn't convenient to keep two versions of a function around, perhaps because the implementation depends on many modules. In this case it is useful to benchmark the implementation against its past performance. You can do this by creating a benchmark-case where you would otherwise create a test-case. An example is in order: Say you have a complicated function foo and you want to ensure your optimisations are making it faster. Then you simply write:

(benchmark-case
 "foo benchmark 1"
 (foo some-big-input))

The Benchmark library automatically saves performance data and fails this test if foo becomes slower. The name of the test case is important, as this is what the Benchmark library uses to find historical data.

That's it. As you can see the Benchmark library is quite simple, but I have found it very useful when optimising code. I hope you do as well!

Posted by Noel at November 13, 2006 02:05 PM

Comments