untyped

It’s been a long time coming, but we are proud to announce the launch of the Untyped open source repository!

This public Subversion repository houses the source code for our open source projects, including popular PLT Scheme packages such as Dispatch, Snooze and Mirrors as well as a few things you won’t find on PLaneT.  It’s all free and open source, but please make sure you agree to our terms of use before you get stuck in.

You can check out any or all the projects using command line SVN. For example:

  svn co http://svn.untyped.com/mirrors/trunk mirrors-trunk

Like most people I have a few projects on the go at once. To efficiently switch between them I must be able to quickly pick up where I left off. In my programming projects I’ve been using failing tests as reminders to myself. This fits in nicely with my programming workflow, and enables me to make progress before I’ve recalled all the details of the project I’m working on. Here’s how it works:

In my programming workflow I cycle between writing tests, writing code, and running tests (this is just test driven development). When I’m about to stop working on a project I write some failing tests, which act as a specification for what I should do next. At this point in time I’ve been working on the project for a while so I have recalled its structure and I’m in a good position to make this decision.

When I pick up a project after a break I enter straight into my normal workflow and run my tests. I inspect the failing tests and start implementing the functionality they specify. At this point in time I don’t even have to remember why I’m implementing this; the tests provide enough detail that I can just start coding. As I do so I invariably recall more details of the project. By the time I’ve finished the feature I’m ready to go at full speed.

This technique allows me to “hide” the time it takes to recall the project details; I still get useful work done in this period. It’s quite a simple idea and no doubt some of you are already using it, but if you haven’t tried it, give it a shot.

Further evidence of the increasing commercial relevance of functional programming (and just as important, the desire of the academic community for said relevance) is the announcement that the Journal of Functional Programming is now accepting Commercial Use articles.

The software engineering (SE) community and the programming language theory (PLT) community have traditionally been quite separate. This has led to much duplication of work. For example, it is well known that patterns are language features, indicating that the two communities have essentially been solving the same problems from different angles. The SE field assumes the language is fixed, and so develops abstractions (i.e. patterns) that exist above the level of code. The PLT researcher changes the language to directly express the abstraction. The SE approach is pragmatic but inelegant. The promise of the PLT approach has been retaining elegance (with associated reduction in development and maintenance cost), but the cost of adopting a new language is often seen as too high. (Incidentally, solving this problem is why you want a language with macros. I.e. Scheme.)

That the JFP is looking to publish essentially SE articles is definitely a good thing. It will, however, be interesting to see how the community adapts to members with quite different aims and values to the typical PLT researcher. For example, the PLT researcher is very focused on formality, and in particular giving precise semantics to language constructs. This leads to a style that deals with concepts in a very abstract mathematical manner. While powerful this is certainly not the best presentation for a working programmer to learn from. Consider, for example, Comprehending monads, which aims to be an easy introduction to monads. Given that this paper was published in 1992, we can see its ineffectiveness as a tutorial by considering the zillions of monad tutorials that have been written since. The problem is the abstract style of presentation. While presenting in an abstract style allows you to generalise to many different situations, people work best going from the specific to the general. Better tutorials start with a concrete example, and abstract from there. The question for the PLT community is how they will accommodate the desire for less formality and abstraction from the working programmer when one of the key differentiating factors between SE and PLT is the use of formalisms. If the right balance can be found this will be a great thing for both researchers and practitioners.

SchemeUnit 3 has been released. Although the interface remains compatible with version 2 the underlying philosophy of SchemeUnit has changed in a significant way. The following is extract from the SchemeUnit manual, describing the new approach.

SchemeUnit is designed to allow tests to evolve in step with
the evolution of the program under testing. SchemeUnit
scales from the unstructed checks suitable for simple
programs to the complex structure necessary for large
projects.

Simple programs, such as those in How to Design Programs,
are generally purely functional with no setup required to
obtain a context in which the function may operate.
Therefore the tests for these programs are extremely simple:
the test expressions are single checks, usually for
equality, and there are no dependencies between expressions.
For example, a HtDP student may be writing simple list
functions such as length, and the properties they are
checking are of the form:

(equal? (length null) 0)
(equal? (length '(a)) 1)
(equal? (length '(a b)) 2)

SchemeUnit directly supports this style of testing. A check
on its own is a valid test. So the above examples may be
written in SchemeUnit as:

(check-equal? (length null) 0)
(check-equal? (length '(a)) 1)
(check-equal? (length '(a b)) 2)

Simple programs now get all the benefits of SchemeUnit with
very little overhead.

There are limitations to this style of testing that more
complex programs will expose. For example, there might be
dependencies between expressions, caused by state, so that
it does not make sense to evaluate some expressions if
earlier ones have failed. This type of program needs a way
to group expressions so that a failure in one group causes
evaluation of that group to stop and immediately proceed to
the next group. In SchemeUnit all that is required is to
wrap a test-begin expression around a group of
expressions:

(test-begin
(setup-some-state!)
(check-equal? (foo! 1) 'expected-value-1)
(check-equal? (foo! 2) 'expected-value-2))

Now if any expression within the test-begin

expression fails no further expressions in that group will
be evaluated.

Notice that all the previous tests written in the simple
style are still valid. Introducing grouping is a local
change only. This is a key feature of SchemeUnit’s support
for the evolution of the program.

The programmer may wish to name a group of tests. This is
done using the test-case expression, a simple
variant on test-begin:

(test-case
"The name"
... test expressions ...)

Most programs will stick with this style. However,
programmers writing very complex programs may wish to
maintain separate groups of tests for different parts of the
program, or run their tests in different ways to the normal
SchemeUnit manner (for example, test results may be logged
for the purpose of improving software quality, or they may
be displayed on a website to indicate service quality). For
these programmers it is necessary to delay the execution of
tests so they can processed in the programmer’s chosen
manner. To do this, the programmer simply wraps a test-suite
around their tests:

(test-suite
"Suite name"
(check ...)
(test-begin ...)
(test-case ...))

The tests now change from expressions that are immediately
evaluated to objects that may be programmatically
manipulated. Note again this is a local change. Tests
outside the suite continue to evaluate as before.

2.1 Historical Context

Most testing frameworks, including earlier versions of
SchemeUnit, support only the final form of testing. This is
likely due to the influence of the SUnit testing framework,
which is the ancestor of SchemeUnit and the most widely used
frameworks in Java, .Net, Python, and Ruby, and many other
languages. That this is insufficient for all users is
apparent if one considers the proliferation of “simpler”
testing frameworks in Scheme such as SRFI-78, or the the
practice of beginner programmers. Unfortunately these
simpler methods are inadequate for testing larger
systems. To the best of my knowledge SchemeUnit is the only
testing framework that makes a conscious effort to support
the testing style of all levels of programmer.

PLT Scheme 4.0 is out. We’ve been using the pre-releases for months so this release isn’t particularly significant to us. However, for Universities and other institutions having an official release is important. I do think that too many individual developers stick with out-dated versions of PLT Scheme. The number of questions about 372, which is at least a year old, amazes me. Pre-releases are so much better! Go upgrade!

Over on LtU I was asked how to help beginning programmers become self-directed learners. I have taught a number of students, but not in a context where I’ve been able to really make a difference in their programming practice, so I don’t have an answer to the whole question (though my instinct is that the apprenticeship method is the right way to go). However I try to teach one process that I think is an essential step towards becoming a self-directed learner. That process is the big idea called science, and I’m not talking about lab coats and chunky glasses

When working with students I always get asked what the result of evaluating some piece of code will be. What I tell the students is to ask the computer via a test case or the REPL. Testing ideas by experimentation is science in its simplest and most immediate form, and a crucial step in developing the student’s ability to solve their own problems.

Application of science to programming is not restricted to students; test-driven development is science. So what is science then? I simplify, but basically three things: a theory to test, an experiment to test it, and a standard of proof (note we can never truly prove a theory, just simply not be able to disprove it). This is exactly when a unit test is. For example, a Scheme programmer might pose the theory “the string->number function will convert strings padded with whitespace characters to numbers”, formulate the experiment (equal? (string->number “ 200”) 200), and have as the standard of proof the boolean output of this single experiment.

When most people think of the scientific method they think of the lengthy and expensive double-blind trials used in, for example, medical trials. A really important point is to realise that when you do science, you choose the standard of proof. For example, as most computer programs are deterministic a few tests can be sufficient to show a property holds. When dealing with a concurrent program, or some other non-deterministic system, you may need to be more rigourous.

So there we have it. All programmers are scientists to some extent, though they might not know it. We can extend the use of experimentation to answer other questions, such as determining if productivity is affected by changes to software development process. Doing this in a lightweight way is the intention of the Simple Improvement package, though I haven’t had the time to get that library to a really useful state. Perhaps in a later post I’ll go through the ideas behind it. In the mean time, get experimenting (lab coats optional).

If you do any macro programming in PLT Scheme you are sure to run into the dreaded “no #%app syntax transformer is bound” error message at some point. Though puzzling, the fix is actually quite simple in almost all cases. Assuming you’re using 3.99, you either need to:

1. (require (for-syntax scheme/base))
2. (require (for-template scheme/base))

What the error means is that some syntax has expanded in a function application, but #%app, the PLT Scheme primitive that actually handles application, is not bound in the phase in which the syntax is being evaluated. Requiring for-syntax will bind #%app in the phase before the current evaluation phase, while requiring for-template will bind #%app in the phase after. In most cases you want for-syntax. However, if you are writing functions that return syntax that is then inserted into a program (such a function would be required for-syntax elsewhere) you must use the other form, to make sure the syntax has #%app available to it.

At last — the long awaited ICFP post! In summary: ICFP was awesome. Freiburg is lovely, the German beer is fantastic, and everyone at the conference was very friendly. It was great to put faces to people we’ve conversed with for years, to meet old friends, and to make new ones.

We gave two talks, one at ICFP on our work building web sites in Scheme (paper here), and one at CUFP (PLT Slideshow slides PDF slides). Both were, I think, well received: a few people expressed some interest in having us come and talk to their groups, and the CUFP organisers invited us to join them at dinner.

There was a definite buzz about ICFP. It seems functional languages are beginning to take off — CUFP doubled its attendence over last year, and ICFP strained the capacity of the hotel. There was something of a reality distortion field in place though. After a few days at the conference you could begin to believe the entire software market consisted of either program verification tools in Haskell or telephony apps in Erlang. There was little representation from web developers, who I think must make up the largest group of commercial developers. I believe this is because Haskell users really dominate ICFP, and Haskell doesn’t have a particularly good web development story as far as I know.

It was interesting to see how the other communities are developing. The Haskell guys had a 3-day Hackathon right after ICFP, which is pretty impressive, and there is a practical Haskell book in development, something which is needed for Scheme. Erlang seemed to have slightly better industry representation and also has several recent practically-oriented publications. I heard that many people had arrived just for the Erlang workshop, which was held the day after CUFP.

Of course the conference revolved around the paper presentations. There were too many to review them all, so I’ll just note the ones that were particularly relevant to our work at Untyped.

Matthew Flatt’s talk on Adding Delimited and Composable Control to a Production Programming Environment was a great presentation on a new feature in PLT Scheme, delimited continuations, that will be very useful in the web server. Matthew hacked Slideshow (something you can do when you’re the core developer) to support animations by quickly fading between slides. His 1028 slides made for some slick animations that quickly and clearly got across the concept of delimited continuations. This was perhaps the best presentation I saw at the conference and it was on something we’ll definitely be using.

The iData toolkit is a Clean library that uses meta-programming to generate code for viewing and editing arbitrary data online (like Ruby on Rail’s scaffolding, but better). At ICFP this year the followup, iTasks: Executable Specifications of Interactive Work Flow Systems for the Web, was presented. Essentially it is a combinator library for specifying workflows, including higher-order workflows. At is happens we may soon be involved in a project that deals with workflows, in which case we’ll review this work.

I really liked Advanced Macrology and the Implementation of Typed Scheme by Ryan Culpepper, Sam Tobin-Hochstadt, and Matthew Flatt. Typed Scheme is pretty cool, and we’ll probably use it when it has matured a bit more, but my favourite bit of this paper is the first half which is essentially a tutorial on intermediate to advanced macrology. There is precious little material available on this corner of Scheme, so it is very welcome addition.

Also of particular interest to us were Applications of Fold to XML Transformation, and Software Transactions Meet First-Class Continuations. We’ve already had occasion to use ideas from the former, while the later gave us some food for thought with regards to Snooze We had an interesting conversation with Adam Wingo, author of the paper on folds, about the advantages of distributed version control. Something we need to look into. Adam also has a great job that allows him to spend two days a week hanging out in Barcelona’s cafes. Some people get all the luck.

One point from ICFP that is particularly relevant for this blog: Dave Herman told me he’d like to see more technical posts. I’ve tried to make the content a bit more technical of late, but if there is anything in particular you’d like me to write about drop me a line.