Earlier this week I gave one of the keynote talks at ISSTA, the International Symposium on Software Testing and Analysis. A year ago Matt Dwyer, the general chair, sent me the following invitation:

I would like to invite you to give a keynote talk to the meeting about the challenges in testing, dynamic and static analysis aimed at fault detection for embedded software and particularly sensor network applications. I believe that as sensor network apps continue to mature that new, perhaps domain-specific, V&V techniques will be needed in order to field reliable systems. This topic has received very little attention…

I thought this was pretty cool because it sounds almost exactly like something that I’d have written. The premise of my talk was that there’s a huge amount of interesting research on testing that still needs to be done in the embedded domain, and that — unlike in the past — there are now a number of really nice open-source embedded platforms such as Arduino, TinyOS, Android, and ROS that should provide ready-made audiences for solid tool work. Here are the slides:

Of the ISSTA talks I saw (which unfortunately wasn’t all that many due to the multi-tracked nature of the event and the fact that I had to skip a couple of sessions to get my own talk done) one of the ones I really liked was Philip Guo’s talk about a hacked Python interpreter that makes computations persistent and also transparently memoizes them when this is possible and profitable. The idea is that a lot of people use languages like Python to do data analysis and end up spending a lot of time parsing and printing temporary files, and also end up having a difficult time figuring out which scripts need to be re-run when something changes. Persistence means you don’t have to explicitly write data to files, and memoization means that you can just invoke all of the code every time you want the results, and that you will get a cached result unless something has changed that actually necessitates re-running. Other than manually deciding what to re-run (which is what I do) it’s possible to write a makefile but that is a big pain. His infrastructure takes care of this stuff automatically. I’d use it except for the fact that I do all of this kind of work in Perl. Oh well. Philip is also the person who wrote the excellent CDE packager for Linux applications.

Lionel Briand gave a great talk about some problems with the idea of adaptive random testing (ART), which is a variation of black box fuzzing where test inputs are selected to be “far” from previous test inputs under some distance metric. The hypothesis is that if we look at the space of test inputs under this metric, buggy regions will be somewhat compact. Therefore, we should be trying to spread test cases as widely as possible over the input space. Briand’s paper makes several points but the main criticism is that an ART implementation requires a number of distance comparisons that grows quadratically. Therefore, for realistic choices of testing parameters, a distance check has to be something like 1e5 times cheaper than running a test for ART to have any chance of paying off. The point isn’t that ART is bad, but rather that its proponents had better think of ways to avoid getting killed by the distance checks. In general I love this kind of talk, which takes a critical look at previous work. I feel sure that it caused contention among the program committee and I’m glad they decided to accept it. Even the targets of this work (the people who wrote the previous ART papers) should be flattered. What I mean is that since most academic work goes unread, we should feel lucky if someone criticizes our work because it means they read it and thought about it carefully.

Another talk I liked very much was the other keynote by Laurie Hendren, whose current project is to provide support for the large number of scientists and engineers who do most of their work in Matlab. Matlab is one of those programming languages whose specification is encoded in a single implementation and she entertainingly described the process of reverse-engineering things like how Matlab looks up a name, which should be — but is not at all — simple.

Overall I found ISSTA (which I had never attended) to be a very friendly conference with a lot of smart people and interesting work. Actually, “smart” doesn’t make it stand out since all computer science conferences are mostly smart people. The thing that I liked most was the practical focus on solving real software problems.