What’s Fun About Teaching

When I started this blog I expected to write a lot about teaching. In retrospect it seems that teaching is similar to raising kids and cooking meals in the sense that these are jobs to just shut up and do, as opposed to writing a lot about them. Even so, I have a short series of pieces about teaching queued up. Today’s topic is the fun stuff.

  • The most basic enjoyable thing about teaching is watching students catch on to difficult concepts and put pieces together into a better picture of how things work.
  • Closely related, it’s great to see students emerging from a course more excited about the material than when they entered it. (Of course, anyone who leaves a course less excited about the material has also learned something valuable. When I was in college I often thought of it as “120 credit hours of things I don’t want to do for the rest of my life.”)
  • Preparing lectures provides an excuse to learn a topic in a bit more depth than I’d previously known it. In particular, it’s almost impossible to give a good lecture based only on a surface-level understanding of the material. Rather, it is necessary to know why certain approaches are the right ones. Since this kind of information is not often written down, figuring it out requires a lot of reading between the lines, and this is fun.
  • Another excellent kind of intellectual puzzle comes from trying to figure out what is going on in a student’s mind well enough that whatever is wrong with their mental picture can be corrected. When I fail to solve this kind of puzzle, I basically end up repeating the facts over and over, hoping something will sink in.
  • Lectures, like other performances, are fun to do well. Nailing a lecture is more than failing to bungle a transition or be confused by an intricate algorithm — a great lecture has a solid focus, a clean motivation, a logical progression through the material, and a punchline where some significant new understanding of the material is reached.
  • It feels good to create course material that fills a serious gap in the curriculum.
  • Perfecting a collection of course material over time is nice. Even when a collection of course material appears at first to be extremely solid, I find that there are always lectures that want to be moved around, details that should be elaborated or eliminated, arguments that need to be refined, etc.

Does a Simulation Really Need to Be Run?

At some point we’ll be able to run a computer simulation that contains self-aware entities. In this piece I’m not going to worry about little details such as how to tell if a simulated entity is self-aware or whether it’s even possible to run such a simulation. The goal, rather, is to look into some philosophical problems posed by simulations.

A computer simulation takes a model in some initial configuration and evolves the state of the model through repeated programmatic application of rules. Usually we run a simulation in order to better understand the dynamics of some process that is hard to study analytically or experimentally. The straightforward way to implement a simulator is to represent the system state in some explicit fashion, and to explicitly run the rule set on every element of the state at every time step. It seems clear that if our simulation includes a self-aware entity, this sort of execution is sufficient to breathe life into the entity. But what about other implementation options?

First, our simulator might be mechanically optimized by a compiler or similar tool that would combine rules or elide rules in certain situations. For example, if the simulation state contains a large area of empty cells, the optimizer might be able to avoid running the rule set at all in that part of the space. Can the entity being simulated “feel” or otherwise notice the compiler optimizations? No — as long as the compiler is correct, its transformations are semantics preserving: they do not affect the computation being performed. Of course a suitable definition of “do not effect” has to be formulated; typically it involves defining a set of externally visible program behaviors such as interactions with the operating system and I/O devices.

(animation is from Wikipedia)

Compiler optimizations, however, are not the end of the story — algorithmic optimizations can be much more aggressive. I’ll use Conway’s Game of Life as the example. First a bit of background: Life is a two-state, rectangular cellular automaton governed by these rules (here I’m quoting from Wikipedia):

  • Any live cell with fewer than two live neighbors dies, as if caused by under-population.
  • Any live cell with two or three live neighbors lives on to the next generation.
  • Any live cell with more than three live neighbors dies, as if by overcrowding.
  • Any dead cell with exactly three live neighbors becomes a live cell, as if by reproduction.

Life has been shown to be Turing complete, so clearly self-aware entities can be encoded in a Life configuration if they can be algorithmically simulated at all. A straightforward implementation of Life maintains two copies of the Life grid, using one bit per cell; at every step the rules are applied to every cell of the old grid, with the results being placed into the new grid. At this point the old and new grids are swapped and the simulation continues.

Hashlife is a clever optimization that treats the Life grid as a hierarchy of quadtrees. By observing that the maximum speed of signal propagation in a Life configuration is one cell per step, it becomes possible to evolve squares of the Life grid multiple steps into the future using hash codes. Hashlife is amazing to watch: it starts out slow but as the hashtable fills up, it suddenly “explodes” into exponential progress. I recommend Golly. Hashlife is one of my ten all-time favorite algorithms.

The weird thing about Hashlife is that time progresses at different rates at different parts of the grid. In fact, two cells that are separated by distance n can be up to n time steps apart. Another interesting thing is that chunks of the grid may evolve forward by many steps without the intermediate steps being computed explicitly. Again we ask: can self-aware Life entities “feel” the optimization? It would seem that they cannot: since the rules of their universe are not being changed, their subjective experience cannot change. (The Hashlife example is from Hans Moravec’s Mind Children.)

If Hashlife is a viable technology for simulating self-aware entities, can we extend its logic to simply replace the initial simulation state with the final state, in cases where we can predict the final state? This would be possible for simulated universes that provably wind down to a known steady state, for example due to some analogue of the second law of thermodynamics. The difference between Hashlife and “single simulation step to heat death” is only a matter of degree. Does this fact invalidate Hashlife as a suitable algorithm for simulating self-aware creatures, or does it imply that we don’t actually need to run simulations? Perhaps to make a simulation “real” it is only necessary to set up its initial conditions. (Greg Egan’s Permutation City is about this idea.)

Aside: Why don’t we have processor simulators based on Hashlife-like ideas? Although ISA-level simulators do not have the kind of maximum speed of signal propagation seen in cellular automata, the programs they run do have strong spatial and temporal locality, and I bet it’s exploitable in some cases. I’ve spent long hours waiting for simulations of large collections of simple processors, daydreaming about all of the stupid redundancy that could be eliminated by a memoized algorithm.

(image is from here)

Techniques like Hashlife only go so far; to get more speedup we’ll need parallelism. In this scenario, the simulation grid is partitioned across processors, with inter-processor communication being required at the boundaries. An unfortunate thing about a straightforward implementation of this kind of simulation is that processors execute basically in lock step: at least at the fringes of the grid, no processor can be very far ahead of its neighbors. The synchronization required to keep processors in lock step typically causes slowdown. Another ingenious simulation speedup (developed, as it happens, around the same time as Hashlife) is Time Warp, which relaxes the synchronization requirements, permitting a processor to run well ahead of its neighbors. This opens up the possibility that a processor will at some point receive a message that violates causality: it needs to be executed in the past. Clearly this is a problem. The solution is to roll back the simulation state to the time of the message and resume execution from there. If rollbacks are infrequent, overall performance may increase due to improved asynchrony. This is a form of optimistic concurrency and it can be shown to preserve the meaning of a simulation in the sense that the Time Warp implementation must always return the same final answer as the non-optimistic implementation.

Time Warp places no inherent limit on the amount of speculative execution that may occur — it is possible that years of simulation time would need to be rolled back by the arrival of some long-delayed message. Now we have the possibility that a painful injury done to a simulated entity will happen two or more times due to poorly-timed rollbacks. Even weirder, an entity might be dead for some time before being brought back to life by a rollback. Depending on the content of the message from the past, it might not even die during re-execution. Do we care about this? Is speculative execution amoral? If we suppress time warping due to moral considerations, must we also turn off processor-level speculative execution? What if all of human history is a speculative mistake that will be wiped out when the giant processor in the sky rolls back to some prehistoric time in order to take an Earth-sterilizing gamma ray burst into account? Or perhaps, on the other hand, these speculative executions don’t lead to “real experiences” for the simulated entities. But why not?

The Hashlife and Time Warp ideas, pushed just a little, lead to very odd implications for the subjective experience of simulated beings. In question form:

First, what kind of simulation is required to generate self-awareness? Does a straightforward simulator with an explicit grid and explicit rule applications do it? Does Hashlife? Is self-awareness generated by taking the entire simulation from initial state to heat death in a single step? Second, what is the effect of speculative execution on self-aware entities? Do speculative birth and death, pain and love count as true experiences, or are they somehow invalid?

These questions are difficult — or silly — enough that I have to conclude that there’s something fundamentally fishy about the entire idea of simulating self aware entities. (Update: Someone told me about the Sorites paradox, which captures the problem here nicely.) Let’s look at a few possibilities for what might have gone wrong:

  1. The concept of self-awareness is ill-defined or nonsensical at a technical level.
  2. It is not possible to simulate self-aware entities because they rely on quantum effects that are beyond our abilities to simulate.
  3. It is not possible to simulate self-aware entities because self-awareness comes from souls that are handed out by God.
  4. We lack programming languages and compilers that consider self-awareness to be a legitimate side-effect that must not be optimized away.
  5. With respect to a simulation large enough to contain self-aware entities, effects due to state hashing and speculation are microscopic — much like quantum effects are to us — and their effects are necessarily insignificant at the macroscopic level.
  6. All mathematically describable systems already exist in a physical sense (see Max Tegmark’s The Mathematical Universe — the source of the title of this piece) and therefore the fire, as it were, has already been breathed into all possible world-describing equations. Thus, while simulations give us windows into these other worlds, they have no bearing on the subjective experiences of the entities in those worlds.

The last possibility is perhaps a bit radical but it’s the one that I prefer: first because I don’t buy any of the others, and second because it avoids the problem of figuring out why the equations governing our own universe are special — by declaring that they are not. It also makes simulation arguments moot. One interesting feature of the mathematical universe is that even the very strange universes, such as those corresponding to simulations where we inject burning bushes and whatnot, have a true physical existence. However, the physical laws governing these would seem to be seriously baroque and therefore (assuming that the multiverse discourages complexity in some appropriate fashion) these universes are fundamentally less common than those with more tractable laws.

Why Verify Software?

People like me who work on software verification (I’m using the term broadly to encompass static analysis, model checking, and traditional formal verification, among others) like to give talks where we show pictures of exploding rockets, stalled vehicles, inoperable robots, and crashed medical devices. We imply that our work is helping, or at least could help, prevent very serious software-related problems. It’s not clear that this sort of claim stands up to a close examination.

What would it take to show a causal link between verification efforts and software safety? A direct demonstration using a controlled experiment would be expensive. An indirect argument would need several parts. First, we’d have to show that flaws revealed by verification efforts are of the kind that could compromise safety. Second, we’d need to show that these flaws would not have been found prior to deployment by traditional V&V — those bugs are merely expensive, not harmful. Third, we’d need to argue that a given dollar spent on verification adds more safety than that same dollar spent on other ways of improving software. Finally, we would need to argue that a successful verification effort won’t have unintended consequences such as emboldening management to substantially accelerate the schedule.

Of course, none of this criticizes software verification research, which I work on and very much believe in. We simply need to be clear about its purpose, which is to reduce overall cost. A top-level goal for software verification that fails to mention cost (for example “minimize damage caused by bugs in software intensive systems”) is untenable because obviously the best way to minimize such damage is to radically simplify, or even eliminate, the software. Of course, in practice we do not wish to radically simplify or eliminate the software because it brings so many benefits.

A more reasonable high-level goal for software verification might be “increase, to the largest possible extent given the methods available, total system utility.” “Total system utility” has both positive and negative components, and verification is mainly about mitigating some of the negative components, or costs, including not just development and manufacturing costs, but also maintenance and accident-related costs. In the next couple of days I’ll post a more specific example where the cost-based analysis of verification is much more useful than the feel-good analysis promoted by exploding rocket pictures.

Do Small-RAM Devices Have a Future?

Products built using microcontroller units (MCUs) often need to be small, cheap, and low-power. Since off-chip RAM eats dollars, power, and board space, most MCUs execute entirely out of on-chip RAM and flash, and in many cases don’t have an external memory bus at all. This piece is about small-RAM microcontrollers, by which I roughly mean parts that use only on-chip RAM and that cannot run a general-purpose operating system.

Although many small-RAM microcontrollers are based on antiquated architectures like Z80, 8051, PIC, and HCS12, the landscape is changing rapidly. More capable, compiler-friendly parts such as those based on ARM’s Cortex M3 now cost less than $1 and these are replacing old-style MCUs in some new designs. It is clear that this trend will continue: future MCUs will be faster, more tool-friendly, and have more storage for a given power and/or dollar budget. Today’s questions are:

Where does this trend end up? Will we always be programming devices with KB of RAM or will they disappear in 15, 30, or 45 years?

I’m generally interested in the answer to these questions because I like to think about the future of computing. I’m also specifically interested because I’ve done a few research projects (e.g. this and this and this) where the goal is to make life easier for people writing code for small-RAM MCUs. I don’t want to continue doing this kind of work if these devices have no long-term future.

Yet another reason to be interested in the future of on-chip RAM size is that the amount of RAM on a chip is perhaps the most important factor in determining what sort of software will run. Some interesting inflection points in the RAM spectrum are:

  • too small to target with a C compiler (< 16 bytes)
  • too small to run multiple threads (< 128 bytes)
  • too small to run a garbage collected language (< 128 KB)
  • too small to run a stripped-down general-purpose OS such as ╬╝Clinux (< 1 MB)
  • too small to run a limited configuration of a full-fledged OS (< 32 MB)

These numbers are rough. It’s interesting that they span six orders of magnitude — a much wider range of RAM sizes than is seen in desktops, laptops, and servers.

So, what’s going to happen to small-RAM chips? There seem to be several possibilities.

Scenario 1: The incremental costs of adding transistors (in terms of fabrication, effect on packaging, power, etc.) eventually become so low that small-RAM devices disappear. In this future, even the tiniest 8-pin package contains an MCU with many MB of RAM and is capable of supporting a real OS and applications written in PHP or Java or whatever. This future seems to correspond to Vinge’s A Deepness in the Sky, where the smallest computers, the Qeng Ho localizers, are “scarcely more powerful than a Dawn Age computer.”

Scenario 2: Small-RAM devices continue to exist but they become so deeply embedded and special-purpose that they play a role similar to that played by 4-bit MCUs today. In other words — neglecting a very limited number of specialized developers — they disappear from sight. This scenario ends up feeling very similar to the first.

Scenario 3: Small-RAM devices continue to exist into the indefinite future; they just keep getting smaller, cheaper, and lower-power until genuine physical limits are reached. Eventually the small-RAM processor is implemented using nanotechnology and it supports applications such as machines that roam around our bloodstreams, or even inside our cells, fixing things that go wrong there. As an aside, I’ve picked up a few books on nanotechnology to help understand this scenario. None has been very satisfying, and certainly none has gone into the kind of detail I want to see about the computational elements of nanotechnology. So far the best resource I’ve found is Chapter 10 of Freitas’s Nanomedicine Volume 1.

This third scenario is, I think, the most interesting case, not only because small-RAM devices are lovable, but also because any distant future in which they exist is pretty interesting. They will be very small and very numerous — bear in mind that we already manufacture more MCUs per year than there are people on Earth. What sensors and actuators will these devices be connected to? What will their peripherals and processing units look like? How will they communicate with each other and with the global network? How will we orchestrate their activities?

Embrace WTF

Most people who do quantitative work, especially involving computers, mutter “What the fuck?” or something similar pretty often. Lately I’ve been thinking about WTF in more detail. WTF is good because it stems from dawning recognition of one’s own ignorance, and without recognizing ignorance we cannot eliminate it. Here I am only discussing serious WTF, not the kind of casual WTF that occurs when the mouse pointer gets lost on another screen or the elevator fails to arrive rapidly. WTF comes is several flavors.

Neutral WTF

This is by far the most common and it’s best described by example. A few years ago I wanted to compute confidence intervals for low-probability events. For example, if I sample a bag of marbles 10,000 times and get 9,999 red marbles and one blue marble, then what is the 95% confidence interval for the proportion of blue marbles in the bag? After a bit of background reading I settled on the Wilson interval as meeting my needs and wrote a bit of Perl code implementing it. Since I’m professionally paranoid I also wrote a simple Monte Carlo simulation to check my confidence interval code: it simply repeated the marble sampling experiment many times and computed the number of times that actual probability was contained in the confidence interval. Obviously, by definition, this should happen 95% of the time — but it didn’t. The actual probability was contained in the confidence interval about 92% of the time. The most obvious possibility was a bug in my code but double and triple checking failed to turn up any problems, leading to a nice solid WTF. More exotic possibilities like a systematic flaw in Perl’s random number generator or FP-related numerical issues seemed remote. Eventually I did some more reading and found out that the phenomenon I was observing is well-known; I won’t attempt to explain it but Interval Estimation for a Binomial Proportion contains an excellent analysis.

Neutral WTF comes from the routine kind of ignorance that plagues all of us, because there is so much to know. We’d rather it didn’t happen, but it’s unavoidable when doing any remotely interesting work.

Negative WTF

No need to say much about this one. There are many sub-species of negative WTF but for me it generally happens when some research idea is generating really nice-looking results. Eventually something stops making sense and we get WTF that is quickly followed by gnashing of teeth because there’s a major flaw somewhere. Worse, this sometimes happens after I’ve started telling people how great the idea is.

Negative WTF comes from a harmful kind of ignorance — the ignorance that we were doing something important wrong.

Positive WTF

Positive WTF is elusive and excellent; it occurs when we begin to recognize that we don’t understand something, and finally discover that nobody else understands it either.

My personal favorite example of positive WTF happened four or so years ago after an undergrad pointed out that some assembly code in a lecture for my advanced embedded systems course was wrong. I tracked the error down to a compiler mis-translating a piece of C code containing the volatile qualifier. A while later I became curious about this and wrote a collection of test cases for accesses to volatile-qualified objects and ran it through a number of compilers. When none of the tools translated all of the examples correctly I experienced WTF. Could there really be systematic errors in translating a construct that essentially all operating systems and embedded systems rely on? I talked to people and investigated more and this turned out to be the case. The result was a paper that is still heavily downloaded from companies who make and use compilers for embedded systems.

My guess is that a lot of great scientific discoveries were preceded by profound WTF. For example, it sounds like that is what happened to Penzias and Wilson.

The broad power of positive WTF crystallized for me recently when I read The Big Short, a pretty awesome book about some people who not only foresaw the self-destruction of the sub-prime home loan market, but also placed big bets against the junk bonds created from these loans. Lewis makes a big deal about the fact that the people placing these bets were in a constant state of WTF: Why is nobody seeing what we’re seeing? Why is nobody doing what we’re doing? Before reading this book I’d have characterized WTF as mainly occurring in math, science, and engineering; it was great to get an extended anecdote from the financial world.

Positive WTF comes from the best possible kind of ignorance: ignorance that, once its shape is known, motivates new discoveries. Disciplined thinking is required to avoid succumbing to the natural temptation to sweep the WTF-inducing issues under the rug. We must embrace WTF because it is a leading indicator that we don’t understand something crucial.

Safe From Compiler Bugs?

A few people have asked me: Does there exist a subset of the C language that is not, in practice, miscompiled? The intuition behind the question is perfectly reasonable. First, it is clear that there exist C features, such as bitfields and volatile variables, whose compiler support is not so reliable. Second, there exist C subsets like MISRA C that are intended for safety critical application development and we would hope they can be reliably compiled.

There probably do exist subsets of C that avoid compiler bugs. For example, if we avoid all math, control flow, and I/O, then it’s at least conceivable that we’re on safe ground. If not, then almost certainly we’d be OK by permitting only “return 0;” in function bodies. However, my group’s experience in reporting compiler bugs has convinced me that there is no remotely useful subset of C that reliably avoids compiler bugs. Let’s take this C subset as an example:

  • only variables of type int (or unsigned int, if that seems better)
  • no dynamic memory allocation
  • only functions of 30 lines or less
  • only if/else statements, function calls, and simple for loops for flow control (no break, continue, do/while, goto, longjmp, etc.)
  • only single-level structs, single-level pointers, and one-dimensional arrays
  • no type qualifiers
  • all expressions are in three-address form; for example “x = y + z;”

We could add more restrictions, but that would seem to make it hard to get work done in the subset. Of course, to make the subset real we’d need to nail down a lot of additional details and also write a checking tool (neither activity would be difficult).

Next we ask the question: what percentage of the compiler’s optimization passes will have useful work to do when compiling code written in this subset? The answer, unfortunately, is “most of them.” For an aggressive compiler, this amounts to a lot of extremely subtle code. In practice, some of it will be wrong. Thus, I am claiming that even a fairly severe C subset provides little shelter from compiler bugs.

The best way to back up my claim would be to rewrite some large C codes in my subset and show that they are still miscompiled. Unfortunately that is too much work. Rather, I’ll point to a few compiler bugs that we’ve found which, while they are not generally triggered by programs in the subset I outlined above, are triggered by pretty simple test cases. I would argue that — taken together — these effectively kill the idea of a bug-safe subset. Keep in mind that most of these test cases are not fully reduced; in some cases a compiler developer was able to do considerably better (GCC hacker Jakub Jelinek is most amazing test case reducer I know of). Also, my guess is that most of these bugs could be tickled by programs in a smaller subset of C. For example, test cases for bugs that we report often use the volatile qualifier not because it has anything to do with the bug, but rather because volatile objects serve as a convenient mechanism for suppressing optimizations that would otherwise mask the bug.

GCC bug 42952 results in this code being miscompiled:

static int g_16[1];
static int *g_135 = &g_16[0];
static int *l_15 = &g_16[0];
static void foo (void) {
  g_16[0] = 1;
  *g_135 = 0;
  *g_135 = *l_15;
  printf("%d\n", g_16[0]);
}

GCC bug 42512 results in this code being miscompiled:

int g_3;
int main (void) {
  long long l_2;
  for (l_2 = -1; l_2 != 0; l_2 = (unsigned char)(l_2 - 1)) {
    g_3 |= l_2;
  }
  printf("g_3 = %d\n", g_3);
  return 0;
}

GCC bug 41497 results in this code being miscompiled:

static uint16_t add (uint16_t ui1, uint16_t ui2) {
  return ui1 + ui2;
}

uint32_t g_108;
uint8_t f3;
uint8_t f0;

void func_1 (void) {
  for (f3 = 0; f3 <= 0; f3 = 1) {
    for (g_108 = -13; g_108 == 0; g_108 = add (g_108, 0)) {
      f0 = 1;
    }
  }
}

LLVM bug 2487 results in this code being miscompiled:

int g_6;
void func_1 (void) {
  char l_3;
  for (l_3 = 0; l_3 >= 0; ++l_3) {
    if (!l_3) {
      for (g_6 = 1; 0; --g_6);
    }
  }
}

LLVM bug 2716 results in this code being miscompiled:

int func_3 (void) {
  long long p_5 = 0;
  signed char g_323 = 1;
  return (1 > 0x14E7A1AFC6B86DBELL) <= (p_5 - g_323);
}

LLVM bug 3115 results in this code being miscompiled:


unsigned int g_122;

int func_1 (void) {
  unsigned int l_19 = 1;
  if (1 ^ l_19 && 1) return 0;
  return 1;
}

I could go on, but six examples should suffice.

A better question than “Is there a bug-safe C subset?” might be “Is there a subset of C that, when combined with restrictions on the optimization passes used, is safe from compiler bugs?” I don’t know the answer to this, but I do know that disabling optimizations is probably the only reliable way to keep large, possibly-buggy parts of the compiler from executing. Also, organizations creating safety critical systems often limit the amount of optimization that is performed by compilers they use (though that is probably as much to give traceable, debuggable code as it is to avoid miscompilations). In the medium to long run, an even better idea would be to forget about artificial constraints on the language and instead use a verified compiler or a tool such as one of the emerging translation validators for LLVM.

Continuous Paper Reviewing (Wonkish)

People running conferences often have a single round of reviewing: papers are assigned to program committee members, reviews are due some weeks later, and then after reviewing is finished everyone decides which papers to accept. This works but there’s room for improvement. First, not all papers need the same number of reviews. Second, the number of reviews that each person has to produce may be large (like 20+ papers), causing problems for those of us who have poor time management skills.

Other conferences review papers in two rounds. The first round might assign three reviewers to each paper, the second round two or three more. This not only spreads out the work over more time, but also enables an optimization where papers that are obviously going to be rejected do not need to make it to the second round.

Since reviewing is typically managed electronically using a system like HotCRP, it should be straightforward to spread reviews out over more rounds, or even make it into a continuous process. My idea is that it would be nice to add a tiny bit of statistical significance to the all-too-random paper acceptance process. As reviews arrive, the system — which knows the total number of papers and can be told the likely number of papers that will be accepted — can start to compute confidence intervals for each paper’s “true” review score. The system would ask for additional reviews for a paper until either some maximum number of reviews is reached (six, perhaps) or it can be established with (say) 90% confidence that the paper will be accepted or rejected using fewer than the maximum number of reviews. Statistics about the distribution of review scores should be easy to dig up from previous conferences. In addition to being statistically superior (I hope) and spreading out the work, this scheme would be more tolerant of reviewers who fall behind.

The most obvious problem seems to be that reviewers won’t have hard deadlines. This can perhaps be solved by asking for some number of reviews per week and relentlessly harassing reviewers who fall behind. On that note I’m not sure why I’m writing a blog post right now since I am behind on some paper reviews.