Orders of Magnitude
Sunday 21 February 2010 20:57
Improving performance is often a desirable goal. Sometimes you'll have a precise number for just how much performance needs to be improved by, particularly in real-time systems. More often, though, the request for improved performance is far more vague. So, what sort of numbers should we aim for when we want things to go faster? This depends on why you want faster performance -- do you just want to save a bit of time, or do you really want things to change?
Take, for instance, the time it takes to run your entire test suite. This can vary wildly, depending on the application, from seconds to days. Let's say we're working on a small project, and we have a test suite that covers the entire application in one minute. This is fast enough that we can run the entire suite every time before we commit, but we won't be running it every time we make a small change. If we made it go, say, twice as fast, we'd definitely save ourselves some time -- thirty seconds for every commit, if we really do run all the tests before every commit. This is still too slow to be running each time we make a small change, but what if we sped up the suite by an order of magnitude instead, so it takes only a few seconds to run? Now, running the entire suite every minute or two is practical, rather than just before every commit.
Sometimes, we really can get performance improvements of an order of magnitude, by improvements in technology or a clever new algorithm. Otherwise, we might still be able to do what all programmers do -- cheat. If we can get most of the benefit in a much shorter time, then this is often good enough. Going back to our test suite, if we can identify some subset of the tests that run in 10% of the time with 90% of the coverage, then most bugs we might introduce are still picked up, while our tool, the test suite, becomes more flexible.
By improving performance not by small amounts, but by orders of magnitude, we can change the way we use and think about our tools. Performance really does matter.
Topics: Software development