Given my background in fault tolerant distributed systems, performance metrics are something that come up time and time again. I've spent many a happy hour, day, week, month pouring over timings, stack dumps etc. to figure out how to make RPCs go faster, replication to scale and transaction logs to perform. (Funnily enough I was doing the latter again over Christmas!) But this is something we've all had to do when working on Arjuna.
If you've ever had to do performance tuning and testing then you'll know that it can be fun, but often an almost never ending task. Then there's the comparisons between different implementations. This isn't just a vendor-specific area: even in academia there's always a bit of "mine is faster than yours" mentality. I suppose it's human nature and in some ways it's good and pushes you to improve. However, performance measurements need to be taken in a scientific manner otherwise they are useless. It's no good to simply state that you get "X transactions a second" if you don't specify the conditions under which that figure was achieved. This is true for a number of performance metrics, not just in the computing industry: there's a reason world records in athletics have defined conditions, for instance.
Now of course you may not have the same hardware to test and the tests themselves can be notoriously hard to access from one vendor/competitor to another. What tuning parameters they use as well as the general configuration can also be difficult to obtain. This is why Jim Gray and others wrote A Measure of Transaction Processing Power (though the principles in there can be applied much more widely.) This eventually produced the TPPC. But the general principle here is a scientific approach to performance metrics is the right way (though it's still possible to skew them in your favour.)
Whenever I see or hear of performance figures I always want to know the conditions (hardware, configuration, wind conditions etc.) It's not possible for me to take them seriously unless I have this additional data. Giving out the tests themselves is also a good thing. However, performance is not something you can support like you do with, say, transactions, where you either are transactional or you're not. With performance it can be a never ending struggle to keep improving, especially as hardware and software improve and your competitors get better too; you've always got performance even if it's bad! Performance is something that for some users can be as critical as new capabilities, or even more so. But it can't be performance at the cost of reliability or robustness: something that screams along faster than a speeding bullet yet crashes after 24 hours is unlikely to be as useful as something that is half as fast yet able to stay available indefinitely.
Subscribe to:
Post Comments (Atom)
3 comments:
Great post! Without the information that defines a test's environment, performance numbers are just numbers. I've always looked at a software tests as algebraic equations that can contain many variables. With performance tests, you have to consider a large set of variables in the that equation as you don't only care about whether the system returns a value of X or Y, but how well, and for how long, it returns that value. You also have to ensure that the tests themselves correctly exercise the system under test. I once saw a telephone voicemail control system performance test return some truly impressive throughput numbers until we realized that the test called the same phone number every time. The number was cached, so the system under test was able to avoid pesky things like disk I/O or most of its routing functions to place the call.
ROFL
Great post! I love Mark's writing as he is able to express in cristal cristal clear words, shared concepts and experience, just like his previous post on google.com page as "internet dial tone".
I have been involved myself in performance testing and totaly agree an all aspects of this post. Now that I moved from academia to industry, I am striving to persuade my company to scientifically evaluate performance figures of our IT products and disclose them along with tech specs in the form of self explanatory "reports" instead of plain useless numbers. It is incredible how this requires a big change of working habits (more precision, more effort...). I believe that this approach will state the quality of our work and therefore of our solutions. No secret it is going to be mayhem but fun and that will push us to improve. And at the same time it will be hard work to get costumers acquainted with this kind of "extra" information.
Dependability vs. performance...is a never ending match as you can't have the best of both worlds! Similarly to Len, I'd like to think to the set of parameters that tune a system to a specific working point, as one of the possible solutions in the space that describes a system model. Moving within such space, from point to point, enables you to tune the system itself in a different dependability/performance trade-off.
Post a Comment