A Simple Portable Timing Library

NOTE: This project is on hold while I work on some higher priority projects for work.

Most software systems of any complexity have at least basic performance requirements and demonstrable performance is a central requirement of many of the systems I work on. I know Knuth said that premature optimization is the root of all evil, and I think he is right about that (in spirit at least), but too many development teams leave performance testing until the last few weeks (or days) before a critical acceptance test, and then wonder why they can’t get the system optimized in time to meet their requirements. At work, I support a very large and complex software system and we have many (over a hundred) development teams as customers. Many of our customers have very steep performance requirements (e.g. near-real time latency and thousands of updates per second peak processing rate) and older or low-power hardware as their target platform, so they should be thinking about performance from the beginning.

In our system, there are usually many ways to accomplish any given task and which way to go is often far from apparent because the system itself and the size of the APIs are just prohibitively complex for the average development team to comprehend in the time allotted for development. We have clear training recommendations for development using our software and there is online and other training available, but it’s rarely used. These teams usually end up poking at the APIs until they manage to get something to work, and then they happily run with that first solution until they get to testing or run into a wall late in their development cycle where they find that they just can’t implement some necessary functionality or meet a critical performance requirement given their chosen path. That’s usually when they come knocking on our door, demanding quick answers or, if they are large or important enough, on-site support by our small and already over-tasked team. Sometimes they get their way and we end up dropping whatever development or other important tasks we were involved in to help get them over the finish line, sometimes practically building a large part of their system for them (in the worst cases).

All this could be avoided, and I could sleep better at night, if these development teams just made their known performance requirements a central part of their planning effort. They could develop a custom timing library that they embed in their applications from the beginning and test their chosen solutions early and regularly, analyzing the results (using the simple analysis application they also build) to make sure they are on target to meet their performance requirements. But they won’t do this, probably for the same reasons they rarely take the training we recommend to get them up to speed on our system. Maybe the training is perceived to be a waste of time, or maybe they’re just ornery developers who (like me) hate to read the manual. The result is a lot of development programs coming up short in testing for entirely preventable reasons.

I’ve been centrally involved in a couple of efforts to build benchmarking or stress testing tools for critical (potentially life critical) systems in the last several years. Both efforts were under intense time constraints but we managed to build working tools that accomplished the mission in both cases. I learned some lessons and gained some knowledge in these efforts that I’d like to duplicate for myself. I’ve long noticed that there is a scarcity of good basic timing libraries available on the net for the stealing. Since I can’t point my work customers to an existing open source or otherwise public library and I really want one for my own use, I’m going to write my own and make it available here.

If you’re still with me through my combination rant and cautionary tale, the bottom line is that I’m going to write a timing library in my free time and make it available on these pages. My first efforts will consist of some simple timing classes in C++ (Java later as I get time), followed by a benchmark analysis tool and other additions as time allows. I will definitely get some good use out of this code and I hope others find it useful as well.