Code instrumentation best practices

Most developers overlook the production aspects of the software they develop. A program is always written to run somewhere into production – hopefully – and writing production-aware code requires some best practices supported by experience.

Without digging into advanced topics such as aspect weaving in aspect-oriented programming, we do think that there are basic dimensions of code instrumentation that should be known by all software developers.

Tracing

The first critical element of code instrumentation is tracing. Adding traces to your code is always a good idea. Yes, sometimes they might impact performances in some scenarios but this can be solved easily by adding different tracing levels, configurable at run time.

Too many developers still favor debugging instead of tracing making their investment disappear once they close their development tool. See our previous article "Our case against debugging". Properly done, tracing is much more powerful, as you should be able to leverage the trace information to understand and address issues into production. Most of the time, issues arise from unanticipated combinations of code paths or data values, so the developer should not make too many shortcuts or assumptions while developing.

Furthermore, over time, different developers will contribute to the application, not to mention to the same code file, but traces added to the code will remain, thus preserving the investment made initially.

Exception handling

Exception handling is the second important area of code instrumentation. When an exception occurs, you might be in unanticipated scenarios, or at least under specific conditions that require attention.

Dealing properly with exceptions, by taking relevant actions to maintain integrity, to provide information about data values or how we got there will prove to be critical to support the application in production.

In .NET, we still see a lot of issues around exception handling, like these ones::

The only result of this is losing the stack trace information, doing nothing would actually be better than catching the exception and reducing the available information. If you decide to manage an exception, make sure you can provide additional information or add value to the specific case.

Finally, if you want the full stack frame information available in production (with source file line numbers), it is also a good idea to deploy symbol files (.PDB), especially when deploying server-oriented application (Web)which is not always done.

Performance monitoring

In Windows, there is a very simple way to provide useful information related to performance at production time.

Performance counters are directly integrated in the operating system (it’s in fact there since the very first version of Windows NT), and you can create your own counters to monitor the relevant information with all the associated Windows tooling (Event Viewer). From the developer point of view, performance counters are also easy to define in .NET with all the associated classes.

From our experience, it is almost never used by application developers, as many tend to consider only the "out of the box" performance counters of the operating system or low-level components, losing an opportunity to measure performance in real scenarios.

Investing in those counters is a simple and pragmatic way of indentifying the key areas for performance improvement over the different releases of your application.

There are also tools that might help in profiling code "afterwards" but this is usually a heavier and more unpredictable process.

Data gathering and logging

Last but not least, for helping people supporting your application in production understand issues or optimize operations, it is often relevant to log some application-related data.

The developer of the application usually knows as he is coding what are the critical data elements that will be valuable when analyzing issues in production.

So don’t hesitate to log that data in a relevant place for your application, be it application log files or databases depending on the architecture and constraints.

In the end, ensuring these four dimensions are covered is not especially complex. It is more a matter of developer discipline than advanced expertise.

To read more

• Commenting, Testing, and Instrumenting Code from Dr Dobbs
• Instrumentation by Thomas Kyte
• An Introduction To Performance Counters by Michael Groeger
• Effective Code Instrumentation? discussion on Stack Overflow