I did a bunch of profiling at work today.  I use the Allocation Profiler for memory profiling.  It does a great job of letting you see your allocations visually in several very useful forms.  Today, I took a second look at nprof, a more CPU-oriented profiler that attempts to give you an idea of what calls are taking up the most time.  When I first looked at it almost a year ago, it was not useable at all.  Now, it seems to only have a few problems with GUI apps, and multi-threaded/multi-AppDomain projects.

The app I spend most of my time on at work is a data analysis application.  Data is loaded into the DB from a spool, so 99% of the app is “read-only” access to the DB.  This means I can optimize the heck out of it since I'm not concerned with transactions and such.  The downside is that i must optimize the heck out of it.  A common query can return millions of rows from the database, so taking even a few milliseconds out of the loop can save alot over that many iterations.

My first insight is: Don't use NUnit to run performance tests.  At first, my unit tests seemed to be a very convenient location to put performance tests.  That is a bad idea.  The perf numbers scared me to death.  It appeared as thought I had created a performance monster, and not the good kind of monster.  Turns out, NUnit goes to alot of trouble to isolate the test runs in separate AppDomains so it can unload them easily and you don't get undesired interaction.  This seems to add a great deal of overhead, probably in marshaling across the boundaries.

My second insight is: Use the Allocation Profiler (or some other profiler that lets you look at memory usage)!  I don't want to go into a big discussion on how to use it.  It's pretty straightforward.  If you interested in specifics, leave me a comment.  Remember, allocation in the CLR is cheap, but excessive garbage collection can be costly.

My last insight is: Use nprof (or another profiler that gives you CPU info).  nprof gives you a good idea of where your CPU bottlenecks are.  I was really able to get a good idea about what needed attention.

In a particular performance test of my data access layer, I was pulling over a million rows.  I was able to cut memory consumption in half, and increase my speed by a factor of 10 by using information gleaned from the profilers.  This, or course, means the code was pretty crappy.  He he, just kidding.  Like I said, it's a very tight loop, and a little goes a long way.

If you want a good reference for performance-related stuff, check out Rico Mariani's blog.  He appears to be THE Microsoft performance guy.