I blogged a few days ago about a crazy problem I had with garbage collection in ASP.net under 2.0.  I finally tracked the problem down to Array.Sort.  I created a framework for working with large sets of data in a dynamic way.  This allows us to group data and do other complex statistical things based on fairly open-ended user input, rather than having to write special case code for each scenario.  It uses some reflection to be able to drill into objects.  When sorting, we pass in a dynamic IComparer that can drill into each object and do dynamic comparisons.  Unfortunately, many of our algorithms were dependent on sorting the data first.  This was causing a huge number of allocations due to drilling in and boxing lots of value types.  And, the nature of the sort causes that to happen multiple times per item.

Under 1.1, we took the hit on memory.  It didn't take that long, and it was soon collected.  Under 2.0, the GC seems to recognize this allocation pattern early and proactively begins collecting aggresively.  The problem seems to be that this slows down the sort tremendously, to the point where it essentially comes to a halt.  After a long time, you get a bizarre looking exception about a failed sort from deep in the framework.

What I can't figure out is, during all this time, there's still plenty of memory, and activity on other requests is not impacted horribly.

Well, I took a long hard look at some of the operations, and re-implemented them with a hashing strategy rather than relying on a sort.  As it turns out, for most cases, this is much more efficient.  Long story short, overnight stress testing turned up with clean results this morning.