A large part of my time at work recently has been addressing scalability issues in some applications.  Most of these stem from poor use of memory.  Most of these applications were proof of concept that made their way into production code for whatever reason.  More and more we identify problems and when we identify them to the developer, this is the typical exchange:

Me: You're taking too much memory here by using type Xxxxx when you could have gotten by with type Yyyyy

Dev: I have to use type Xxxxx, that's what the data is.

Sometimes, you just have to take a step back and look at your bytes.  Sometimes, you need a profiler, but sometimes you can just do some math.

For example, if I have an array of dates that correspond to events, a logical choice for a datatype is System.DateTime.  However, if you've got millions of events, System.DateTime is relatively expensive.  Especially if you consider that your event times only have resolution down to the nearest minute and only represent times within the last several weeks.  The range, or domain of DateTime (ticks since the epoch) is overkill for your circumstances. You may have been able to get away with UInt16, which would have been 2 bytes instead of 8, which is a huge reduction. (Provided that you've shown this array of DateTime to be a significant part of your memory consumption.)

I'm sure there is some official name for this, but I'll refer to it as "constrained domain", where you know something about your circumstances or usage that allows you to reduce the range or domain of a concept in order to store it as something smaller to improve memory consumption.  Remember, just because a big, rich datatype makes things easier, matches a db schema, or is named the same as the concept you need to store, doesn't mean it's a good match for your scenario.

There are, of course, other considerations to make.  The Y2k problems were the direct result of taking this concept to the extreme (of course resources were in much shorter supply then than they are today).

So, when you are looking for places to trim your memory usage, look for places where you can constrain the domain of a concept and store it in fewer bytes.  Also, if you're in CLR-land when you do this, look for boxing value types.  Remember, bytes in a System.Collections.ArrayList take up 5 bytes each, not 1. (9 bytes in 64-bit land) (Yeah generics)

I stumbled across something very cool while investigating my MSBuild problem.  Former MSBuild team member Jomo Fisher (now on the C# team) has an extremely simple way to target the 1.1 framework with VS 2005.  For many people I know, this was the only roadblock keeping them from using 2005.  Awesome stuff.  I think this should be definitely be part of the final release.

I ran across a known bug in MSBuild in Whidbey beta 2 that caused me considerable headaches today.  It seems beta 2 will not copy project references to the bin directory of web projects when compiling with MSBuild.  This issue will be fixed in the final release, but I thought some Google juice might help someone else find the definitive answer with less frustration than I had.  Here's a relevant forum post on the subject.

In the meantime, I'll simply add a copy task to copy the relevant assemblies to the bin directory.

About 2 years ago, I wrote about wanting an edger.  I finally got one.  I have no idea why I waited that long.  I've been using my weed eater to do it, thinking that was almost as good.  I was wrong.  While the weed eater works better than a spoon, it is quite time-consuming and uncomfortable (mine doesn't automatically swivel to spin vertically), so I put it off, then the lawn encroaches on the sidewalk and driveway so much that it takes even longer to do, which makes me want to put it off longer.  I finally settled on the Black and Decker Edge Hog since I've been pleased with my Leaf Hog.  It works very well.  I edged the front for the second time with it in about 3 minutes (the first took longer because I had alot of material to cut).  Now, my lawn is the best looking on the block. (now that the guy across the street moved out.  He was a fanatic and had an amazing lawn)

This weekend was a momentous occasion for my bathroom project.  I got the tub into place in the bathroom!  With the help of my brother, dad, and friend Dave (Thanks a million!), we successfully navigated the maze of narrow doors and hallways from the garage to the master bathroom. (I'm still not quite sure how) Anyway, the tub is in place, ready for levelling and hooking up.  After that, it's time to prepare the walls and get ready for tiling and installing the new pedestal sink.  I've been stalled for a while waiting for the right opportunity to get everyone together to help carry it in there, but now the project can continue forward.  The photoset of this project is getting too big and confusing to just link to the whole thing, so here's a link to the start of this weekend's pictures within the set.

As usual, I was busy, so my Dad got more action shots than me.  I'll link to his pictures when he gets them up.

Despite being pretty old, Becky's 95 Nissan Maxima still runs great.  The alternator recently went out, so I replaced it.  Being more of a geek than a gearhead, I wouldn't have attempted this a few years ago. But, after some help from my buddy Mark Leech and the internet, I had it running again.  The Maxima does not appear to be designed for replacing the alternator without several extra double-jointed limbs and a device that allows matter to pass through other matter.  I had to remove or loosen quite a few items including the A/C compressor and radiator fans to get it out.  I also had trouble with the belt tensioner being frozen in place.  A new pair of sweet vise grips and a hammer alleviated that problem.

Here's a link to the related photoset.  My hands were pretty dirty during most of the ordeal, so I was reluctant to pick up my camera. As a result, there are not that many pictures.  Enjoy them anyway.

Back last year, I outlined a problem I had with calling Response.End from a thread other than the thread that handled the request.  It stemmed from a progress mechanism we had implemented that had the ability to cancel long-running tasks when the user hit stop or closed the browser.  As it turns out, that wasn't the whole story.

We recently added a few reports where, the majority of the time, gathering the data takes over a couple of minutes.  While doing memory and performance optimizations, we noticed our worker process was recycling with the following message in the System event log:

A process serving application pool 'XXXXXXXXXX' terminated unexpectedly. The process id was '####'. The process exit code was '0xff'.

Googling for this error doesn't do you much good since it could be caused by a variety of reasons.  Most of the advice I came across wasn't well thought out and made quite a few bad assumptions.  So we added some more debug logging and determined that this was happening when our canceling mechanism kicked in when a user decided they didn't really want to wait 5 minutes for the report.  I was greatly puzzled by this since this feature had been tested thoroughly and had been running in production for some time.  Looking back through the server logs, it was evident that it had been happening all along, just not very often. We had just gotten the performance on the vast majority of pages to be very good and it wasn't an issue.  The problem only came to the surface when we added the report that always takes a while.

Here's the problem.  If the handling thread is aborted, it causes a condition that IIS considers to be bad and that forces the worker process to recycle. (presumably, there is some communication that doesn't occur) This doesn't happen when Response.End() is called because it passes a special exception as the exception state to Thread.Abort.  The HttpApplication catches ThreadAbortException and checks the ExceptionState.  If it is an HttpApplication.CancelModuleException, it knows there was either a timeout, or Response.End() was called, and it cancels the Thread.Abort by calling ResetAbort which allows the thread to continue running at that point.  I thought that was pretty slick.

When I was aborting the handling thread, I was on a different thread, so I had to call Thread.Abort manually, so CancelModuleException was not being used, so the thread was ending completely and causing the recycle.  Since HttpApplication.CancelModuleException is internal (and rightly so) I could not simply use that mechanism.

The good news is that the Unload event always happens (for all practical purposes), even when the thread is being aborted.  So I added my own PageIsCancelling property to our base class Page and check it, along with the current ThreadState in Unload, and cancel any pending abort if the page is canceling.  So, the abort is contained within the callstack of the page, and the thread stays alive and all is well.  No more crazy recycling.

As an aside, it seems this is aggravated by multiple processors, which might explain how it passed testing on the dev's machine.  Although I don't have any proof of this.

Those of you who follow my vacationing activities know that I have a certain affinity for aquariums, which is odd because one of my biggest phobias involves murky water.  Anyway, while in New Orleans over the weekend, we stopped by the aquarium (map) after the wedding. Enjoy the photos.

This past weekend, we (Me, Becky, my Mom and Dad, and my aunt Kenny) piled in a rented Ford Expedition and headed to New Orleans where my cousin Jeremy was getting married.  We stayed in the J.W. Marriot on Canal street where we had a great view of Canal street and much of the city.  All the wedding festivities went well.  We were very tired from the drive, and had to duck out early from several of the events, but it was really great.  We hit the aquarium (pictures coming soon) and took a stroll around the French Quarter, which I really didn't like at all.  Something about lots of people crammed into a confined space that smells like a mixture of B.O. and pee just seems to get to me for some strange reason.  I've got photo sets of the rehearsal dinner and the wedding/reception.  My dad and I also created a Flickr group to pool all our pictures.

I also played with myGmaps and created this little map that I will continue to annotate with stuff from the trip.

It was really fun to see the family and participate in the festivities, but I probably won't be planning any more New Orleans vacations any time soon.