My brother-in-law posted a note on Facebook that was basically one of those silly things you do and perpetuate across the internet.  I usually don’t take part in such things, but this one seemed interesting, and I’ve been thinking about ways to jumpstart my blogging again now that the embargo on all the cool stuff is lifted.  So, I thought I would do it.  Here are the rules:

• Grab the book nearest you. Right now.
• Turn to page 56.
• Find the fifth sentence.
• Post that sentence along with these instructions in a note to your wall. (this was on Facebook, so it is referring to that wall)
• Don't dig for your favorite book, the coolest, the most intellectual. Use the CLOSEST.

So, it took me a while to determine which book was the “closest”, as my position is roughly normal to the bookshelf in my office.  I finally decided to be honest and pick the one that was really closest, but I will share another that was very close, as it is a good segue into future blogs.

The first (and official) one:

Semiconductor materials at 0 K have basically the same structure as insulators – a filled valence band separated from an empty conduction band by a band gap containing no allowed energy states (Fig. 3-4).

Solid State Electronic Devices, Ben G. Streetman

The second, and more relevant/interesting one:

The shim’s algorithm for picking a version in the COM interoperability scenario is much more straightforward – the latest version installed on the machine is always used.

Customizing the Microsoft .NET Framework Common Language Runtime, Steven Pratschner

What is amazing about this second one is that this is directly related to one of the features I’ve been working on for CLR v4 (and yes, that is actually the 5th sentence on page 56).  Namely, this feature is known as “in-process side by side” (or in-proc SxS for short), and was announced publicly at PDC last month.  This feature allows you to have more than version of the CLR loaded and running in a single process.

This feature is primarily a compatibility feature, targeted precisely at the behavior noted in the quote above.  When we use the latest version, we can break existing COM objects.  Not only because of breaking changes we make (of which the number is fairly small), but because of other, more subtle behavior dependencies.

Previously, loading a CLR version into the process locked the process to that CLR version.  Any other policy than “pick the latest” results in a load order dependency problem that can result in “for sure” breaks because COM components targeting newer runtimes cannot run on old runtimes.  So, clearly, that was the best choice of policies.

Now that we support multiple runtimes in the process (v2 and above), we can make a smarter, more compatible choice about runtime activation.  The precise policies are still being worked through, so I’ll avoid stating them explicitly, but you can imagine us being able to make a much better choice about what runtime to activate to run a given managed COM component.

I’ll be posting more about this feature and it’s implications soon.  Feel free to seed my future posts with questions in the comments.  Hopefully, this is the jumpstart I needed.  As for the “game” above, feel free to do it, or ignore it.  It won’t result in any difference to your luck, financial situation, or anything else.

Some of my posts that I get the most recurring email/inquiries on are my various posts about the STDF (Standard Test Datalog Format) parser that I originally created as an experiment back in 2005.  After some help from a former collegue, I am pleased to announce that this is finally available on CodePlex as LinqToStdf!

It is a managed library for processing STDF files, and gives you a model to explore the data via Linq queries.  This means, you can leverage the wide variety of managed languages (C#, VB, F#, JScript, IronPython, IronRuby, Managed C++, Boo, etc.) to process the data in STDF files.  It also works in Silverlight!  It has built-in support for the V4 spec, but it's highly extensible and should be able to parse any version of the file format as well as custom records.  It can be configured to be highly strict and throw on format errors, or be robust in the face of issues that normal STDF processors would choke on to the point of being able to detect and repair corruption on the fly.

If that interests you, I'd love for you to drop by and take a deeper look at it and get involved in its ongoing development.  I've already got at least one person interested enough to contribute and ensure its success as a community project.  There is currently a "beta" release available, and hopefully we'll whip it into shape enough to call it v1.0 soon.

My hope is that this can be an adoption driver for .NET in the semiconductor industry and that through this project I can be an ambassador for the CLR in that area.

I've been playing alot with beta 2 of Silverlight 2, and I've been totally amazed at the scenarios it enables.  Working hard down inside the CLR engine, we're sometimes insulated from some of the innovation going on higher in the stack and it blows us away when we do get a chance to see it.

One of the very cool things in Silverlight is "Deep Zoom", which came from the SeaDragon project from Microsoft Research.  I decided to try it out myself on a very large panorama that I made a long time ago in New Orleans.  Unfortunately, the current toolset seems to trip over the large file size (20516x15291).  I'm trying to find out the real story behind the limitation.  All I know right now is that smaller files work.

So, I decided to try to slice it up into smaller, manageable chunks and just butt them against one another to simulate one large file.  The problem was that I couldn't find a tool to do this that didn't also trip up over the size of the file.  So, I wrote my own.  There are likely better ways to do this, this was just a quick and dirty attempt to make something that didn't totally crawl to a halt due to page faulting (or outright throw OutOfMemoryException).

The code below is what I came up with.  I use System.Drawing.Bitmap, lock the portions of the image I need, and do the data copying myself.  I ended up with this solution because GDI+ (DrawImage) seems to like to make alot of buffers (big ones in this case), and I couldn't fall back on good ol' bitblt because I couldn't get the right kind of data structures without more copying of the data.  This runs plenty fast and only takes up marginally more memory than it takes to represent the original and the size of one of the destination tiles.

Unfortunately, The MultiScaleImage does some weirdness with image that are butted up against each other, and you get as much as a whole pixel of "space" between them, depending on your zoom level. I'm still looking into other possible workarounds. So, it's wasn't ultimately useful, but I thought the code was interesting enough, and probably has academic usefulness.  So, enough talk, here's the code:

   1:  using System;

   2:  using System.Collections.Generic;

   3:  using System.Drawing;

   4:  using System.Drawing.Imaging;

   5:  using System.IO;

   6:  using System.Linq;

   7:  using System.Runtime.InteropServices;

   8:   

   9:  namespace ImageSlicer {

  10:      static class Extensions {

  11:          public static IEnumerable<int> Times(this int number) {

  12:              for (var i = 0; i < number; i++) yield return i;

  13:          }

  14:      }

  15:   

  16:      class Program {

  17:          [DllImport("msvcrt.dll", SetLastError = false)]

  18:          static unsafe extern byte* memcpy(byte* dest, byte* src, int count);

  19:   

  20:          static void Main(string[] args) {

  21:              var sourcePath = args.FirstOrDefault();

  22:              if (String.IsNullOrEmpty(sourcePath)) {

  23:                  Console.WriteLine("No source image path provided");

  24:                  ShowUsage();

  25:                  return;

  26:              }

  27:              if (!File.Exists(sourcePath)) {

  28:                  Console.WriteLine("Source image path doesn't exist");

  29:                  return;

  30:              }

  31:              var gridSizeStr = args.Skip(1).FirstOrDefault();

  32:              int gridSize = 4;

  33:              if (gridSizeStr != null && !int.TryParse(gridSizeStr, out gridSize)) {

  34:                  Console.WriteLine("Could not convert {0} to a valid grid size",

  35:                      gridSizeStr);

  36:                  return;

  37:              }

  38:              if (gridSize < 2) {

  39:                  Console.WriteLine("The grid size must be greater than 1.");

  40:                  return;

  41:              }

  42:              try {

  43:                  Console.WriteLine("Slicing {0} into a {1}x{1} grid.",

  44:                      Path.GetFileName(sourcePath), gridSize);

  45:                  Console.WriteLine("Loading...");

  46:                  using (var sourceBitmap = new Bitmap(sourcePath)) {

  47:                      Console.WriteLine("Source Image: {0}x{1}",

  48:                          sourceBitmap.Width, sourceBitmap.Height);

  49:                      var sliceWidth = sourceBitmap.Width / gridSize;

  50:                      var sliceHeight = sourceBitmap.Height / gridSize;

  51:                      Console.WriteLine("Each slice: {0}x{1}", sliceWidth, sliceHeight);

  52:                      int tile = 0;

  53:                      foreach (var row in gridSize.Times()) {

  54:                          foreach (var column in gridSize.Times()) {

  55:                              Console.WriteLine("Creating {0} of {1} ({2},{3})",

  56:                                  ++tile, gridSize * gridSize, column, row);

  57:                              using (var destBitmap = new Bitmap(sliceWidth, sliceHeight)) {

  58:                                  var sourceData = sourceBitmap.LockBits(

  59:                                      new Rectangle(column * sliceWidth,

  60:                                          row * sliceHeight, sliceWidth,

  61:                                          sliceHeight),

  62:                                      ImageLockMode.ReadOnly, PixelFormat.Format24bppRgb);

  63:                                  var destData = destBitmap.LockBits(

  64:                                      new Rectangle(0, 0, sliceWidth, sliceHeight),

  65:                                      ImageLockMode.WriteOnly, sourceData.PixelFormat);

  66:                                  unsafe {

  67:                                      byte* pSrc = (byte*)sourceData.Scan0.ToPointer();

  68:                                      byte* pDest = (byte*)destData.Scan0.ToPointer();

  69:                                      foreach (var line in sliceHeight.Times()) {

  70:                                          memcpy(pDest, pSrc, sliceWidth * 3);

  71:                                          pSrc += sourceData.Stride;

  72:                                          pDest += destData.Stride;

  73:                                      }

  74:                                  }

  75:                                  sourceBitmap.UnlockBits(sourceData);

  76:                                  destBitmap.UnlockBits(destData);

  77:                                  destBitmap.Save(String.Format("{0}_{1}_{2}.png",

  78:                                      Path.GetFileNameWithoutExtension(sourcePath),

  79:                                      column, row), ImageFormat.Png);

  80:                              }

  81:                          }

  82:                      }

  83:                  }

  84:              }

  85:              catch (Exception ex) {

  86:                  Console.WriteLine("Error processing image.");

  87:                  Console.WriteLine(ex.ToString());

  88:              }

  89:          }

  90:   

  91:          private static void ShowUsage() {

  92:              Console.WriteLine("Usage:");

  93:              Console.WriteLine("ImageSlicer.exe sourceImage [gridSize]");

  94:          }

  95:      }

  96:  }

I've been doing some app-building with Silverlight lately and exploring the limitations of the platform in comparison to the full desktop CLR and what that means for the Silverlight "ecosystem".  Those limitations can be summed up with 3 items:

1. Strict sandbox security model
2. Reduced managed framework surface area
3. No binary compatibility with libraries targeting the desktop CLR (eliminates the number of 3rd party components you can leverage).

I think #3 will begin to become a non-issue as more component providers provide builds for Silverlight. #2 can be broken down into 2 areas:

• Full technology areas that are unavailable on Silverlight (LinqToSql, WinForms, etc)
• Reduced/pruned APIs

The second is where my interest lies, and intersects with #1 (the new security model).  We've done extensive threat modeling against our APIs as well as automated tooling that has either removed (or made internal) certain APIs, or marked them as SecurityCritical, meaning they cannot be called directly from "user" code.

In addition, the security model requires safe, verifiable user code.  Normally when developing on the desktop CLR, unless you are specifically targeting a low trust environment, you can do whatever you like.

So, for this exercise, I pulled out my trusty STDF parser (blogged here and here).  I've used this project as a test vehicle for both v2.0 and Orcas, and it's served well as a project that leverages a large cross-section of features in the Framework, from high-level stuff like Linq down to expression tree inspection and further down to LCG/RefEmit.

In short, I was able to get the parser working without too much trouble. I felt like creating a source bed that intended to target both Silverlight and the Desktop should be an attainable goal.  You just have to switch your mindset from binary compatibility to source compatibility.

I combated the reduced surface area with extension methods, which worked quite well to centralize the "overload shims" that needed to be "Silverlight-only", as well as for a general refactoring tool.  My goal is to have all the desktop vs. Silverlight differences centralized into files that are either included or excluded from the build depending on which platform I'm targeting.  I wish you could create extension properties.  That would let me close all the surface area discrepancies that aren't caused by missing/irrelevant technology areas.

I was pleased that 99% of my LCG codegen stuff "just worked".  I make heavy use of Reflection.Emit via DynamicMethod to generate my record parsers based on attributes on the record classes.  The 2 problems I ran into were:

• Visibility restrictions - The new security model won't let my dynamic methods see internals.  I had used this ability to keep the API clean.  I'm still figuring out the best approach, but it was simple enough just to expose those methods.
• Verifiability - I had a few places where I was generating unverifiable code.  Some of these were my own codegen bugs, but others were just bad assumptions on my part.

This brings me to constrained callvirt, an interesting little IL tidbit I discovered in the porting process.  Calling conventions are subtly different between reference types and value types, and it also depends on whether the given method is actually overridden in the value type (which can lead to confusing breaks when that state changes).  In the v1.x days, you always knew the type you were dealing with, so it didn't matter that much and you could generally always create the right sequence of IL to make a call.  V2.0 introduced generics, which meant that you couldn't emit unified IL for both the possibility of reference types or value types.  This meant there needed to be a way to unify the IL for the 2 cases.  This is where the constrained prefix came in.  It allows you to write unified IL that works regardless of whether you're working with a value type or a reference type (think generics constrained by an interface, or calling a method defined on System.Object like ToString()).

Anyway, I was able to fix my unverifiable code by utilizing the constrained prefix.  It also simplified my codegen logic significantly in a number of places where I had different paths based on whether I was working with a value type or not.

All in all, I was pleased with the results.  I'll be posting a sample Silverlight app using the library when I get some UI stuff figured out.

Some of my posts are really reactions to search queries that have previously landed on my blog.  If they did a search that got to my blog, but I know they didn't find what they were looking for, chances are they (or someone else) will do the same again.  And, if I HAVE the information they are looking for, it makes sense to just add the information, even if it's what I would consider well-known or common sense information. (common sense for software developers, that is)

One general search query I see again and again is something like "What is Action<T> for?" or "What is Func<T>?"

These are framework-provided, generic delegate types.  If you'll recall, delegates can be thought of as type-safe function pointers.  A delegate type really just captures a signature as "callable" object.  Leveraging generics to define delegate types that can capture common signatures is goodness, since they are very flexible and can be used by anyone.  This also aids in interop between different components, since a general signature is far more interopable than custom delegate types.

In v2.0, several functional-looking APIs were added that took delegates as arguments (think List<T>), so instead of adding a special delegate type for each API, several "generic" delegates were added to capture the "essence" of a signature such as Action<T> which takes T and does some action (returning void), Predicate<T> which takes T and returns bool (presumably doing some test against T), Comparer<T> which compares 2 T's, etc.

In v3.5, even more generalized functional patterns were introduced (used heavily in Linq).  And we added a bunch more Action<> "overloads" for functions returning void, and added Func<> "overloads" for functions with a return value.  (I use overload loosely since these are classes and not methods) These patterns dropped the semantic "meaning" of the delegate, and just went straight to the idea of capturing a signature.

These framework-provided delegates are useful for using in your own code rather than creating your own.  Whether you leverage the Linq-centric, super-generic Action/Func pattern, or opt to consume the more meaningful v2.0 Predicate, Comparer, etc. is up to you.

This is one of those entries that attempts to fill a void in online search for a particular topic.  I ramble on for a while to give enough context so that a search engine can match it up in a relevant manner.

I was debugging something the other day, and thought I had come across a heinous bug in the CLR.  Turns out, everything was working fine and the bug was in the app.

A program was crashing and it was a managed exception, so I attached to it with VS2008 and dug in.  The first thing I noted was that this was a retail build, so my stack was collapsed in a number of places... expected, keep moving.  Then, I noted an interesting frame on my stack (this is a contrived example, not the actual thing I saw):

>    CanonTest.exe!CanonTest.GenericType<System.__Canon,int>.SomeOperation() + 0x55 bytes   

That's weird, what is System.__Canon and what's it doing here? Surely this must be a horrible CLR bug where my method tables are being corrupted!  Internet searches seem to be confirming my thoughts.  A few others with random __Canon's showing up on the stack do look like bugs.  After a bit more reasoning, I come to the conclusion that System.__Canon must be special in some way since it follows the pattern for such things...  marked internal, pre-pended with double-underscore.

A few emails later, I had my answer.  System.__Canon is the special type that is used to identify "canonical" generic type instantiations. It typically only shows up in release builds, so you don't normally see it on the stack while debugging.  So, it's easy to assume something's wrong when you do see it.

If you'll recall, one of the really cool things about the Generics implementation is that it allows for code sharing.  Jitted methods can be shared between compatible type instantiations.  For instance, the code for SomeGenericType<string> can likely be shared with SomeGenericType<Foo> (where Foo is another reference type. We don't currently share code between value types, so you'll continue to see those on the stack such as in the example).  That shared code needs to live somewhere, so we have the concept of a "canonical" generic type instantiation that is identified by __Canon as the type parameter.

Also, in alot of debugger stack representations you'll get the GenericType`2 form rather than expanded form.  In that case, you never see __Canon.

For some more info on Generics and code sharing, see Joel Pobar's excellent blog entry on the subject.

The CTP for the MVC framework includes support for master-page, page, and user-control based views.  I thought it might be interesting to enable .ashx-based views for things like RSS generation via System.Xml.Linq, or other more "raw" view output.

As it turns out, this is fairly trivial.  The place we need to extend is the IViewFactory returned by Controller.ViewFactory.  This is the component that is responsible for creating the view when a call to RenderView is made.

The default view factory is the WebFormViewFactory, which knows how to generate views based on .master, .aspx, and .ascx views.  Since we want to add support for .ashx, we'll use WebFormViewFactory as a starting place.  We'll inherit from WebFormViewFactory and override CreateView to supply our extra .ashx lookup.

using System;
using System.Globalization;
using System.IO;
using System.Web;
using System.Web.Compilation;
using System.Web.Mvc;

public class SpecialViewFactory : WebFormViewFactory {

    static readonly string[] ViewLocationFormats = new string[] { "~/Views/{1}/{0}.ashx", "~/Views/Shared/{0}.ashx" };

    ControllerContext _ControllerContext;

    #region IViewFactory Members

    protected override IView CreateView(ControllerContext controllerContext, string viewName, string masterName, object viewData) {
        _ControllerContext = controllerContext;
        //check to see if there is an ashx that matches here.
        object value = null;
        controllerContext.RouteData.Values.TryGetValue("controller", out value);
        string controllerName = value as string;
        if (controllerName == null) {
            throw new InvalidOperationException("No route data value available for controller.");
        }

        Type viewType = null;
        foreach (var loc in ViewLocationFormats) {
            var path = string.Format(CultureInfo.InvariantCulture, loc, viewName, controllerName);
            viewType = GetCompiledType(path);
            if (viewType != null) break;
        }
        if (viewType == null) {
            return base.CreateView(controllerContext, viewName, masterName, viewData);
        }

        if (!typeof(IView).IsAssignableFrom(viewType)) {
            //TODO: better exception
            throw new InvalidOperationException("Type not a view");
        }
        var view = (IView)Activator.CreateInstance(viewType);
        var viewHandler = view as ViewHandler;
        if (viewHandler != null) viewHandler.ViewData = viewData;

        _ControllerContext = null;
        return view;
    }

    private Type GetCompiledType(string path) {
        Type compiledType = null;
        try {
            if (File.Exists(_ControllerContext.HttpContext.Request.MapPath(path))) {
                compiledType = BuildManager.GetCompiledType(path);
            }
        }
        catch (HttpCompileException) {
            throw;
        }
        catch (HttpParseException) {
            throw;
        }
        catch (HttpException) {
        }
        return compiledType;
    }

    #endregion
}

GetCompiledType had to be replicated as it isn't exposed in the base class.  Note, I added a File.Exists check before I attempt to get the compiled type from the BuildManager. This was really to avoid having to deal with a bunch of first chance exceptions in the debug, although it seems likely that avoiding the exception is a good thing.  It wouldn't catch handlers that are mapped in the app dynamically or via web.config.

As you can see, I also added a ViewHandler class that my handlers can inherit from that gives them the same goodies that the other views get, but I'll leave that as an exercise for the reader to implement.

So, now the only thing remaining is to inject our special view factory into the pipeline instead of the default.  A simple way to do this is to set the ViewFactory property in the constructor of any controller that needs .ashx support. Now, you can create .ashx files and use them as views!

Next time, I'll show you how to add support for routing controller actions based on data not in the URL.

This release was exciting me before I made the move to Microsoft, and getting to be a part of it has made me even more excited about what we're giving our customers in this release.  If you're not familiar with Visual Studio, it is a suite of software development tools for creating just about any application or library you can think of.  I'll be writing more about this release in the coming weeks, but for now, go to www.microsoft.com/vstudio and check it out!

An interesting pattern I've seen emerge since the early releases of Orcas is what I might refer to as "delegate properties".  What I mean by that is a property (or field, I suppose) that returns a delegate.  This pattern has some interesting implications.

First, in a language that treats delegates as directly callable objects, this pattern looks just like a method (someInstance.TheProperty(args)).  You can't tell the difference (although VS gives you different intellisense) by looking at a callsite like this. Among other things, this leads to some interesting naming issues.  Do you name it like you would a method?

Second, it opens up opportunities to do some really powerful (and slightly insane) hybrid inheritance models.  Think about a virtual delegate property that has both a getter and a setter, now think about trying to predict what that delegate will do when you call it.  It doesn't sound like something to recommend as part of a public API, but I think there are some interesting scenarios there.

If I come up with something interesting and useful, I'll let you know.

I was helping a friend with a problem recently.  He was taking a set of serial web service calls and doing them in parallel to save time, and was not up-to-speed on the best approach for that.  Once he settled on an approach, he realized that since his web service calls were being wrapped in an abstraction layer, he didn't have the Begin/End asynchronous call methods that are provided by the proxy class.

"No problem, just wrap them in a delegate".  The compiler automatically gives you Begin/EndInvoke methods in addition to the synchronous Invoke method.  And, you're guaranteed not to mess up the implementation because it's all provided by the CLR!  Just one of those things you might forget if you find yourself in the same situation.

After my last few CLR posts, I've had a couple of private inquiries regarding the usefulness of closed static delegates.  To bring everyone up to speed, a delegate pointing to an instance method needs a "target" instance to operate on (we'll get to open instance delegates later).  A static method, needs no such target, so we can leverage the "space" used for the instance case to carry around another object of interest.  We call a delegate with a provided value for this space "closed over the first argument".

For example, let's say we have a static method that does some operation on two numbers.  For simplicity, let's just say it adds them.  Our silly class and method might look like this:

public static class NumberFunctions {
   public static double Add(double first, double second) {
      return first + second;
   }
}

Normally, a delegate for this method would look like:

public delegate double BinaryOperation(double first, double second);

But, we're going to create a closed static delegate, which means we're going to "burn" the first argument into the delegate itself, so it's not needed in the delegate signature.  Instead, we'll use the following delegate signature (I didn't spend much time thinking up these names, I hope they make sense:

public delegate double ClosedCall(double other);

So, how do we create the delegate?  Normally, since C# (pre-Orcas) doesn't have syntax for creating closed static delegates, you are forced to use one of the Delegate.CreateDelegate overloads:

ClosedCall addToOne = (ClosedCall)Delegate.CreateDelegate(
        typeof(ClosedCall),
        1.0,
        typeof(NumberFunctions).GetMethod("Add", BindingFlags.Public | BindingFlags.Static));

Of course, we just spent 2 entries looking at a helper that can do this for us (I'm not claiming this is better, I just want you to be able to see what's happening):

ClosedCall addToOne = DelegateBinder.Bind<ClosedCall>(1.0,
        typeof(NumberFunctions).GetMethod("Add", BindingFlags.Public | BindingFlags.Static));

Now, a call to addToOne(someNumber) will yield the result of adding the supplied argument to one.  This is a contrived example, but you could imagine taking a method (perhaps generated on the fly via LCG), and "attaching" an instance to it via the first argument.  Then, being able to call it many times with different subsequent arguments, or passing it to another component that would provide the rest of the arguments.  In this way, you get the benefits of not having to keep track of an instance, without having to own the API for the instance.  Additionally, you could "chain" delegates together so that many arguments are captured in a stack of delegate calls, allowing closure-type semantics at the cost of some stack space (although since C# has closure support, you'd never really need to do that).

What's really cool is that with C# 3.0's Extension Methods feature, we now have language support for creating early-bound closed-static delegates.  If you bind a delegate to an extension method (using the regular syntax for an instance method), you will get the exact IL for creating an early-bound closed static method without our fancy helper class.  Let's see how that would look.  Let's use a different example to keep us on our toes.  Here's a helper function that creates email addresses:

public static class StringExtensions {
   public static string MakeEmailAddressWithAlias(this string domain, string alias) {
      return string.Format("{0}@{1}", alias, domain);
   }
}

Notice the "this" in front of the first parameter, this tells the compiler that the method should be considered when resolving method calls for string.  We'll use one of the delegate types provided in Orcas. Now, here's how the bind looks:

string fooDotCom = "foo.com";
Func<string, string> makeFooDotComAddress = fooDotCom.MakeEmailAddressWithAlias;

string email = makeFooDotComAddress("bar");

So, the result is that email will be bar@foo.com.

Hopefully, through these contrived examples, you can see the scenarios that closed static methods provide, as well as learn how you can create one the easy way with extension methods in Orcas.

In my last post, I showed a nifty way of constructing "early-bound" delegates using LCG.  Here's the same helper class implemented without LCG:

public static class DelegateBinder {
	public static TDelegate Bind<TDelegate>(object firstArg, MethodInfo method) {
		return (TDelegate)Activator.CreateInstance(
			typeof(TDelegate),
			firstArg,
			method.MethodHandle.GetFunctionPointer());
	}
} 

This one is quite a bit simpler, and extrapolating from what we learned last time, it's easy to see what's happening.  Hopefully, you are already familiar with the Activator class.  Basically, this just shows the managed call chain that produces a function pointer to a method given a MethodInfo.

I really like the LCG-based implementation, but only because of my love of DynamicMethod.  It's pretty complex, and aside from opportunities for caching, doesn't really have anything over this implementation. This one is just plain simple, and would have a single-line implementation if I hadn't put some line breaks to avoid formatting problems.  It does, however, highlight the annoyingness of having to work around the compilers' "helpfulness" when it comes to delegate construction.  If only I could just call the constructor directly.

It is worth noting that this doesn't work in the Silverlight 1.1 alpha or the compact framework (or XNA for that matter), neither of which expose RuntimeMethodHandle.GetFunctionPointer().

In a previous post about delegates, I discussed the following interesting cases of delegates:

• Closed static
• Open instance

See the previous post for the full explanation, but these basically open up some interesting dynamic scenarios.  The problem is that C# and VB do not expose syntax for constructing these in an "early-bound" fashion, that is using the special constructor on the delegate type rather than Delegate.CreateDelegate (which more or less binds via reflection).

For most scenarios this is not a huge problem, but there are some performance considerations and other issues to consider that I don't really want to dig into at the moment.  One sufficiently important scenario is testing early-bound invocation.  If your language doesn't support something, how can you test it?  Well, you can write the whole test in IL, but that is not a terribly maintainable proposition.

Another option is to only write the part you need in IL.  Unfortunately, C# doesn't allow you to write inline IL, but you can use Reflection.Emit.  And, since v2.0, you can use LCG (Lightweight Code Generation) via DynamicMethod.

The trick here is to understand how delegates are instantiated.  Delegates are just classes like any other.  They inherit from MulticastDelegate (typically).  The special part is that the runtime provides all the implementation and they have a special constructor.  Here's (approximately) the constructor signature for System.Action<T>:

public Action(object o, IntPtr method)

Object? IntPtr?  What the heck? Well, it's not as bizarre as you might think.  The object is simply the first argument for the invocation.  This allows binding to a particular instance ("this" for instance methods, arg 0 for static methods). The IntPtr is a pointer to the method.  "Pointers?!!?!?! in managed code?!?!" you say?  That's right, a pointer.  An object is easy enough to come by, but where do I get the pointer?  Well, the pointer can be easily retrieved via the ldftn opcode.  It loads the address of a given method (described via a token in IL, and a MethodInfo in Reflection.Emit).

Lets cut to the chase.  Here's a little class that can bind a method to a delegate type and allow you to provide the first argument (you'll need System, System.Reflection, System.Reflection.Emit using statements):

public static class DelegateBinder {

    public delegate TDelegate Binder<TDelegate>(object firstArg);

    public static TDelegate Bind<TDelegate>(object firstArg, MethodInfo method) {
        DynamicMethod dynMethod = new DynamicMethod("PassthroughBinderImplementation", typeof(TDelegate), new Type[] { typeof(object) }, typeof(DelegateBinder));
        ILGenerator gen = dynMethod.GetILGenerator();
        //load the first argument
        gen.Emit(OpCodes.Ldarg_0);
        //load the address of the method
        gen.Emit(OpCodes.Ldftn, method);
        //create the delegate
        gen.Emit(OpCodes.Newobj, typeof(TDelegate).GetConstructor(new Type[] { typeof(object), typeof(IntPtr) }));
        gen.Emit(OpCodes.Ret);
        return ((Binder<TDelegate>)dynMethod.CreateDelegate(typeof(Binder<TDelegate>)))(firstArg);
    }
}

With this class, you can dynamically construct all the early-bound variants (ignoring variants for signature relaxation) like so:

using System;
using System.Reflection;
using System.Reflection.Emit;

public delegate string Passthrough(string str);
public delegate string BoundPassthrough();
public delegate string ProgramPassthrough(Program p);

public class Program {
    static void Main(string[] args) {
        Console.WriteLine("Open Static:");
        Passthrough ospt = DelegateBinder.Bind<Passthrough>(null, typeof(Program).GetMethod("StaticImplementation", new Type[] { typeof(string) }));
        Console.WriteLine(ospt("Hello World"));

        Console.WriteLine("Closed static:");
        BoundPassthrough cspt = DelegateBinder.Bind<BoundPassthrough>("Hello World", typeof(Program).GetMethod("StaticImplementation", new Type[] { typeof(string) }));
        Console.WriteLine(cspt());

        Console.WriteLine("Open Instance:");
        ProgramPassthrough oipt = DelegateBinder.Bind<ProgramPassthrough>(null, typeof(Program).GetMethod("InstanceImplementation", Type.EmptyTypes));
        Console.WriteLine(oipt(new Program("Hello World")));

        Console.WriteLine("Closed Instance:");
        BoundPassthrough cipt = DelegateBinder.Bind<BoundPassthrough>(new Program("Hello World"), typeof(Program).GetMethod("InstanceImplementation", Type.EmptyTypes));
        Console.WriteLine(cipt());
    }

    public static string StaticImplementation(string str) {
        return str;
    }

    public Program(string payload) {
        _Payload = payload;
    }

    string _Payload;

    public string InstanceImplementation() {
        return _Payload;
    }
}

So, there are certainly cases that will break this, most involving incompatible signature issues between the method, delegate, and the first argument.  But I didn't want to make things more complicated for an example. Besides, the point of this is not really to give you some neat tool (you'll probably never need to do this), but to give people a better idea what the compiler is doing for you when you create a delegate.

During the last PDC, I drooled over the announcements regarding C# 3 and LINQ.  Now we are one step closer. Yesterday, they released a new preview of those features that bind against the current version of the framework.

Skeptical language junkies* should take a gander at Don Box's latest post that attempts to explain what the big deal is. I agree with some of the people saying that the syntax really gets in the way (and I feel that way even more after spending alot of time in Ruby lately).  They need to add a language feature that indicates whether you want the lambda or the expression, then you can fall back on type inference to keep the typing to a minimum.

Overall, I couldn't be more giddy about the way integrated query works, and being able to get your hands dirty with it now is really awesome.  We're already working with data in this way in our APIs at work, so I've been able to experiment with the language and library features directly with some of our data and it's an absolute dream.  They should not wait for the next version of the framework to give us a go-live license for this.

Now, this kind of thinking needs to be applied to the web stack and MS could have a compelling alternative to Rails.

* That is, language junkies who are skeptical, not junkies of skeptical languages.

I've been loving my new phone.  In an effort to fully exploit and justify my unlimited data plan, I've been dreaming up applications for it.  My latest one is something I've always wanted to do.

I hooked up a bluetooth GPS and capture my position.  Every 10 seconds or so, I send the logged data to a server.  I then have a spiffy Rails app set up to display my current location on a Virtual Earth map.  The page uses AJAX to poll the server every 10 seconds or so to update my position on the map.

I've got lots of ideas for how to mine and visualize the data, as well as provide nifty tools for my friends and family to track my location, as well as hooking the data up to my Flickr uploader (rewritten now in just about 20 lines of Ruby code) to automatically geo-tag my photos when I upload them.  I've got lots of cool things I plan to do with it while we're on vacation this summer.

I'll have live demos for the public available soon.  Some of my friends have been playing with it already as I debug it. For everyone else, here's a shot:

If you're a friend of mine and want to play with the live version, lemme know and I'll shoot you the temporary link.  Most of the time, it's pretty boring because I'm at work, but hold on to you hat when I start going somewhere.

One of the most popular search hits for my blog is "Managed XMP Parser".  A while back (actually it was 1 year ago today...whoa, freaky), I blogged about extracting the XMP data out of my pictures after screwing up the upload into Flickr.  I ended up writing my own code to pull out the XMP data.  I mentioned making it available, but it was relatively straightforward, so I never got around to posting it.

In the last week, I've gotten lots of requests for the code, so here it is, uglyness and all.  One interesting thing about my approach is that I do not rely on any particular file format.  I simply look for the XMP markers and pull out the XML in-between.  This means it will work on ANY file with embedded XMP.

All the usual disclaimers apply.  I don't claim this is the best way, but it works.  I've just plucked it out of my little date fixing app I built.  At the end, you'll have an XPathNavigator and a namespace manager setup to run XPath queries.  There's probably some sweet stuff the 2.0 can help us out with, but I haven't updated it.  Enjoy:

MemoryStream xmpStream = new MemoryStream();

byte[] beginPattern = Encoding.ASCII.GetBytes("<?xpacket begin");

int beginIndex=0;

bool beginFound = false;

byte[] beginStopPattern = Encoding.ASCII.GetBytes(">\n");

int beginStopIndex = 0;

bool xmlStartFound = false;

byte[] endPattern = Encoding.ASCII.GetBytes("<?xpacket end");

int endIndex=0;

bool endFound = false;

bool backedUp = false;

using (Stream stream = new FileStream(path, FileMode.Open)) {

      int data;

      while ((data = stream.ReadByte()) != -1) {

            byte b = (byte)data;

            if (!beginFound) {

                  if (b == beginPattern[beginIndex]) {

                        beginIndex++;

                        if (beginIndex >= beginPattern.Length) {

                              beginFound = true;

                        }

                  }

                  else {

                        if (beginIndex != 0) {

                              beginIndex = 0;

                              stream.Seek(-1, SeekOrigin.Current);

                        }

                  }

            }

            else if (!xmlStartFound) {

                  if (b == beginStopPattern[beginStopIndex]) {

                        beginStopIndex++;

                        if (beginStopIndex >= beginStopPattern.Length) {

                              xmlStartFound = true;

                        }

                  }

                  else {

                        if (beginStopIndex != 0) {

                              beginStopIndex = 0;

                              stream.Seek(-1, SeekOrigin.Current);

                        }

                  }

            }

            else if (!endFound) {

                  //load up the memorystream

                  if (backedUp) {

                        backedUp = false;

                  }

                  else {

                        xmpStream.WriteByte(b);

                  }

                  if (b == endPattern[endIndex]) {

                        endIndex++;

                        if (endIndex >= endPattern.Length) {

                              endFound = true;

                              xmpStream.SetLength(xmpStream.Length-endPattern.Length);

                              break;

                        }

                  }

                  else {

                        if (endIndex != 0) {

                              endIndex = 0;

                              stream.Seek(-1, SeekOrigin.Current);

                              backedUp = true;

                        }

                  }

            }

      }

}

if (!endFound) {

      Console.WriteLine("No XMP data found");

      break;

}

//load up the xmp

xmpStream.Position = 0;

XPathDocument xmpDocument = new XPathDocument(xmpStream);

XPathNavigator xmpNav = xmpDocument.CreateNavigator();

XmlNamespaceManager nsManager = new XmlNamespaceManager(xmpNav.NameTable);

nsManager.AddNamespace("x", "adobe:ns:meta/");

nsManager.AddNamespace("rdf", "http://www.w3.org/1999/02/22-rdf-syntax-ns#");

nsManager.AddNamespace("iX", "http://ns.adobe.com/iX/1.0/");

nsManager.AddNamespace("crs", "http://ns.adobe.com/camera-raw-settings/1.0/");

nsManager.AddNamespace("exif", "http://ns.adobe.com/exif/1.0/");

nsManager.AddNamespace("aux", "http://ns.adobe.com/exif/1.0/aux/");

nsManager.AddNamespace("pdf", "http://ns.adobe.com/pdf/1.3/");

nsManager.AddNamespace("photoshop", "http://ns.adobe.com/photoshop/1.0/");

nsManager.AddNamespace("tiff", "http://ns.adobe.com/tiff/1.0/");

nsManager.AddNamespace("xap", "http://ns.adobe.com/xap/1.0/");

nsManager.AddNamespace("xapMM", "http://ns.adobe.com/xap/1.0/mm/");

nsManager.AddNamespace("dc", "http://purl.org/dc/elements/1.1/");

XPathExpression dateExpr = xmpNav.Compile("string(/x:xmpmeta/rdf:RDF/rdf:Description/exif:DateTimeOriginal)");

dateExpr.SetContext(nsManager);

string dateTimeStr = (string)xmpNav.Evaluate(dateExpr);

DateTime date = XmlConvert.ToDateTime(dateTimeStr);

Back the the days of .NET 1.1, you'd use a Hashtable to store values with a fast lookup using a key.  Hashtable used a single array of "buckets", which stored the key, value, and a collision index.  The index to the the bucket array is a transform of the hashcode of the key (unless there's a collision).  Long story short, there was no stable ordering to the values in a Hashtable (technically, the order is super-important to the algorithm, but it's not useful on the outside).

Fast-forward to 2.0, we have the Generic Dictionary.  It's algorithm is quite a bit different.  It has an array of Entry<Key, Value>, a private nested class similar to bucket in the Hashtable.  But, the "bucket" data is stored in a separate array of int, which holds the index into the Entry array for that hash.  When you add something to a Dictionary, it is simply added to the Entry array, and the bucket array is the one that's updated and possibly re-ordered.  Long story short, there IS meaning to the order you get from a generic Dictionary (using the enumerator).  It's the order you added them.  This subtle change adds alot of value to Dictionary in my opinion.

It was just a very interesting realization to me, and I thought someone else might find it useful.

As I mentioned earlier, I've been playing with WF.  When debugging in Visual Studio, a program running a workflow pretty much uses 99% of CPU the whole time.  I thought that was pretty scary, especially since I had alot of Delay activities in my workflow.  Well, not to worry, running the .EXE outside of devenv gives you what you expect, a very well-behaved program.

I suspect it has to do with the sweet debugging stuff that's built into the workflow designer surface.  You can add breakpoints to activities.  It's pretty cool, despite crashing on a regular basis.  Hopefully, they'll fix this before RTM.

In tracking down a bug today, I needed to put "control" characters into an Oracle SQL statement.  Since it was difficult to find using available docs and search engines, I decided to put it here to help out others who may be looking for the same thing.  I do not claim this is the best or easiest way of accomplishing it.  It just works, and that's what I needed.

So, here is how to escape control characters (or presumably any unicode character) within a string in Oracle SQL or PL/SQL:

UNISTR('\0008')

This will give you the backspace (ASCII 08) character.

So, the UNISTR function encodes backslash ("\") followed by four (hex) digits as a single unicode character. (actually it doesn't need to be 4 hex digits.  It will stop at the first non-hex digit. But, it will still "eat" 4 characters from the string.)

(Note: this is for XXXX2 column types like VARCHAR2 that support unicode.  I don't know if/how this works on non-unicode types. Also, this is for Oracle 10g.)

UPDATE: Shortly after writing this, I discovered ASCIISTR, which is identical in syntax to and probably preferable in the case of control characters (as opposed to real unicode characters like accents, symbols, etc.)  ASCIISTR is likely compatible with earlier versions of Oracle as well.

I saw an unofficial announcement of the new Windows Live Custom Domains Beta from Dare's post.  It sounded pretty cool, and I have a domain without current mail capability, so I decided to give it a shot.

The "setup" was surprisingly painless, I merely signed up and enabled my existing passport as the "administrator", and it told me the settings I should enter for the DNS mail exchanger (MX) entry.  No problem.  It quickly verified the settings and told me everything was set up.  Fantastic!

I added a new account, and it said it was ready.  Here's where the fun begins.  If you haven't figured it out, you're basically getting hotmail accounts that are hooked up to your own domain name.  That's cool, despite the fact that it's hotmail, which still sucks in my opinion compared to GMail, despite the wider offering of calendaring and reminders.  (Tons of in your face adds and not very many features)  So, now, I have to figure out how to sign out of my current passport account and sign in with my "new" one to test it out.  I don't know if anyone has trouble with that, but it never works for me.  After logging off and logging back on, I visited hotmail and was in.

Unfortunately, Hotmail still has no "forwarding" functionality that I'm aware of, so this really isn't giving me anything useful.  Now, I can get email at a nice address, but I still have to go out of my way to go get it.  I'd rather have it forwarded to wherever I want.

Additionally, now I have 2 passports, and there seems to be no support for being logged into two passport accounts simultaneously, bringing up a whole variety of issues. And I can't get rid of the original passport because it is the "administrator" account.

In my opinion, the most likely usage scenario for this is bloggers or other content authors who have purchased their own domain, and want the "coolness" and other identity benefits of getting mail at that domain.  The current functionality doesn't seem to support that scenario in an optimal way.  I think it would be far better to be able to associate other email accounts with the same passport and be able to specify either to receive that email through hotmail or to forward to another account.  I don't want to create and manage a bunch of passports just to have a couple of extra email aliases.

As an update to a previous entry, Peter Golde has updated the PowerCollections library to be compatible with the final release of Whidbey.  Yeah! I can go back to using the official release!

We made a discovery today that the most recent release (1.0) of the PowerCollections library is not compatible with the .NET 2.0 RTM.  This is because of the changes in System.Collections.Generic.KeyValuePair<TKey,TValue> to make it immutable. (The change was actually in a release candidate before the final release.  Basically the public Key and Value fields were abstracted via readonly properties.) A few of the PowerCollections classes make use of the formerly-exposed fields to set or modify the key and value.

So, I made the necessary changes to the project so that we could continue to use it.  But, I'm surprised that Peter Golde, who owns the project, has not at least acknowledged there is a problem and provided an updated version.  I'm tempted to post my changes because there seems to be alot of people who are complaining about it in comments and the forums. But I wouldn't want to cause a "fork" in the project at all, especially for such a trivial thing.  I know there are several other things that ought to be changed to accomodate changes in the release, like arrays properly implementing generic IList.  There is a specific workaround for the issue in PowerCollections.

[UPDATE:] This has been fixed officially!

Another geeky post.

We built our own query system several years ago.  It used the concept of "query by example" with templates that were interpreted into SQL.  You could generate the equivalent of an "IN" clause by assigning something that implemented System.Collections.ICollection to a template property.  Recently, with the introduction of generics and the generic collections, we decided to relax that contract to IEnumerable since that's really the least common denominator of all collections.  This produced some hilarious results.  As you can imagine, anywhere where we were using strings to specify template values, they were being interpreted as collections of characters.  This manifested itself mostly by having queries return nothing, since our system properly handled Chars and treated them as strings.

But, this reminded me of a similar incident a few years ago that involved a framework that rendered objects as comma-separated values. So, instead of:

Mark, Miller

You'd get:

M, a, r, k, M, i, l, l, e, r

I thought since I had been bitten twice, I would write it down.  I usually don't forget stuff I take the time to write down.

So, the moral of the story: Remember that even though System.String implements System.Collections.IEnumerable (and now System.Collections.Generic.IEnumerable<System.Char>), you usually don't want to treat it as a collection.  You may need to special-case it.

Anyone else think of another type like this?

I ran into a nasty performance problem recently.  It involved some AJAXy type dynamic requests.  The problem was that two requests always seemed to be occuring serially rather than taking advantage of the wonders of a multithreaded server and running in parallel.  After much spelunking and debugging, it suddenly occurred to me that the handlers were marked with IRequiresSessionsState to pull a trivial piece of information out of the session.

You may not be aware, but accessing the session usually results in an exclusive lock.  Normally, this isn't a problem since users very seldomly open multiple windows or send simulataneous requests, and sessions usually represent unique users.  But fire off two simultaneous requests, and they will be processed serially.

But, if you mark your page as being "ReadOnly" (or disabled) via the EnableSessionState attribute of the @Page directive (or IReadOnlySessionState for IHttpHandlers), you'll help yourself out.  ReadOnly will get a reader lock on the state, allowing multiple readers access, while disabled (via "false") will not lock at all.  In addition, if you're running your session state out-of-process, disabled will keep you from incuring a hit from the db or other store access.

Just an interesting bit of information that thought would be useful.

What a geeky post title.  Well, I guess it's a geeky post.

Based on the number of people actually in the bathroom at work when I'm in there, I would think that two people entering/exiting the bathroom at the same time would be a rare event.  Yet, I've come to expect the door to magically open in front of me as my hand nears the handle, or swing violently at my face as I exit.

I always find it amazing the number of things in real life that have dualities in software.  In software, we'd solve the problem of collisions in the doorway by taking a lock on it (although locking on a public object is not always a good idea).  This becomes a problem in the real world because of visibility.  The bathroom door has no window (which is probably good), so you can't see when someone is right on the other side.  You seldom have a collision when the door has a window because you can see who else is going to try to use the door before you can get there.  There are tons of "shared resources" in the world where we've created synchronization mechanisms of various authority.  I think we could learn a thing or two by observing them.

For instance, when roads intersect, we have several different ways of dealing with collisions (we sometimes call them that in the software world as well.):

• Raw intersection - No traffic signs or anything. Typically found on dirt roads where traffic is scarce and the probability of collision is low.
• Yield - One street is a lower priority and "yields" the intersection to the other road.
• Stop signs - you see where I'm going
• Flashing Lights
• Traffic Lights
• Traffic circle - This one always amuses me.  It is actually a very efficient means of managing an intersection IF the drivers are familiar with navigating it.  If they are not, hilarity ensues.
• Overpass

And there are tons of "in-between" flavors.  Hopefully you see what I'm getting at.  The granularity of synchronization in software can be just as complex.  Simple locks, Mutexes, Reader/Writer locks, etc.

Where am I going with this? I don't know.  I just thought it was interesting how many real-world examples of synchronization there are, and how we avoid them in different scenarios, sometimes delegating that responsibility, sometimes taking a more strict position. Prioritizing etc.

That is all.

• A generic method with the following characteristics:
  • More than one type parameter.
  • A return type composed of one or more of the method's type parameters other than the first one.

Some C# example signatures (Not useful or practical, just simple examples that show the problem):

static IEnumerable<TValues> FindDictionaryValues<TKey, TValue>(IDictionary<TKey, TValue> dictionary)

static TValue FindAValueInADictionary<TKey, TValue>(IDictionary<TKey, TValue> dictionary)

I've been distracted all morning, drooling over the C# 3.0 features.  First, I was upset that they would tease us with 3.0 features before 2.0 is even released yet (and it's still a little silly), but being able to download, play with them, and run them makes a world of difference.  Here's my favorites distilled for those who don't want to check it out.

• Tons of type inference stuff.
• List<int> list = new List<int>();
• becomes
• var list = new List<int>();
• I'm not sure how I feel about "var".  It makes it feel a little like javascript.  I can see how that makes the language less ambiguous, and three letters isn't that bad.
• Tons of initialization stuff
• Initialize any collection-type-thing like you would an array. (new List<int> {1,2,3,4})
• Intialize objects with a named-parameter feel (new Person {FirstName="Mark", LastName="Miller} //not a constructor)
• A few AOP-ish goodies with extension methods - This is extremely cool.  Add methods to string, or anything else.  Very sweet.
• "Fix" the anonymous delegate syntax with lambda expressions
• delegate(int x) {return x+1;}
• becomes
• x => x + 1
• This will be hard to approach for some, but I think it's great
• Anonymous Types - This is excellent for working with data where you don't really need a formal type, you just need to work with some data.
• Language Integrated Query - This is the grand-daddy of them all.  Query against objects, query against XML, query against relational data, query against anything!  All using first-class features of the language!  I'm pretty excited about this.

I'm dissapointed in the lack of first-class duck typing support (late binding).  Yeah, we get half-way there through the use of query and anonymous classes, but there's some scenarios where that seems like hammering a tack with a sledgehammer.

I'm really excited in the innovation I see in the CLR-world.  Sure, lots of the above represent old ideas, but the query stuff represents big innovation in that space that fits right in where the "hurt" is.

Most of my day today was tracking down a problem with our app when we obfuscated it.  We were getting a System.EntryPointNotFoundException in the obfuscated version, while the normal one worked fine. The stack trace contained the following on the top of the stack trace:

at System.Collections.Generic.IEnumerable`1.GetEnumerator() +0

No concrete class, and an assembly code offset of 0. Weird, eh? After tweaking with lots of settings and digging through the IL of several iterations of differing obfuscation techniques, I finally determined the problem.

I had three overloads of a method that each took a different parametrization of IEnumerable<T>.  For the sake of example, we'll say IEnumerable<A>, IEnumerable<B>, and IEnumerable<C>.  In the obfuscation process, calls to any of the three were being changed to the IEnumerable<A> overload.  Sure enough, PEVerify was giving errors about the wrong type being on the stack. (I was going to show the IL before and after, but I just realized that I deleted the non-working one after fixing it. I'll post it if I can find it again)  It seems that Dotfuscator was not distinguishing properly between overloads if they only differ by parameterization types. I have not verified it, but this may have been aggravated by iterator methods.

We're working up some sample code to send to Preemptive so they can address the problem, but debugging it was frustrating enough to post a synopsis of the account for Google to find.

A while back, I showed you my file-based viewstate persistence solution.  Thanks to Google search hits and traffic from Scott Hanselman's analysis, it's been one of my more popular entries.  With ASP.net's improvements in this area, I felt that it was due for an update.  So, here's a quick whack at it.  The usual disclaimers apply.

2.0 adds the notion of "control state" to state persistence, which is very cool.  It's an opt-in mechanism for things that need to survive postbacks even if ViewState is turned off.  In addition, there's some new flexibility with page adapters and such, but we'll ignore that complexity for now and go for a direct port of my old sample, but include the new ControlState mechanism.

public class FileBasedPageStatePersister : HiddenFieldPageStatePersister {

      public FileBasedPageStatePersister(Page page) : base(page) {}

      public override void Load() {

            //let the base class do its thing

            base.Load();

            //get the control state

            object baseControlState = base.ControlState;

            if (baseControlState != null) {

                  //the control state should be our Guid

                  Guid guid = (Guid)baseControlState;

                  //read the contents of the file and set the two states

                  using (TextReader reader = new StreamReader(CreateOfflineViewStateFilePath(guid))) {

                        Pair pair = this.StateFormatter.Deserialize(reader.ReadToEnd()) as Pair;

                        base.ViewState = pair.First;

                        base.ControlState = pair.Second;

                  }

            }

      }

      public override void Save() {

            //create a guid for this viewstate

            Guid guid = Guid.NewGuid();

            //serialize the states into a temp file

            using (TextWriter writer = new StreamWriter(CreateOfflineViewStateFilePath(guid))) {

                  Pair pair = new Pair(base.ViewState, base.ControlState);

                  writer.Write(this.StateFormatter.Serialize(pair));

            }

            //trick the normal system into thinking all it needs to save is the guid

            base.ControlState = guid;

            base.ViewState = null;

            base.Save();

      }

      string CreateOfflineViewStateFilePath(Guid guid) {

            //TODO: put these files whereever you like

            return Path.Combine(Path.GetTempPath(), string.Format("{0}.viewstate", guid));

      }

}

So, we immediately see that it's much shorter.  This is because ASP.net uses a mechanism very similar to my 1.1 solution, so alot of the plumbing is built-in. The only thing you need to do to use it is override the PageStatePersister property on Page and return one of these. Again, we're piggybacking on the "hidden field" persistence mechanism and using that to store our Guid for the request.  Not much different, and I'm pretty happy to say that converting to this model from my old is very simple.

Another interesting idea would be to leave the ControlState in the hidden field, and only store the ViewState in the file. it would be a simple change that I'll leave as an exercise to you.  Then, you could be very aggresive about purging old or large files without worrying about breaking anything (provided of course that you've made your controls tolerant to such a method).

My friend Peter and I often use Remote Assistance to collaborate on projects.  He and I were doing some things today in Visual Studio, and while I was watching, a beautiful window appeared that showed the open files and windows and such, complete with appropriate icons. "Whoa!", I said, "What was that?!"  "What", he replied, "this?"  The window re-appeared.  Turns out, it was the Ctrl-Tab window.  For some reason, I never even thought of using it to browse the open windows. I always assumed I had to navigate the frightful tabbed interface, which just might actually be usable with that little trick.

I felt better when I typed in a type name and Intellisense didn't recongnize it.  A quick right-click->Resolve instantly added the appropriate using statement.  Peter hadn't seen that feature.  Although I guess Ctrl-Tab's been a standard for much longer than Resolve.

This past week, I've found myself writing alot of methods with the following pattern:

IEnumerable<T> DoSomething(IEnumerable<T> stream);

Then I use C#'s wonderful new iterator syntax to manipulate the stream, or simply intercept and process the data.  I've found this wonderfully useful for alot of the statistical type algorithms that I implement for our project.  It's been especially cool to combine this with a bit of reflection for creating "dynamic filters" based on user input.  All of a sudden, all your algorithms are incredibly flexible with very little effort.

The beauty of this pattern is that it can be chained.  Obviously, there is a practical limit here, but I've found it extremely useful and I thought I'd share it for those who haven't stumbled across this concept before.

Now, if we could only create iterators using anonymous delegates, then I would proceed to drown in my own drool.

[UPDATE] fixed title grammar. Sorry if that pops it back up in your aggregators, RSS readers.

I blogged a few days ago about a crazy problem I had with garbage collection in ASP.net under 2.0.  I finally tracked the problem down to Array.Sort.  I created a framework for working with large sets of data in a dynamic way.  This allows us to group data and do other complex statistical things based on fairly open-ended user input, rather than having to write special case code for each scenario.  It uses some reflection to be able to drill into objects.  When sorting, we pass in a dynamic IComparer that can drill into each object and do dynamic comparisons.  Unfortunately, many of our algorithms were dependent on sorting the data first.  This was causing a huge number of allocations due to drilling in and boxing lots of value types.  And, the nature of the sort causes that to happen multiple times per item.

Under 1.1, we took the hit on memory.  It didn't take that long, and it was soon collected.  Under 2.0, the GC seems to recognize this allocation pattern early and proactively begins collecting aggresively.  The problem seems to be that this slows down the sort tremendously, to the point where it essentially comes to a halt.  After a long time, you get a bizarre looking exception about a failed sort from deep in the framework.

What I can't figure out is, during all this time, there's still plenty of memory, and activity on other requests is not impacted horribly.

Well, I took a long hard look at some of the operations, and re-implemented them with a hashing strategy rather than relying on a sort.  As it turns out, for most cases, this is much more efficient.  Long story short, overnight stress testing turned up with clean results this morning.

This is awesome.  Joe Duffy reveals what I had suspected.  A change to the CLR that will fix the issues I've previously discussed about nullable types in version 2.0!  From his blog:

The core of this change is that the IL box instruction has been modified to recognize Nullable<T>s. For non-Nullables, behavior remains the same; but upon seeing one, it inspects its HasValue property. If HasValue is true, box peeks inside the structure, extracts the T value, and boxes that instead; otherwise, box simply leaves behind a null reference. Obviously, unbox has also been changed to allow nulls to be unboxed back into Nullable<T> structures. This had a rippling effect in the CLR codebase and also required changes to late-bound semantics to mimic the static case.

This is fantastic, and reveals just how strong Microsoft's commitment is to the development community.  I gave my feedback on this before.  I felt it was a problem that aware developers could understand and live with, but I felt that novice developers would struggle with it, and ultimately it would make the feature, and the platform less understandable and approachable.

As you can tell from my earlier post, I had pretty much decided that MS would be unable to fix it at this late stage in the game, and that would be a shame.  But, thanks to some good decision making, they've done it.  Also, the solution was quite similar to my suggestion, I'm happy to say.

Be sure to follow through and read Somasegar's post on the subject.

We're migrating an app to 2.0, and I've come across a rather bizarre behavior.  Basically, after running for a while, the app will begin taking up 100% of one CPU.  Perf counters would indicate that the process is in GC (% Time in GC is very high).

Now, some background.  This is an ASP.net app. We've got very aggresive caching such that the static memory footprint is about 600MB. When things first get going, everyting behaves wonderfully.  Then, at some point, the GC gets hungry or something and starts chewing up cycles.  The heaps never go down.  No allocations are being made.  Nothing tangible seems to be going on.

When requests comes in, they are handled normally, and the GC seems to "get out of the way".  (DB wait time is accompanied by 0% CPU) But after the request is completed, it's back up to 100%.

This app worked fine under 1.1.  And I guess there's an argument that says it still does.  It just doesn't play nice with anything else on the system.  I'm just kind of writing this to get this problem out there in case anyone else is seeing it and is searching for a solution.  I'll probably also ping a few folks at MS to see what they have to say.

[UPDATE] Check out this update.

A few months ago, I had an entry that showed how to easily get to 32-bit explorer land within 64-bit WindowsXP using Internet Explorer to navigate the file system.  This makes it possible to use 32-bit shell extensions.  In Vista (and possible the IE7 beta on XP), this behavior appears to have changed.  Navigating to the file system simply opens up a new explorer window in the current explorer process rather than loading it within the IE frame.  This is the only gripe I have so far on Vista, and it's probably an IE change, and probably by design.  I'm sure this was changed to reduce the surface area for possible phishing schemes I can think of, but it sure is annoying.

I've seen some possible workarounds on TweakVista and elsewhere, but none that I found satisfactory (I don't want to run the 32-bit explorer as my shell).  So, I may see about throwing together a little app that can host a 32-bit explorer process so I can use my shell extensions like TortioseSVN.

Sorry for the sparse updates.  I've been head-down on quite a few projects recently, so most of my ramblings have taken the form of internal blog entries and emails.  For the last several months, I've been pushing my co-workers to think about migrating to Whidbey (The next version of Microsoft's .Net Framework, the CLR, whatever you want to call it) sooner than later.  Most people know I'm an early adopter of new technology, and always take my suggestions as the ravings of someone infatuated over new things.  However, we're finally moving that direction for one of our biggest tools as a result of some performance optimizations I was able to do with new features provided in 2.0 (namely generics, iterators, and some new asp.net goodness).  The performance opportunities alone are return enough for the investment.

Rico shares with us an overview of these opportunities, and it's a good enough list that even a manager can see the benefits, many of them handed to you without any additional work.

I had occasion to build a shell extension the other day to add a column to explorer's detail view.  I was getting all mired in the interop between the Win32 interfaces and such until I found this CodeProject sample.  Big props to it.  It's one of the finer CodeProject samples I've seen.  I thought this entry might send some more Google juice its way for other people looking to do shell extensions in the managed world.

Anyway, it was pretty easy after seeing a working version of all the interfaces and structures for the interop.  There's alot of tricky legacy stuff in there like interesting byte packing schemes.  Hopefully, with WinFX, this will be much simpler to do from managed code in the future.

A large part of my time at work recently has been addressing scalability issues in some applications.  Most of these stem from poor use of memory.  Most of these applications were proof of concept that made their way into production code for whatever reason.  More and more we identify problems and when we identify them to the developer, this is the typical exchange:

Me: You're taking too much memory here by using type Xxxxx when you could have gotten by with type Yyyyy

Dev: I have to use type Xxxxx, that's what the data is.

Sometimes, you just have to take a step back and look at your bytes.  Sometimes, you need a profiler, but sometimes you can just do some math.

For example, if I have an array of dates that correspond to events, a logical choice for a datatype is System.DateTime.  However, if you've got millions of events, System.DateTime is relatively expensive.  Especially if you consider that your event times only have resolution down to the nearest minute and only represent times within the last several weeks.  The range, or domain of DateTime (ticks since the epoch) is overkill for your circumstances. You may have been able to get away with UInt16, which would have been 2 bytes instead of 8, which is a huge reduction. (Provided that you've shown this array of DateTime to be a significant part of your memory consumption.)

I'm sure there is some official name for this, but I'll refer to it as "constrained domain", where you know something about your circumstances or usage that allows you to reduce the range or domain of a concept in order to store it as something smaller to improve memory consumption.  Remember, just because a big, rich datatype makes things easier, matches a db schema, or is named the same as the concept you need to store, doesn't mean it's a good match for your scenario.

There are, of course, other considerations to make.  The Y2k problems were the direct result of taking this concept to the extreme (of course resources were in much shorter supply then than they are today).

So, when you are looking for places to trim your memory usage, look for places where you can constrain the domain of a concept and store it in fewer bytes.  Also, if you're in CLR-land when you do this, look for boxing value types.  Remember, bytes in a System.Collections.ArrayList take up 5 bytes each, not 1. (9 bytes in 64-bit land) (Yeah generics)

I ran across a known bug in MSBuild in Whidbey beta 2 that caused me considerable headaches today.  It seems beta 2 will not copy project references to the bin directory of web projects when compiling with MSBuild.  This issue will be fixed in the final release, but I thought some Google juice might help someone else find the definitive answer with less frustration than I had.  Here's a relevant forum post on the subject.

In the meantime, I'll simply add a copy task to copy the relevant assemblies to the bin directory.

Back last year, I outlined a problem I had with calling Response.End from a thread other than the thread that handled the request.  It stemmed from a progress mechanism we had implemented that had the ability to cancel long-running tasks when the user hit stop or closed the browser.  As it turns out, that wasn't the whole story.

We recently added a few reports where, the majority of the time, gathering the data takes over a couple of minutes.  While doing memory and performance optimizations, we noticed our worker process was recycling with the following message in the System event log:

Googling for this error doesn't do you much good since it could be caused by a variety of reasons.  Most of the advice I came across wasn't well thought out and made quite a few bad assumptions.  So we added some more debug logging and determined that this was happening when our canceling mechanism kicked in when a user decided they didn't really want to wait 5 minutes for the report.  I was greatly puzzled by this since this feature had been tested thoroughly and had been running in production for some time.  Looking back through the server logs, it was evident that it had been happening all along, just not very often. We had just gotten the performance on the vast majority of pages to be very good and it wasn't an issue.  The problem only came to the surface when we added the report that always takes a while.

Here's the problem.  If the handling thread is aborted, it causes a condition that IIS considers to be bad and that forces the worker process to recycle. (presumably, there is some communication that doesn't occur) This doesn't happen when Response.End() is called because it passes a special exception as the exception state to Thread.Abort.  The HttpApplication catches ThreadAbortException and checks the ExceptionState.  If it is an HttpApplication.CancelModuleException, it knows there was either a timeout, or Response.End() was called, and it cancels the Thread.Abort by calling ResetAbort which allows the thread to continue running at that point.  I thought that was pretty slick.

When I was aborting the handling thread, I was on a different thread, so I had to call Thread.Abort manually, so CancelModuleException was not being used, so the thread was ending completely and causing the recycle.  Since HttpApplication.CancelModuleException is internal (and rightly so) I could not simply use that mechanism.

The good news is that the Unload event always happens (for all practical purposes), even when the thread is being aborted.  So I added my own PageIsCancelling property to our base class Page and check it, along with the current ThreadState in Unload, and cancel any pending abort if the page is canceling.  So, the abort is contained within the callstack of the page, and the thread stays alive and all is well.  No more crazy recycling.

As an aside, it seems this is aggravated by multiple processors, which might explain how it passed testing on the dev's machine.  Although I don't have any proof of this.

I've had a bunch of new pictures in the queue to get uploaded.  Flickr announced that the per-photo size limit had been raised to 10MB (from 5), so I thought I'd go crazy and save my raw conversions as lossless PNG files rather than JPG.  This made most of them between 5 and 10MB a piece.  Then I find out that the currently available uploaders had the limit hard-coded, so I could not upload them.  So, I built my own uploader.  Unlike most of the loaders, this one runs as a service and watches a directory.  You can just drop files into the directory and it will load them.  I used it last night to upload some and it worked great.  It can use sub-folders to apply general tags to the images. I have a couple of improvements to make, like:

• A task tray "agent" that will allow me to "suspend" it when I'm playing Halo, as well as do some configuration stuff.
• Bandwidth throttling - I'd like to specify a cap to make it play nicer with other outbound traffic.
• Better retries - Right now if a picture fails, it goes immediately back in the queue.  It needs more standard retry intelligence, like trying again at successively longer intervals.
• Utilizing the user's monthly limits to stop uploading before some percentage of the limit is reached.
• Priority spools - I've still gots tons of old photos to upload, so I want to be able to use up my limit at the end of the month without managing that process myself.
• Automatic processing.  I'd like to just drop my RAW files in there and have my standard photoshop processing applied.  I need to think about that some more because I usually like to tweak them before sending.  With the huge per-month limit, maybe I could do that and use them as "proofs".  Then delete the ones I was to tweak manually and resend them.  Oooh, and perhaps automatically deleting the one it replaces
• I also don't like having the password in the config file.  I'm going to try to fix that as well.  At least encrypt it a bit.

Anyone got any other cool things it ought to do?

The generic Nullable type in Whidbey and the C# language features it supports are great.  They've enabled me to eliminate gobs of code and simplify things incredibly. But, Joe Duffy's latest request for feedback on some of the "syntactic suger" on this feature brings up some interesting points.  He shows us several different equality tests against a nullable value type and asks if the results are intuitive.  I suspect it's a loaded question, because they are not.  Someone with a good understanding of the framework can fairly easily guess the outcome, but deep down I think they would have to admit it's not ideal.

What it boils down to is what is the purpose of the generic Nullable type.  I originally viewed it as helping to code against scenarios where values can be null like databases.  Without nullable value types, you frequently have to check to see whether you're dealing with a null value, and carry around alot of state information, and Nullable simply does that for you.  I think some of the designers of this feature see it much more broadly as a concept to unify the type system in a more performant way than simply being able to cast to object.  This requires some special casing, as is evident by the special case for type parameter constraints on generic types.  Specifying struct will not allow you to use a generic Nullable, despite the fact that it is a value type.  The problem is that the special casing stops short of completely abstracting the concepts.

The problem can readily be seen in Joe's second two tests, boxing a valueless "int?", and using the equality operator "==" against a generic argument that is instantiated as "int?" and has no value.  Both do the opposite of what you might expect if you don't think through it first.  Both do not equate to null.

I pondered this for quite a while before deciding how bad this was (and I'm still not exactly sure).  For me, this is no problem since I understand the issues at work.  For the future of the language, the outcome is not so clear.  The real question is, "Is a value-less value type different from a null reference?"  I don't think it should be.  If you cast a null string reference to object, does it retain its identity as a string? No. Can you cast it back to a string reference? Yes. So, should there be a difference in behavior with nullables? I'll let you decide.  Then there is the issue of how (and if) to effectively fix it.  Adding a conversion from Nullable to object that makes it a null reference instead of a boxed value type might do it, at least for the boxing scenario.  That's probably a pretty naive solution.  I haven't really thought that through.  For the generic situation, Nullable seems to be already special cased to some degree.  It doesn't seem like it would be too much of a stretch to generate appropriate code for Nullable.  The big problem here is that since this happens at JIT time, it's no longer a C# feature. Unfortunately, it's likely too late to address the issue in the in any real way, and that will likely set too much precedence to change it in the future if a solution could be reached.  Perhaps some FxCop rules could find these types of situations and at least flag them so they could be looked at more closely.

We have a big configuration file that maps business logic concepts to their currently used implementations so we can dynamically switch these out.  Under ASP.NET 1.0, all those implementations were in assemblies with "well-known" names (meaning we know them deterministically before compile-time).  Some of the implementations were in nested classes of user controls (we'll save the debate over whether that is a good idea for another time).  This means that the ASP.NET compiler is now in charge of them, which makes their names more difficult to discover.

Luckily, there's someone looking out for us here.  The System.Web.Compilation namespace has all kinds of goodies to help us out here, namely the BuildManager class.  It has a GetType overload that at first glance appears to do exactly what we want, unfortunately it only seems to work if the assembly in question has already been loaded.  This is not usually the case when our configuration code runs the first time.  Instead, you can use a combination of GetCompiledAssembly and good ol' Assembly.GetType.  So now, instead of knowing the assembly name, we need to know the virtual path to the compiled control. So here's a snippet that does generally what I want:

if (!String.IsNullOrEmpty(urlString)) {

    System.Reflection.Assembly assembly = BuildManager.GetCompiledAssembly(urlString);

    theTypeIWant = assembly.GetType(typeString, true);

}

else {

    theTypeIWant = Type.GetType(typeString, true);

}

John Lam claims that C#'s "using" blocks aren't "extensible to let us do arbitrary things".  He shows us an example of doing arbitrary things in statement blocks in Ruby.  I'll agree that his example is slick, using a continuation to yield control to the block, but there's certainly nothing keeping you from getting same functionality in C#.  The "using" block is simply a mechanism that utilizes the Dispose pattern to do things when you're done with them.  I've done everything from the standard releasing of resources to implementing reader/writer locks with them.

To speak to John's example, there's nothing keeping you from creating a wrapper or subclass that calls SetInfo in the Dispose method to accomplish the same thing.  Or, you could create a generic factory pattern that uses delegates to define the creation and disposing actions: 

public delegate T Creator<T>();

public class DisposingFactory<T> : IDisposable {

    Creator<T> creator;

    Action<T> disposeAction;

    public DisposingFactory(Creator<T> creator, Action<T> disposeAction) {

        this.creator = creator;

        this.disposeAction = disposeAction;

    bool valueCreated = false;

    private T value;

    public T Value {

        get {

            if (!valueCreated) {

            return value;

    public void Dispose() {

        if (valueCreated) {

With Don Box doing all manner of crazyness with anonymous delegates recently, I decided to do some experimenting of my own. While doing so, I ran into an interesting problem.  I was toying with a robust action scheduling mechanism that serializes actions to disk to make sure that they won't be lost in the case where the host process is stopped for some reason.

I was using the generic Action delegate to represent the actions and using anonymous delegates to inject things like action rescheduling.  I was pleased to see that basic anonymous delegates were serialized without issue.  But, when I started to use the closure features of anonymous delegates, it stopped working.  I was dissapointed, especially when I figured out why.

When you create an anonymous delegate without using outer variables, it is hoisted into the enclosing class as a private method with a compiler-generated name (that appears to be generated based on a combination of the signature and some randomness).  For example:

[Serializable]
public class Example {
  void Test() {
    Action action = delegate(DateTime scheduledTime) {
      Console.WriteLine("blah");
    };
  }
}

This actually turns into something like:

[Serializable]
public class Example {
  void Test() {
    Action action = new Action(CompilerGeneratedName);
  }

  void CompilerGeneratedName(DateTime dateTime) {
    Console.WriteLine("blah");
  }
}

Actually, it does some niftyness to cache the delegate, but that's not important for this example. As you can see, serialization isn't a problem here.  However, if you use an outer variable within the method, the compiler hoists both the method and the variable into a nested class so it's available later when the delegate actually runs.  For example:

[Serializable]
public class Example {
  void Test() {
    string foo = "blah";
    Action action = delegate(DateTime scheduledTime) {
      Console.WriteLine(foo);
    };
  }
}

This turns into something like:

[Serializable]
public class Example {
  void Test() {
    CompilerGeneratedClassName something = new CompilerGeneratedClassName();
    something.foo = "blah";
    Action action = new Action(something.CompilerGeneratedName);
  }

  private class CompilerGeneratedClassName {
    public string foo;

    void CompilerGeneratedName(DateTime dateTime) {
      Console.WriteLine(foo);
    }
}

Pretty cool, eh? Yeah, so what's the problem?  Well, the nested class is not marked with SerializableAttribute, so it can't be serialized, which makes the delegate serialization fail.  This seems like a big problem to me.  There may be a legitimate reason for this, but I don't see it.  I think the compiler ought to decorate the nested class if the enclosing class is also decorated.

There may be a way around this, but I haven't found it yet.  Any suggestions?

[Update] If you think this is a problem too, then vote for it.

We're making the move to 64-bit at work, and one of the things that's annoying me the most is all the 32-bit shell extensions that I rely on and didn't realize it.  For example, we use Subversion for source control and use the excellent TortoiseSVN shell extension to work with it from explorer.  Since it's a 32-bit extension, it doesn't show up in the context menu.

There are two workarounds I've found for using these 32-bit extensions.

1. Use the 32-bit explorer.exe in the SysWow64 directory.  You have to have the "Launch folder windows in a separate process" option turned on, otherwise it will just see that explorer is already running and start a new 64-bit window instead of a new 32-bit process.
2. Use 32-bit IE.  There's already a shortcut to it by default in the start menu.  Just fire it up and navigate to the filesystem instead of a web page.  Voila! 32-bit extensions start showing up.  I like this method since I don't have to have a bunch of explorer processes for each window.

There's probably a cleaner way of using the 32-bit explorer, but I haven't figured it out yet.

Last night, I completed my STDF file parser that I started over the weekend.  I'm pretty happy with it.  I now fully support all of the version 4 spec.  It's written with the Feb. 2005 CTP of Whidbey.  It was primarily a good example of a "real world" application, so I wanted to use it to try out some of the spiffy new features and see how they might fit into an application that was not built for the sole purpose of showing them off.

I'm not entirely sure what I should do with it now that I've built it.  There's not much non-commercial demand for something like that that I am aware of. I might be persuaded to license it for use commercially once there is a "go live" license for Whidbey.  I'm sure it will need some testing to work out issues.  I'll probably build a viewer using it and use it at work to get the kinks out of it.  Anyway, I thought it might be useful to capture how some of these features played out in a real scenario.

Lightweight Code Generation

I used attributes to annotate my record classes in order to define the "on disk" layout of the record.  I originally put the attributes on the properties of the record, but decided to move them to the class level so that I could define the fields that are only used for parsing, and not useful after that.  At runtime, I register the record types with the parser, which uses lightweight code generation to generate converters from unknown records to the concrete records.  If you're not familiar with LCG, it is essentially Reflection.Emit without the overhead of a dynamic assembly, module, or type.  If you're unfamiliar with Reflection.Emit, it is essentially generating executable IL code on the fly, which has may benefits over generating C# or some other language and running it through the compiler with CodeDOM or soemthing else. (If you're not familiar with IL, then this entry is somewhat irrelevant) Having dealt with assembly on lots of different instruction sets from 68000 to x86 to DSPs and microcontrollers, I must say that working in IL is wonderful.  I was a bit worried about the startup time for the parser, but it seems to happen very quickly.  I'll need to experiment more to come up with the overhead of the dynamic code.  Again, I was disappointed in the lack of debugging support in LCG.

Generics

Unfortunately, generics did not work into the equation near as much as I had hoped.  I did find them very useful in places where I would normally pass a Type.  Now I can have a much stronger contract on such methods using generic methods with constraints.  I must say that I love the generic collections.  Working with strongly-typed data without having to generate the classes is very nice.  I also like being able to do a custom sort with an inline anonymous delegate with closure-type semantics rather than have the overhead of a separate class with the code in a different place.

Iterators

Iterators came into play in several places.  In the pull-based record parser, it was simple to implement IEnumerable.  I believe it only took 3 or 4 lines of code.

Delegate type inference

I don't remember what the official name of this feature is, but instead of having to do something like SomeEvent += new EventHandler(SomeMethod), you can just do SomeEvent += SomeMethod.  This seems small, but I appreciated it quite a bit.

Visual Studio Enhancements

These were a very pleasant surprise.  I hadn't thought about the exercise as testing the VS enhancements.  The snippets were the most useful improvement in this project.  Implementing properties didn't make me want to rip my hair out. Just type 'prop' and hit tab and fill in the blanks.  The built-in snippets for things like exceptions, attributes, indexers, etc. all work really well.  The one for implementing an indexer/iterator as a nested class was very interesting as well. It was also easy to create my own snippets.  I also found the strongly-typed resources to be very handy.

The debugging stuff is awesome.  I love being able to mouse over variables and drill into them dynamically.  The little popup windows for exceptions are cool, but don't really give you any good information that you couldn't get before.

I found the refactoring support to be extremely useful, although it was a little hard to shift into the mode of making sure everything compiled all the time.  Before, I would jump around alot in the code, so I would leave things in a state where it didn't compile.  It worked great once I got used to it.

I had two projects over the weekend.  One was to get my bathroom ready to install a new bathtub.  The other was an experimental coding project.

The bathroom preparation went fairly well.  I ripped out all the cabinets and countertops.  I'm glad my brother showed up unexpectedly because that countertop was incredibly heavy.  We also got alot of the carpet ripped up.  I'm going to be putting down new flooring as well.

On the coding front, as an experiment in adopting new features in Whidbey, I implemented a binary file parser for Standard Test Datalog Format (STDF) files.  These files make up 99% of the data we work with at work and that fill our many-terabyte test result database.  We have a fairly complex parser and db loader framework, implemented in C# on 1.x.  It works very well, but it was written early on in our adoption of .net with little knowledge of what the CLR could do for us. So, my experiment was basically to see how new features in Whidbey, along with my now deep experience in .net, could make the parser better.

STDF is record-based.  The spec defines alot of records, and leaves room for user-defined records.  The new parser reads chunks of the file based on the record headers and produces "unknown records".  I define the record layouts using attributes on record classes.  Then, the parser uses LCG (lightweight code generation using DynamicMethod) to generate converters to read the content of the unknown records into the concrete record classes (based on the attributes).  The benefit of using LCG is that record types could be registered or removed on the fly and the GC could collect the generated code.  I could have just as easily implemented it using on-the-fly interpretation of the attributes.  I'll measure and see how the performance works out.  The parser is pull-based, meaning that you ask it for records, or alternately just "foreach" through them using an iterator-based IEnumerable implementation, which is pretty sweet.  On top of the pull-based parser, I built an event-based "processor" where a consumer can register to receive certain record types.  This is the model used in our current parser, but after the XmlReader vs. SAX discussions, I thought exposing the pull-based approach was the right thing to do.

I had a few challenges, which I think represent work for the next version of the CLR:

• Endian-ness - To my knowledge the framework does not have any mechanism to work with binary data with non-native endian-ness.  The STDF is written in whatever endian-ness is native to the platform, so the parser must adapt.  This was a simple enough problem to solve, but now that most of the other gaps have been filled, endian-ness represents a hole in what the framework provides.
• Generics' proliferation - Generics are great, and saved me tons of code, but they have not made their way into the rest of the platform where they could be leveraged.  For instance, if I create a RecordField, there's not a simple way to do something like BinaryReader.Read() to actually get one, so I was forced into tons of ifs and switches, and passing Types around to get the work done.  It just didn't feel right.
• LCG debugging - From what I understand, this was cut from Whidbey.  The workaround for me was to have two generation paths.  One would do LCG, and the other would do traditional Reflection.Emit that could be debugged and PEVerified, etc.  The problem with this was the argument were not aligned between the two.  When doing the traditional Reflection.Emit, ldarg.0 would give you the "this" instance, which didn't exist in LCG.
• Handler registration (Generics compatibility) - Ideally, record handlers should work with concrete record types, but the way generics work a Converter is not assignable to a Converter even though Mir : StdfRecord.  Of course implementing that would complicate many things.  Interestingly, delegate(UnknownRecord unknownRecord) { return new Mir(); } will satisfy both delegate types.  So, this was just frustrating that Generics didn't help me out in solving my record handler registration problems.  There may be a solution that I'm not seeing here because of my approach.  Any ideas?

Oddly enough, I spent about equal time on both projects, but I seem to have alot more to say about the later.

[UPDATE] I realized that the entry box swallowed some of my generics syntax, so I fixed that, as well as fixing some minor spelling and grammatical errors.

The other day, Don Box announced the availability of the Indigo Community Technology Preview docs.  Now the builds are available on MSDN! (I suppose only to MSDN subscribers)  I've been looking forward to this for a while now.  I'm definitely going to be playing with it this evening.  I suppose I'll have to give up some Halo time.

For those who don't know what Indigo is, it's the unified programming model for building message-based, service-oriented applications.  It abstracts all the trickiness from you.  I won't get too much into it because the documentation does a fairly impressive job of explaining what it is.

I've been toying with ECMAScript for the past week or so, mostly to update some dynamic user interface elements in a web application we have.  Every time I use it, I'm always impressed at the cool things I can do with it.  It also makes me wish that Microsoft had not messed with it as part of the CLR (JScript.NET).  Their newest version of JScript seems to merely make it into an alternate syntax, and not really take advantage of what it could do as a prototype-driven language with closure support.

I found this article that claims that it is the world's most misunderstood language.  I think I agree.  It's incredibly powerful, but very underutilized because of the confusion surrounding it.  Closures coupled with prototypes can yield some very slick code if you understand what you're doing (and I guess maybe that's the problem).  All you really need is a nice IDE to help you manage your objects and their prototype chain.

I'm also getting more into Python and Ruby, which are also very slick.  I'm just not sure how they should fit into a project.  In general, a project with fewer languages is more maintainable.  But imagine trying to build a skyscraper when your only tool is a hammer that works really well. (OK, maybe hammer is not the best analogy.  Maybe a blowtorch?  I don't know)

[UPDATE] I just had to include this link, where the author shows us how to do lot of different code reuse patterns, including multiple inheritance in Javascript.  My favorite quote:

This large set of code reuse patterns comes from a language which is considered smaller and simpler than Java

I've been playing with the Flickr API alot, experimenting with some different things that I have planned for the future.  One of the things I'd like to do is to create context sensitive image collages on the fly for my blog entries.  Thought I'd show you this cool example from a first-pass collage creator.

Of course most collages wouldn't have this many pictures, but I thought it was cool.  I need to do some randomizing.  I'm filling from the upper left corner in diagonals for reasons you'll see when I get it working the way I want.

I've also been playing with some client-side Flickr stuff using XmlHTTPRequest to dynamically load images for a slide show type thing like MSN spaces has.  I might upload an example of that later.

I just got back from Brad's talk at the Austin .NET User's Group meeting.  It was my first meeting, and I felt a little bad about not being a more active part of the local community.  It was a solid, interesting talk, even though I was pretty familiar with both topics (exceptions and memory management).  I managed to ask an intelligent question about a change in Whidbey regarding unhandled exceptions on all threads terminating the app rather than on the main thread only.  My question was on the ThreadAbortException, which seemed like a monkey wrench in that plan.  Turns out that there's nothing to worry about there as that exception is already pretty special.

Despite the great technical content, these were my favorite quotes (emailed to myself via my mobile so I wouldn't forget them:

• Everyone should be using ... using (referring to C#'s using blocks)
• It's like feeding peanut butter to a dog (referring to using some of the new GC (garbage collector) methods to help in memory management.)

I'm always really impressed with the CLR team's desire to do the right thing with the framework and to help developers do the right thing. Thanks for adding Austin to your "world tour", Brad. BTW, the slide template seemed like the right choice to me.

Whew, what an ordeal.  I finally got my Adobe Photoshop CS in the other day, and last night, I created a "Droplet" for batch converting all my raw files from my Canon Digital Rebel. Droplets are a really sweet feature that lets you create a little executable from recorded actions for batch operations.  So, I burned through about 2000 images, creating big jpgs suitable for uploading to my newly upgraded Flickr account.  It was getting pretty late, so I fired up an uploader an set it to upload my 1GB limit (roughly 900 pictures).

This morning, I checked my account, only to find that none of the metadata had been uploaded.  All my images appeared to have been taken on Feb 11, 2005...uh oh.  So, I set out to find out what happened.  Turns out, Photoshop saves the metadata in it's XMP format within the file. XMP is simply an rdf encoding of the data in an XML payload within the file.  It's actually pretty cool, but Flickr doesn't read this data yet.  So I set out to "fix" my pictures, since I can't upload anymore until next month and I have lots more to upload.

After looking at lots of libraries and Adobe's XMP SDK, I decided it would be easy enough to pull the data out myself.  So, I built a little app using my FlickrApi library I just created that would blast through my uploaded pictures, find the corresponding image on my local pc, pull the xmp data out of the file, and set the "date taken" on the Flickr site.  That way, I can at least organize them more easily.

It worked perfectly.  It blasted through about 900MB in less than a minute.  Look for the pictures as I tag them, annotate them, and change them from private to public.  I'll have to see if there's a way to have Photoshop preserve that data next time because I'd really like to have the rest of the metadata available.  I'll probably make my XMP parser available as well if anyone's interested.  As far as I know, there is not another managed implementation available.

Last night, I made alot of progress creating a .NET API for Flickr.  There doesn't seem to be alot of activity on the Flickr.NET project, and I don't like some of the decisions they made in the design.  They seem to have taken a direct wrapping approach to it rather than designing an API that fits in the .NET world.  It's amazing how spoiled you get when you're used to using good API's.  You start saying things like, "You mean I get an array back from this method?  Eww."  or "That's a silly name for that member."  Of course, it's probably appropriate to note that Flickr.NET seems to have been created with an emphasis on uploading.  I'm more interested in sifting through the metadata.

Anyway, I've got most of the functionality that I'm interested in implemented, and I'll be doing the rest a little at a time.  I've used the design guidelines fairly strictly, and I think I'm coming up with something that's really approachable for .NET users, and is still correlated enough to the Flickr API documentation that the parallels are easily discoverable.  You may notice some bizarreness here as I play with the different kinds of integration I have planned for my blog.

This is one of the coolest things I've seen in a long time.  Try my flickr name (marklio).  It lets you drill through my contacts and see their information.  Keeping clicking and you'll see all kinds of cool things.  It really helped me see Flickr as a web service rather than just a photo sharing site.  It gave me alot of ideas.  Things like using tag matching to automatically creating blog entry backgrounds that are relevant to the content.  I'm going to be playing with that.

Their API is really nice and well-documented.  Some people made a .NET wrapper for the API, but it's not up-to-date and I wasn't really impressed with the design.  I'll probably roll my own and just implement the things I need as I go.

In an earlier post, I mentioned that I wished HttpModules were more scriptable in ASP.NET.  You can easily create Controls, Pages, HttpHandlers, and web services all without having to compile anything, but as far as I know HttpModules have to be compiled and registered in the web.config in order to start working.

So, I created a ScriptableHttpModule.  You register it in the config the same as a regular module, but it allows you to create .asmodx files that are compiled and called just like regular modules and give you the same kind of dynamic compilation model as the other .asXx files.  I'm still tweaking it a bit, but it looks pretty promising.

After some banging of my head on a wall, I found what I believe to be a bug in Oracle's .NET data provider (or ODP).  Or, at the very least, it is some non-intuitive behavior.  We have a byte-based enum that is stored in the db for a reporting tool.  As we all know, bytes are unsigned, but somehow a bunch of my values were negative in the database.  Closer inspection indicated that my values had been treated as if byte was signed.  After a thorough review of my code, I fired up Reflector to dig through the ODP code.  After a little while, It became clear ODP was treating my byte as if it were signed.  Although the actual problem is in unmanaged world, here's a troubling method that's called during the binding process:

internal unsafe void SetPrmValCtx(byte b, int index)
{
*(((sbyte*) (this.m_pOpoPrmValCtx.pBltVal + index))) = b;
}

It was easy enough to work around, since our API is several layers removed from ODP itself.  But annoying nonetheless, especially since using an sbyte (signed byte) as a parameter value will throw an exception.  Just thought Google might be able to help someone else out in the future using this information.  in the meantime, I'll make sure Oracle knows about this little annoyance.

I use a dynamic DNS service to host my DNS entries.  Back when I started hosting my blog, I was using marklio.dnsalias.com instead of www.marklio.com.  Since I still have that name pointed at this server, some search engines still have not let go of that old dns name.  In a few months, I'm going to have to move this to a hosted environment (it's currently just running on a server at my house), at which point the two names will point to different places.  I've been strategizing the move for a while now, and I've just implemented step 1, which is to ween http traffic from the old alias.

I've done that by implementing a permanent redirect for any request to the old name.  I created a little IHttpModule that hooks into the BeginRequest event, and looks for the old dns name.  If found, it returns the status 301 (permanent redirect), and dynamically builds the appropriate request to www.marklio.com.  It seems to work well.  Hopefully the search engines will handle this appropriately.

For any interested, here's the relevant code:

Under duress, Brad Abrams recently posted a set of internal coding guidelines to his blog, meant mainly to be a "tie-breaker" in the holy war type arguments that result from seasoned developers who love their own coding styles. I'm a big fan of the work Brad and his team have done on the CLR and their focus on consistency in the API. And, even though there are one or two I don't care for, I'm a huge supporter/follower of the design guidelines.  That's why I was surprised at how many of the internal guidelines I disagreed with.  Roughly 50% of them made me recoil in horror.  Of course, now I understand why I don't like the Visual Studio defaults.  I won't get into which ones I like/dislike because that's not a productive area of discussion.

Now that I've had a couple of days to let them sink in, I realize why Brad was reluctant to share them.  I think they may cause more controversy than solve.  The big questions is how useful will these be in helping my team be more productive?  At this point, I don't think much.  I think we'll stick to the public API guidelines and use the somewhat informal guidelines our team has created for the internal stuff.

I've had my performance hat on at work for the last couple of weeks, optimizing memory and CPU performance ahead of the deployment of a new system.  I've been pretty pleased with the improvements we've made.  We've got alot of in-process caching that increases speed incredibly, but has a hefty footprint.  We were able to get that down quite a bit with some pretty clever ideas.

One of the major pushes was to reduce references, especially boxed references.  That has proven a very effective strategy to reduce memory.  I wish I could just throw generics at the boxing problem, but we'll have to wait a while longer for that.  Another strategy was to take a close look at the data structures that hold the cached data.  I rolled my own AVL tree implementation for a date-based index of the cached data, and I was able to improve both the CPU performance and it's footprint substantially.

We also implemented what I've called a local string intern pool.  We've got alot of redundant string data, and we've used String.Intern in the past fairly successfully, but some analysis of our data revealed some local redundancy that we could use to reduce a 4-byte reference to a single byte that acts as an index into the local string pool.  This will also help keep our memory from bloating on the move to 64-bits when all the references in the system double in size.  (although at that point, we should have loads more memory to work with)  This as the added benefit of eliminating the need for string comparisons in running searches against the cache.

All in all, we were able to reduce the footprint of the cache by about 40%.  I thought someone googling for ways to reduce memory footprint and increase performance might benefit some from this information.  Oh, and so Google puts this in the right context, this is related to: ASP.NET, CLR, C#, DotNet.

I just got finished debugging a crazy problem with some C# code.  This particular code is part of an application that deals with file spooling, and it deals with alot of ambiguous file locking issues by always locking greedily.  Any spooling task that is unable to gain an exclusive lock on a file simply assumes another thread must be working on the file and it leaves it alone.

The developer used a pattern that handles System.IO.IOException and assumes that any IOException must be because of a concurrency issue.  The problem with this is that there are lots of calls that can cause file IO that you may be unaware of.  In this case, there was an assembly binding issue caused by some plugin-style dynamic loading that was throwing System.IO.FileLoadException (Which actually seems to be out of place as a subclass of IOException since it is specific to assembly loading and not IO in general).  The pattern in the code was assuming that, in general, any IOException was not an exceptional event and signified another condition. So, the task never did any work and never reported the exceptional condition.

Eric Gunnerson wrote a nice overview piece on exceptions in C# on msdn.  Some of his guidelines are

Catch the most specific exception

If your code needs to recover from some exceptions, make sure to catch only those exceptions. If you catch more general ones, it's more likely you'll mistakenly swallow exceptions you don't want to swallow.

Only swallow if you're sure

This is really a corollary of the previous guideline. When you swallow an exception, your saying that you understand all cases where this exception could arise, and that the recovery code you're writing handles all of those cases.

Use lock or using if applicable

If you can use the lock or using statements, use them. They make the code more readable and make it more likely you'll do the right thing.

Wrap exceptions if applicable

If you can add additional information to an exception, by all means do so. If I pass a parameter on to another function, it might be useful for me to add additional information about the parameter.

The first two here are obviously directly applicable to the scenario, and would have at least raised some flags if I had first checked all the IOExceptions to see what they encompass.  The solution for me was to take a look at the pattern and reduce the scope of the IOException catches to only those statements that I expected might throw the exception for locking.

[UPDATE] I wanted to note that the exception handling pattern worked great until it was extended by me to a more complicated scenario involving dynamic assembly loading.

Brad Abrams points to an article that shows an upcoming feature of Oracle 10g that appears to be CLR support for stored procedures. Wow, that's very interesting!  I wonder what runtime they're using for their *nix platforms.  Mono?  Or is this only for Windows installations?  I'm going to have to find out.

We use Oracle as the persistence layer of a rather enormous (in terms of disk usage) web-based data analysis tool.  The questions I am really interested in here are:

What runtime are they using? .NET? Mono?  Will this be available only on Windows boxes?  I doubt very much that Microsoft, which is soon to release CLR support in SQL Server (Yukon), would be working closely with Oracle to give them that same feature.  Maybe I'm wrong.

How will they map between Oracle types and CLR types?  This is a huge issue right now just with using Oracle with .NET.  If we're forced to use Oracle-specific types for numbers, it defeats part of the argument in my mind.

I'm going to do some more looking into this, but I'm pretty excited that the CLR would get an endorsement like this from a company that has been pretty far in the Java camp.

Joe has a cool entry on using a “Stream” concept to approach algorithmic sequences.  I realized it's particular beauty when I saw his Fibonacci example, which blows the doors off some code I barfed out recently for this type of problem.

This is one of those “duh” type moments, but I just realized that, even though SP2 installs HTTP.SYS, it stil uses IIS 5.1, which means IIS is NOT using HTTP.SYS.  This would explain my recent trouble and why trying to control port usage with the httpcfg tool didn't seem to do anything.

I'm pretty bummed about this. I probably would have installed 2003 server instead of XP had I realized this sooner.  I'm going to give a serious effort to write a managed Subversion server that uses HTTP.SYS.  That would probably reduce the barriers for using Subversion sufficiently to get more people interested in it.

[UPDATE]  In case you didn't make the leap and realize what the implications are for IIS not using HTTP.SYS, it means you can listen on the same port that IIS is.

I don't really want to become a link blog, but Brian Grunkemeyer has an exceedingly interesting piece on arrays and how they mesh with generics in the 2.0 CLR.  He goes into alot of interesting information that you don't need to know, but knowing makes you appreciate the the work going into it.  I've had the pleasure of speaking with Brian, and thoroughly enjoyed it.

Pardon my ripping off of the title, but I really found that it quite succinctly summarized the post.  Don Box has a pretty valueable little tidbit on doing just that.  Very handy indeed.

I'll have to digest it a bit to see how applicable it is to 1.x.

For a split second, I was baffled at Scott Hanselman's recent post which claimed I had been working with him on some documentation tools.  Then I realized that, of course, this was a different Mark Miller.  This was not the first time I've been frustrated by the seemingly large number of Mark Miller's, especially in the blogosphere.

The real question is how can I distinguish myself from the others?  Certainly not by continuing to post rather bland entries to my blog.  I'm not able to make my work blog public (which is what I'd like to do), where there's alot of nifty activity.  I guess I just need to make a more concentrated effort to get my thoughts out in the open without revealing the application of the thoughts.  This probably could have a nice side-effect of thinking a little more abstractly about the decisions I make and maybe see a more broad application than I originally had in mind that make make a design work better.

Also, why WASN'T I working with Scott on some project? Why aren't I connecting with others more? That's a problem.

So, stay tuned, i guess.

I finally got Subversion set up over SSL hosted by Apache side by side with IIS.  The biggest problem was getting IIS to let go of port 443.  I'm running XP Pro SP2 and I'm not hosting any SSL from IIS, but it would still bind to port 443.  I found lots of information on configuring HTTP.SYS using httpcfg.exe (There's lots of other very cool SP2 tools in the download), which I though was really cool, but nothing that kept it from binding to 443.

My rather rudimentary solution was to start Apache first.  IIS/HTTP.SYS didn't complain about the port already being bound to.  It works like a charm now.  Now I can get to my repository from work through the firewall!  What I don't know is if the sequence of events at boot time will keep it from working right.  Any ideas?

I'm wondering how hard it would be to write a Subversion server on HTTP.SYS.  Now THAT would be cool!  I think it would only be a matter of implementing (or finding an implementation of) WebDAV and hooking into the Subversion API.

I am not a morning person.  Anyone who's ever spent any time with me in the morning knows that.  I've been tracking my productivity more closely lately (in terms of tangible, work-related results anyway), and I've found that I am far more productive after lunch than I am before lunch.  This doesn't mean I don't do anything in the morning, I just don't seem to have the same kinds of satisfaction with how I've spent my time.

I originally thought that I was being productive in a different way in the morning.  For instance, in the late afternoon and evening, I enter a fairly reflective state.  In this state, I question the design decisions I've made or the implementation I've chosen.  This is when I do alot of measurement in the form of tests or profiling.  That's plenty productive and useful activity, but in a different way.  I thought maybe the morning was perhaps an analogous, forward-looking phase, readying me for the tangible productivity ahead.  But, no, I'm just not very productive in the morning.

I'd like to find some activity or state of mind I could concentrate on in the morning in order to accomplish more in the morning.  Any ideas?  Does anyone else feel like their mind is more dormant at points during the day?  What do you do about it?

Since the 64-bit windows versions can only run the 2.0 framework, everything is having to run on 2.0.  This is a good test for the framework.  DasBlog does not run as smoothly on the 2.0 beta framework as I hoped.  So far, it's been problems with client-side interaction with server-generated html (javascript).  For example, the sign-in code had some nifty javascript that was formed in a fairly short-sighted way. I had to change the way that controls were being referenced.

I'm also having trouble getting apache installed to run Subversion.  I may have to make due with svnserve until I figure that out.

I just came across the Subversion Book. From the site:

a free book about Subversion, a new version control system designed to supplant CVS. As you may have guessed from the layout of this page. this book is published by O'Reilly Media.

At work we adopted Subversion and I really love it.  We made (in my opinion) a mistake early on by separating our projects into multiple repositories.  This makes it very difficult to handle the versioning synchronization when those projects collide.  It looks like the svn:externals property may keep us from  having to migrate to a single repository before we're ready to take on that problem.

As you can see, I've already taken away some useful tidbits.  A must read for anyone using Subversion.

This entry originally was a response to Heath's comment on my last entry on my ChickenScratch project, but it started to get long, so I figured I'd make it an entry.

He asked about my ideas for relational algorithms between topics.  The first phase of "auto-linking" is based on matching phrases in content with available topic names and meta-data like categories/properties with the content of entries.  Later, I might do things like Google-type analysis of content and present these “related“ topics in a way similar way to the “ads“ you get in Gmail.  We use FlexWiki at work to manage all our projects and their bugs, feature requests, and infrastructure projects, and more than once we've had “happy accidents“ in auto-linking within that data that have led to better design, and FlexWiki uses a very simple topic matching algorithm.

He also mentioned hierarchy.  Hierarchy is a key reason for me to create something new rather than retrofit one of the Wiki implementations available like the excellent FlexWiki. I love the "magic" linking and organizational ability of Wikis, but I don't like giving up the ability to direct organization through hierarchy.

Security and permissions is also core to the goals of ChickenScratch.  It's always frustrating to me when I have to separate my public information from my private information from the information I only want to share with friends just because a program doesn't let me.  My security model is working pretty well, and was pretty easy to implement thanks to the new Role and Membership providers in Whidbey

I'm definitely open to collaboration.  I'd like to solidify my ideas a bit more.  I've changed my parsing model about 4 times now.  What I've got going on now appears to be working out pretty well.

Yeah, an overview of the .NET API Design guidelines!  Now I can point people at a more consumable resource rather than trying to keep my own overview document in sync.

During some portion of my free time, I've been experimenting with coding a hybrid weblog, wiki, and general content management system I have called ChickenScratch.  This has been the project I've used to familiarize myself with the new features in Whidbey.  My goals thus far have been:

• Maintain socialness of a weblog (link tracking like referrals, pingbacks, and trackbacks)
• Maintain the ease of use, openness, and automatic cross referencing features of a Wiki (You don't have to worry with markup)
• Maintain the organizational/security capabilities of a content management system (Role-based ACLs, user profiles, deeply hierarchical content structure)

This project was the result of some frustration with how hard it would be to add these features to FlexWiki.  I wanted a system that would be just as suitable for holding personal brainstorming ideas not suitable for public consumption as for holding blog entries, or collaborating with my friends on coding projects or film scripts.

I've succeeded on these goals to a great extent, but one of my biggest problems has been deciding on grammar for the content.  I think one of the most powerful features of a Wiki is when you mention a topic and that topic gets cross referenced “accidentally”.  At work, we use a Wiki to help us track features, bugs, and changes to our API (which we treat differently than features) and we have lots of these happy accidents, but only because we are disciplined on how we name topics.

Most Wikis create topics by concatenating words together (or removing the space between them) like MyVacation2004.  The downfall of this (I've found) is that non-coders find this unnatural and annoying.  Other wikis use some syntax to identify a topic such as square brackets like [My Vacation 2004].  The problem here is that this removes the ability to have those happy accidents that I like so much.

So, I've decided to not put the burden of topic identification on the user (the person entering content), but rather on the system itself.  I think it makes a much more interesting problem to solve.

Sheesh another nerd post.  I thought this might be useful information for the community.

If you're using multiple threads in the execution of an ASP.net page, don't count on Response.End() to do what you think it might do.  Looking back it's pretty intuitive, I just didn't think about it.

At work, we have some pages that can take several minutes to execute.  They can pull millions of rows from the database and process them and do all kinds of craziness.  As a result, users don't like to sit around and watch Internet Explorer's progress bar decieve them time and again.  I implemented a progress mechanism to let the user know what is happening.  I went through several iterations before I come up with something really easy to work with.  It's completely transparent to the developer, and allows them to “push” progress messages onto a stack (using the wonderful “using“ statement), which is rendered for the user at an interval during the lifetime of a page.  It preserves the behavior of postbacks and Server.Transfers, but does break redirects if they happen after the progress gets started (which I hardly ever do).

Anyway, the point of the story is to tell you about Response.End.  The progress updating happens on a worker thread from the ThreadPool.  When I implemented it, I thought it would be nice to detect if a user has closed the browser or hit stop.  Then we could save some resources if no one is listening anymore.  So, before updating the progress, I do a Response.IsClientConnected, and if they aren't, I was calling Response.End().  This seemed to work at first, because it does do something, the problem is it doesn't stop what's going on in the main thread.

Reflector (the most useful CLR tool ever) reveals that Response.End() is basically just calling Thread.CurrentThread.Abort(), along with doing some clean-up and such.  Notice the CurrentThread...oops.  All I was doing was ending my progress rendering thread.  Like I said, it makes perfect sense now.

So, what do you do about it?  Well, I'll come up with something.  It shouldn't be too hard, but it will be pretty specific to my situation, so there's no real universal answer, unless Microsoft were to change the implementation so that it was keeping track of the threads and killing the main one, although then my worker threads wouldn't die.  Chances are if you've implemented some kind of multi-threading in your ASP.net pages, you can figure it out.

I'll forever chuckle when I read that title.  Maybe you will too after you read this.

I was aware of the Nullable generic struct in the 2.0 CLR, but I only just now figured out that they've built direct support for it into C# 2.0.  This is awesome!  At work, we currently have a whole bunch of structs that add null support to the value types in the CLR.  So we have NullableSingle, NullableDouble, NullableUInt64, etc.  This helps us incredibly to match the database concept of numbers and such that can have null values.

I've been looking forward to Nullable for some time, but now I'm absolutely ecstatic about the C# support for it.  Using the magic question mark, I can declare a value type to be nullable.  For instance:

bool? hasHairOnHead = false;

Now you get the title joke, right? This means that hasHairOnHead can be set to null, not just through Nullable's HasValue method, but by actually setting it to null:

hasHairOnHead = null;

Freaking awesome, eh?  So now, the Headless Horseman's hasHairOnHead value can more accurately express his status. (OK, it was the first example I thought of).  The same goes for null comparisons and such!

Of course now I'll get confused reading documentation...Is that a question or a nullable type?  Just try to decipher this faq: Why don't nullable relational operators return bool? instead of bool?

That little question mark is going to save me alot of headaches.  Now I just have to wait for it to be out of beta. 

I've been playing with the new membership framework in ASP.net 2.0.  The existence of this, and other nice things like the login controls are fantastic, but it really lacks the polish of other areas of the new framework.  Here's some constructive feedback:

The MembershipProvider looks like a pretty well designed base class, but it should implement more of its interface by default.  For instance, when creating your own provider, you're forced to implement Initialize, which configures the instance for the configuration in the web.config.  Initialize is defined in ProviderBase, but there should be a default implementation in MembershipProvider that at least configures the values for the required properties such as EnablePasswordReset and such.  I ended up using the Reflector to look at the AccessMembershipProvider to implement this method.  Also, the majority of the proeprties should have default implementations as well.

The documentation for implementing a MembershipProvider is also pretty inconsistent.  Some of the methods are supposed to throw exceptions for failures, some of them return booleans for success/failure (which I thought had been deemed not a good idea), and some of them say they should throw exceptions, but also return a boolean.

The MembershipUser class is also pretty good, but its ISerializable implementation is weird.  Deserializing one would leave it in an unusable state since its _Provider field would be uninitialized, and since it's not settable, its useless.

I love the idea and the ease of which you can use the membership concept. (The default “just works”) But to create the “pit of success”, there are some things to address and I hope someone is addressing them.

[UPDATE] Brad Abrams helped me get this feedback to the right people.  Thanks Brad.  It makes me feel really good about the direction of the .NET and Microsoft as a whole to know I can get my feedback to the people who need to hear it.  I may be making incorrect assumptions or drawing the wrong conclusions, but chances are if I am, then others are too.

The managed API for the windows performance counters is great.  It was just what I needed to track down some memory issues at work, and now I've got real-time information about what's going on.  It's much simpler than trying to use them with the old school api.  However, I need some more guidance on the usage pattern for creating and deleting the categories and counter definitions.  Are those things that should be part of an install/uninstall procedure, or should they be done on startup/exit of the program?  What kind of overhead should I expect in constructing/incrementing/disposing them?  The counters implement IDisposable, which makes me want to dispose them quickly.

Maybe some spelunking with the Reflector will shed some light.  I'll keep you posted.  In the meantime, feel free to share your experiences with performance counters in the CLR.

[Update] Duh.  I ran across System.Diagnostics.PerformanceCounterInstaller the other day.  Looks like that's how you should install them.  I love the installers as well.

In a comment discussion, I mentioned my disdain for using Perl for anything larger than simple scripts or line-based text parsing (which is what it was invented for...see below). I received some pushback on this opinion from Shelly, obviously from her own experiences with the language.  I enjoy pondering a dissenting opinion when based on experiences or data.  She says:

Using a good OO design and programming techiques, a perl program is not a nightmare for maintainability...just like any other language (yes even assembly).

Obviously, I need to open my mind here. I guess my question becomes: Once you get to the point where you need to apply OO design and programming techniques, are there not more suitable languages for a project like that?  I'm curious to know what other people's opinions are on this.  What are the advantages of Perl over other truly OO languages?  Why would you choose it instead of another language?

For your enjoyment, here's the original manpage for perl:

 NAME
  perl | Practical Extraction and Report Language

 SYNOPSIS
     perl [options] filename args

 DESCRIPTION
      Perl is a interpreted language optimized for scanning  arbi-
      trary  text  files,  extracting  information from those text
      files, and printing reports based on that information.  It's
      also  a good language for many system management tasks.  The
      language is intended to be practical  (easy  to  use,  effi-
      cient,  complete)  rather  than  beautiful  (tiny,  elegant,
      minimal).  It combines (in  the  author's  opinion,  anyway)
      some  of the best features of C, sed, awk, and sh, so people
      familiar with those languages should have little  difficulty
      with  it.  (Language historians will also note some vestiges
      of csh, Pascal, and  even  BASIC|PLUS.)   Expression  syntax
      corresponds  quite  closely  to C expression syntax.  If you
      have a problem that would ordinarily use sed or awk  or  sh,
      but  it exceeds their capabilities or must run a little fas-
      ter, and you don't want to write the silly thing in C,  then
      perl  may  be  for  you.  There are also translators to turn
      your sed and awk scripts  into  perl  scripts.   OK,  enough
      hype.

Today I needed an AVL tree implementation for some time-based indexing of cached data.  Unfortunately, I can't use the wonderful-looking OrderedDictionary in the PowerCollections project since .NET 2.0 is not due out for a while.  I searched around without any luck, so I rolled my own.  It's been a while since I messed with tree-based data structures.  I was pretty pleased with its performance characteristics, even with millions of records.

I'm thinking about releasing it as a short-term alternative for people having to wait on 2.0 and Peter Golde's PowerCollections project, so I thought I write a short blurb about it and link to some appropriate sites to generate some referrals and feedback.  Anyone interested in using it?

For those unfamiliar with AVL trees, it's a self-balancing binary search tree. Its characteristics that are of interest to me are:

• self-ordering - values are stored and can be retrieved in order simply by traversing the tree, something a hashtable cannot give you.
• self-balancing - This ensures that search times are O(log(n))
• fairly straightforward to implement, as opposed to Red-Black, or other self-balancing BSTs

Again, if you're interested in using it, leave me some feedback and maybe I'll make it a sourceforge or GDN project.

I've mentioned the thumnail generator, “Thumbo”, several times.  I wrote it about a year ago while I was on vacation wanting to upload pictures to my blog over dial-up.  All you had to do was drag images onto it and it would spit out thumnails sized to your specification.  I had a “duh” moment the other day when I right-clicked on an image in explorer and was surprised to see “Resize Image” in the context menu.

Turns out, I had installed the Image Resizer powertoy for XP, which provides basically the exact same functionality integrated into the shell.

Anyway, I know a few of my readers are using Thumbo to generate thumbnails.  In the interest of not having to support it anymore, I would recommend using the powertoy instead.

I had a request from Scott Hanselman (who has been vocal on the subject of ViewState on many occasions) to share my approach to file-based ViewState persistence that I mentioned in my previous post.  Feel free to comment on my approach.  I need to get code formatting set up.  Pasting from Visual Studio is a pain.

Here's my overrides of my base class page's LoadPageStateFromPersistenceMedium and SavePageStateToPersistenceMedium:

protected override object LoadPageStateFromPersistenceMedium() {
     return _viewStatePersister.LoadViewState();
}

protected override void SavePageStateToPersistenceMedium(object viewState) {
     _viewStatePersister.SaveViewState(viewState);
}

Doesn't tell you much, except I'm delegating persistence to a ViewStatePersister, which looks like (with some things renamed to protect the innocent):

public override object LoadViewState() {
     object viewState = base.LoadViewState();
     if (viewState != null) {
          Guid guid = (Guid)viewState;
          LosFormatter formatter = new LosFormatter();
          using (FileStream fileStream = new FileStream(CreateOfflineViewStateFilePath(guid), FileMode.Open, FileAccess.Read, FileShare.None)) {
               viewState = formatter.Deserialize(fileStream);
          }
     }
     return viewState;
}
public override void SaveViewState(object viewState) {
     //create a guid for this viewstate
     Guid guid = Guid.NewGuid();
     LosFormatter formatter = new LosFormatter();
     using (FileStream fileStream = new FileStream(CreateOfflineViewStateFilePath(guid), FileMode.CreateNew, FileAccess.Write, FileShare.None)) {
          formatter.Serialize(fileStream, viewState);
     }
     //trick the regular system into thinking all it needs to save is the guid
     base.SaveViewState(guid);
}

The appropriate persister is created with a call to a virtual CreateViewStatePersister() method, which a page can use to create the persister of its choice.  As I said in my previous post, I was dissapointed that I had to use the LosFormatter rather than the BinaryFormatter.  The downside of this approach is the possibility for creating LOTS of files, as opposed to creating a single file per session.  But I already have a file cleaning mechanism in place that cleans up files created by my graphing library, which creates more files.

On a recent iteration of a project at work, we were analyzing our cache usage for several data analysis pages we have using ASP.net.  The usage of these pages evolved as we updated them with features, and in a pinch we were forced to turn off caching because of the hit to memory we were taking.  As we loaded more and more data into our database, and we had more and more users, more stuff was being cached, but we were not seeing benefits from the caching across users because each user was looking at different data.  Turning off the cache caused individual users to take a big performance hit when reloading the same report twice or making minor adjustments to the options, but we just couldn't justify the memory usage for that small percentage.

I decided the solution was a multi-level cache, where objects expired to lower levels, like from memory to high-speed disk, to slower disk, to db (really, I only want the memory and disk levels, but there's no need to limit it to that).  Unfortunately, there is no mechanism for injecting new behavior into the System.Web.Caching classes to accomplish that, which is a shame because they have already implemented the expiration and dependency code that would be the most complex part.

So, I'm faced with the possibility of creating my own caching framework, or tricking the System.Web.Caching classes to do my bidding using the existing mechanisms like dependencies (which is a possibility).

On another note, I implemented file-based ViewState quite successfully.  It was much more simple and straightforward to address issues like a user with multiple windows than any article led me to believe.  In doing so, I noticed some VERY annoying things about that ViewState persistence mechanism.  The normal behavior uses the LosFormatter to serialize the ViewState to base64 to be put inline with the html.  This is fine for storing ViewState inline, but if I'm serializing to files, I'd rather have the speed and efficiency of the BinaryFormatter.  The problem is that LosFormatter is special and doesn't play by the same rules as the other formatters in the framework (BinaryFormatter, etc).  Most of the built-in controls use Pair and Triplet to store their data in ViewState, but they aren't marked with SerializableAttribute which means that BinaryFormatter can't serialize them!  So I had to continue to use the LosFormatter, which bloats data horrendously.  I hope some of this is cleaned up for ASP.net 2.0.

I was adding some complex usage logging in a data abstraction layer today, and I wanted to minimize its impact on performance.  I was dreading having to manage worker threads, then I remembered the ThreadPool. It was so easy to shuffle off my logging to a managed worker thread.  That saved me alot of trouble.  One of those “pit of success” moments for sure.

(I would have rather linked directly to Rico's site, but Brad had a higher google rank.

I'm experimenting with a project which will need some complicated file parsing abilities and I don't want to go the XML route for various reasons.  Therefore, I am looking for a parsing framework built to take advantage of the CLR.  Everything seems to be a port of some archaic C library, or some Java framework that's been modified to produce C# code.  I'm used to being able to do a simple Google search to find the kinds of libraries I need, but I'm getting very few good hits for this.  I've only built a few pieces of code that really qualify as parsers over the years, so maybe its my inexperience that's my problem, but it just seems like it ought to be easier to find a good tool.

We've had enough time for the CLR tools to take on their own identity and take advantage of the CLR rather than remain lagging clones of their Java or C++ counterparts.  I see this in almost every space.  There just seems to be a huge hole for file parsing.  Maybe most CLR developers have embraced XML as the one and only file format.  But, when it comes to human-edited files or content, XML is pretty cumbersome and bloated.

For now, I've settled on Grammatica, which is the most straightforward (and working) parser generator thus far.  It's written in Java, but that's not a big deal.  The problem is that its output is Java-centric, and only modified slightly to be C# code.  It's got callbacks instead of events to handle tokens or products.  I found myself modifying the output (a no-no in code generation) to make it simpler before I realized I'd just be doing it again if I changed my grammar.  So, I'm really frustrated and just ranting, but I should be able to mold it to my purposes.

I finally took some time today to explore NMock with a collegue.  We wanted to unit test a .NET component that was consuming some COM interfaces.  This is difficult, since the instances we are dealing with are created by another process, and they are not createable outside that environment. “Mocking” allows us to create an object that looks and behaves like the desired component for a specific condition, as well as provide an indication if the object under test behaved in the expected way.

NMock provides a full framework for mocking, and works well with NUnit, a top-notch unit testing framework.  While seeing a great potential for usefulness and making unit testing less painful, I was sad to see that there are some issues that kept it from working in our case:

1. Practically zero documentation - No official documentation, some ambiguous samples here and there.  You can use the jMock documentation to learn concepts, but NMock is hardly a transparent port to the CLR.
2. Some issues with mocking interfaces - It doesn't seem to recognize inherited members.
3. Some issues with COMInterop - This was what made #2 a show stopper.  It was trying to instantiate a COM object when all we wanted was to mock it.

We'll screw with it some more in the next week or so before giving up.  There seems to be much more recent code in SourceForge than is released.  Maybe some of the issues are fixed.

I've been re-reading more of Martin Fowler's content. Tonight was “The New Methodology”, kind of an overview of the Agile Development movement.  Some of the quotes that really hit home:

There's a refrain I've heard on every problem project I've run into. The developers come to me and say "the problem with this project is that the requirements are always changing". The thing I find surprising about this situation is that anyone is surprised by it. In building business software requirements changes are the norm, the question is what we do about it.

I'm always complaining about moving requirements.  I guess what I should be complaining about is that I am not empowered to deal with them.

(On people as replaceable parts of the development cycle):

This creates a strong positive feedback effect. If you expect all your developers to be plug compatible programming units, you don't try to treat them as individuals. This lowers morale (and productivity). The good people look for a better place to be, and you end up with what you desire: plug compatible programming units.

I see this every day.

OK, it's been a while since I had a good nerd post.  Here goes.

I subscribe to Martin Fowler's updates RSS, meaning I get notified when he adds or updates new content.  He doesn't do it often, but when he does, it's always interesting reading to me?  He thinks like I want to think...about software anyway.

His latest update is to a relatively old paper looking at the way software design is evolving and how thinks like Extreme Programming (or XP, not to be confused with Windows XP), refactoring, and Design Patterns have driven that change.  He asks the question, “Is Software Design Dead?”  Of course he answers it as well...No it's not.  As I looked at the updated sections, I couldn't recall the rest of the article, so I read it again.

It's an excellent read for any interested in the realm of software design.  I've found it to be particularly true for me in the past few months.  I can bang out code with what seems to be very little “design”, when in reality I'm just moving design inline with the coding.  The code designs itself as it evolves. Anyway, I know one or two of my readers that may find it enlightening.

Several people have asked me where I got the latest version of dasBlog.  What is not readily apparent by looking at their site is that the project is now hosted on GotDotNet Workspaces.  The latest source and binaries are available there.

I did a bunch of profiling at work today.  I use the Allocation Profiler for memory profiling.  It does a great job of letting you see your allocations visually in several very useful forms.  Today, I took a second look at nprof, a more CPU-oriented profiler that attempts to give you an idea of what calls are taking up the most time.  When I first looked at it almost a year ago, it was not useable at all.  Now, it seems to only have a few problems with GUI apps, and multi-threaded/multi-AppDomain projects.

The app I spend most of my time on at work is a data analysis application.  Data is loaded into the DB from a spool, so 99% of the app is “read-only” access to the DB.  This means I can optimize the heck out of it since I'm not concerned with transactions and such.  The downside is that i must optimize the heck out of it.  A common query can return millions of rows from the database, so taking even a few milliseconds out of the loop can save alot over that many iterations.

My first insight is: Don't use NUnit to run performance tests.  At first, my unit tests seemed to be a very convenient location to put performance tests.  That is a bad idea.  The perf numbers scared me to death.  It appeared as thought I had created a performance monster, and not the good kind of monster.  Turns out, NUnit goes to alot of trouble to isolate the test runs in separate AppDomains so it can unload them easily and you don't get undesired interaction.  This seems to add a great deal of overhead, probably in marshaling across the boundaries.

My second insight is: Use the Allocation Profiler (or some other profiler that lets you look at memory usage)!  I don't want to go into a big discussion on how to use it.  It's pretty straightforward.  If you interested in specifics, leave me a comment.  Remember, allocation in the CLR is cheap, but excessive garbage collection can be costly.

My last insight is: Use nprof (or another profiler that gives you CPU info).  nprof gives you a good idea of where your CPU bottlenecks are.  I was really able to get a good idea about what needed attention.

In a particular performance test of my data access layer, I was pulling over a million rows.  I was able to cut memory consumption in half, and increase my speed by a factor of 10 by using information gleaned from the profilers.  This, or course, means the code was pretty crappy.  He he, just kidding.  Like I said, it's a very tight loop, and a little goes a long way.

If you want a good reference for performance-related stuff, check out Rico Mariani's blog.  He appears to be THE Microsoft performance guy.

With Mozilla (FireFox, Netscape, etc) becoming a more viable alternative to Internet Explorer, we've had a push for Gecko (the Mozilla rendering engine) compatibility at work, so we've been going back and redoing some things to layout correctly.

We noticed a few things that just weren't working.  When we looked at the HTML source using FireFox, I was intrigued by the fact that all my div tags had become tables.  Turns out, the ASP.NET browser capability detection identifies Gecko as a down-level browser, and gives you an Html32TextWriter rather than a regular HtmlTextWriter.  And, sure enough, digging around using the Reflector confirmed that div tags are replaced by tables when using Html32TextWriter.

This guy, has a solution.  Just thought some of you would be interested.

The browser capabilities section is extremely powerful and insulates you from alot of headaches, but the default configuration is very annoying.  Take a look at your machine.config and you'll see what I mean.

I finally had some “spare” time at work today, so I re-worked the layout of our huge data analysis web site using CSS styling and layout rather than the original old skool methods we used when we first built it many years ago.  The original design was done back in about 2000, and I didn't have alot of time to do it, so I just threw it together.

I've had a fair amount of experience in the last couple of years with all-CSS layouts.  My favorite place to learn is CSSZenGarden.  Anyway, the benefits of separating content from presentation are well known, and CSS is not the only method of decoupling, so I won't really go into why I was doing it.  Although I did come to the conclusion that CSS is lacking some things that would make what I was doing much better.  Anyway, I did do some searching for some examples when I couldn't eliminate a stray pixel spacing here and there.

What I found was alot of people raving about having no tables on their site.  Personally, I think that's a little silly.  Not to say I'm not impressed by what people can do without tables, but that would be like me saying, “I built a house, but I didn't use any bricks” and expect people to be impressed.  Certainly, there are many houses built without bricks.  Very functional, beautiful, well-built houses.  But you can't build a brick house without bricks.  Most people who brag about not having any tables don't realize why they would want to banished tables other than they read someone else bragging about being table-free.

The point is not to rid the earth of the table because it's evil. Tables are perfectly fine for identifying content as being part of a table, after all, you don't replace all the images on your site with thousands of tiny DIV tags perfectly sized, colored, and positioned to replace every pixel.  That sounds silly, but I've seen people do the equivalent of that in trying to replace a table in the quest of table-less HTML.  The point is to organize the structure of the content so that it can be interpreted as simple data, and can then be “styled“ for presentation.

Well, looking back over this entry, I find it to be one of the most lame entries ever.  Oh well, I feel better having griped about that.

I know what you're thinking...“Too many posts!  He's mad!  He's beating the pants off of Jenkies!“  Well, this one's technical.

If you're like me, you've always been annoyed at the inherent coupling between pages that pass data to each other through a Server.Transfer() call.  I think it leads to poorly designed, tightly-coupled workflows, and tends to lead people to taking shortcuts or breaking the model to make their stuff work.

Until now, I've tried to minimize this issue by giving my base class page a TransferData property typed as an object so every page can use it to pass data.  This has its own problems.  For instance, if you call Server.Transfer twice in the same Request and use Context.Handler to retrieve the transferer (which seems like a hack to me), it's the first page, and there's not a reference to the second page in the call chain.

I now have discovered a nifty little storage location for putting things like this... HttpContext.Items. It's just an IDictionary that stores stuff in the context of the current request.  Since you can always get the current HttpContext with HttpContext.Current (an implementation worth taking a look at with Reflector), you can get to it from anywhere, regardless of whether you have a reference to ANY page.  It works even if you're passing control to or from some handler that's not even a page.

Think of it as the analog to Application state, or Session state, but for the current Request only.  It automatically decouples your pages because they only have to agree on a common key.

UPDATE: I should note that it was my co-worker Casey Marshall who initially brought the property to my attention.  Thanks, Casey

I've been on a real software philosophy kick lately.  Here's more of my thoughts (which are subject to change).

I've spent alot of my time trying to develop methodologies and code to help me write database independent code.  Invariably, I run across problems, some caused by a db with a crappy internal type system (Oracle), some caused by a broken unification model (ADO.net), and others caused by poorly implemented providers.

I ran across a heated discussion on another blog (which I can't find now, or I'd reference it) on whether database independent code should be something to give up on.  After all, even though they compete for business, Oracle was designed for something very different than SqlServer, or MySQL was designed for.  They each have unique features, different SQL syntax, stored procedure concepts, etc that keep there from being a universal API for database access.

So there are two common approaches to take (excluding giving up):

1. Implement to the lowest common denominator.  Solutions for this exist already.  ODBC, JDBC, OLEDB, all fit this model.
2. Move the unification layer inside your application, so you are implementing a unified API for your application.  This allows you to focus on the functionality you need and use object oriented techniques and patterns to reduce redundancy and increase maintainability.

I first approached ADO.net as the answer to #1, for which it is woefully inadequate.  I now realize it is an enabler for #2, defining concepts that databases share (which make them databases).  This gives you some commonality between implementations without forcing you to the lowest common denominator like previous attempts.  Besides, most applications are targeted to a specific db anyway and don't benefit from that approach.

ADO.net is not without it problems, but viewing it in that light makes it a much more satisfying tool...for me anyway.

If you've ever used NAnt, you'll love NAntPad.  From their site:

Nantpad is a powerful editor for NAnt XML build scripts. It provides an explorer-type interface saving you from having to write XML which can be tedious and sometimes difficult to maintain. User friendly icons indicate targets, tasks, parameters, comments and other NAnt entities, and attributes are modified in a property window.

I just downloaded it and loaded a few of my more complicated build scripts.  No troubles.  It helps you enforce the schemas so it's alot easier to write a script without consulting the documentation to find out the name of that one important attribute.  Me likie.

Here's some info straight from the horse's mouth (SqlTeam.com).  It seems using @@Identity is potentially bad.  They say to use scope_identity() instead. See for yourself.

I'm not a TSQL guru, but I know several people using TSQL in the mojority of their products, and I've seen @@Identity in their code.

CodeSmith is (according to it's website):

CodeSmith is a FREEWARE template-based code generator that can generate code for any ASCII-based language. CodeSmith templates use a syntax nearly identical to ASP.NET syntax so that creating templates should feel immediately familiar to ASP.NET developers. Check out the overview for more information, visit the download page to get the latest version, or visit the support forum to get your questions answered.

So, if you like that sort of thing...Check it out.

I found a really cool logging component today in my search for...a logging component. It's a port of the very popular and useful log4j logging component for Java. It's called log4net oddly enough. It's very well executed, making ingenius use of static members to create more of a logging "environment" than a simple logging component. It's completely configurable on the fly for named hierachy as well as logging levels.

Some of you may ask, "Why do you need a logging component with all the trace functionality built into the framework?" Well, you've got alot of nerve, mister! You'll be happy to know that log4net makes excellent use of these built-in features, as well as adding a whole slew of extensible options. Just go check it out and I'll shut up. It's also licensed with the Apache license so there's no worries about paying anyone for use in a commercial application.

While I'm on a soapbox about delicious open-source/free tools for the CLR, I'll also mention the following tools I use at work:

• NAnt - An open-source build tool based on Java's Ant. I use it exclusively at work since no one will pony up the dough for Visual Studio. I've found I have much more control over my builds.
• NDoc - An amazing documentation creator that uses the xml files created when building with the /doc switch on the C# compiler. The documentation comes out looking just like the framework documentation.
• NUnit - A unit testing framework that was originally based on Java's jUnit, but completely rewritten to take advantage of the CLR's marvelous features.

This is a test! It was done entirely over a dial-up connection. The picture was sent via ftp. I'm really just trying to fill space so I can see what a long entry might be like. I'm not in this picture, it's Dave, Jen and Becky. Looks like I'll have to tweak my thumbnail program (which I called Thumbo!). It makes pretty crappy looking thumbnails.