This post is intended to fill a gap in the current MSDN documentation for this attribute (http://msdn.microsoft.com/en-us/library/bbx34a2h(VS.100).aspx).  This gap should be filled by the time .NET 4 ships.

There is alot of confusion about what the useLegacyV2RuntimeActivationPolicy attribute does.  Most often, it is used to allow a pre-v4 mixed-mode assembly to load in v4.  In that context, the name makes very little sense.  Below is an explanation I’ve provided to people internally that explains the attribute in the context for which it was named.  This should give people a better idea of what it does, as well as understand some of the subtlies of in-proc SxS.

Ultimately, this attribute has to do with the behavior of the “legacy shim APIs”.  You can think of these as encompassing several categories of CLR activation:

• CorBindToRuntimeEx and friends - This includes most of the flat exports of mscoree.dll defined in mscoree.h (GetCORSystemDirectory, GetCORVersion, LoadLibraryShim, etc). Note, this also includes the strong name APIs defined in strongname.h)
• Pre-v4 COM activation – This includes CoCreateInstance of a CLSID (or type identifier) whose latest registration is against a pre-v4 runtime version. Note this includes both the “new” operator on such a co-class from managed code, or the result of Activator.CreateInstance against a type created by Type.GetTypeFromCLSID on such a CLSID.
• Pre-v4 IJW (mixed mode) activation – For example, calling into a native export on such an assembly
• Native activation of a native runtime-provided COM CLSID – Such as CoCreateInstance on ICLRRuntimeHost’s CLSID
• Native activation of a managed framework CLSID – Such as CoCreateInstance on System.ArrayList’s CLSID (extremely rare)

All these have a “single runtime per process” view of the world, so we try to make those codepaths believe they still exist in that world by “unifying” the version that they see.  After a given version has been chosen by one of these codepaths, that’s the version that all of them see for the remainder of the process lifetime.  Additionally, all of these activation paths had some kind of roll-forward semantics associated with them.  We “cap” those semantics at v2, meaning by default none of these codepaths see v4 at all.  This allows us to claim that installing v4 is “non-impactful”.  It should not change the behavior of existing components when installed. (Note that this has the interesting side-effect of a v4 only machine appearing to have no runtimes installed at all via these codepaths.)

This is all well and good until someone WANTS those codepaths to see v4.  Rolling a v2 managed app forward to v4 using a config (without the attribute) works just fine, unless that app also expects interaction with these “legacy” codepaths to be associated with the current runtime (v4).  For instance, a p/Invoke to GetCorSystemDirectory in order to construct a path to Fusion.dll (please don’t do that, BTW) will give you v2’s fusion.dll.  COM activation of a managed COM object will prefer the runtime it was built against rather than load into the current runtime (meaning you may be dealing with interop rather than a concrete CLR type). That may work, and it may not, depending on what you’re doing.

The useLegacyV2RuntimeActivationPolicy attribute basically lets you say, “I have some dependencies on the legacy shim APIs.  Please make them work the way they used to with respect to the chosen runtime.”  In that context, hopefully the name makes more sense to you. It is *mostly* equivalent to calling CorBindToRuntimeEx using the full version string for v4.  We also have a method in our new shim APIs to do this programmatically, the difference being that in a config file, it can be done declaratively, which is useful for a host that uses config files to determine which runtime to load plugins into. (the attributes value (or lack of value) is conveyed back to a host via the pdwConfigFlags parameter of ICLRMetaHostPolicy::GetRequestedRuntime)

One of the big reasons people need to do this is if they have a dependency on a pre-v4 IJW assembly.  By default, we can’t allow those to load into v4*.  Putting this attribute in your config allows this to happen.

Why don’t we make this the default behavior? You might argue that this behavior is more compatible, and makes porting code from previous versions much easier. If you’ll recall, this can’t be the default behavior because it would make installation of v4 impactful, which can break existing apps installed on your machine.

Well, why don’t we make this the default behavior for v4 managed apps?  Well, that is precisely the behavior we had for beta 1.  As we started trying to explain the behavior to people, we found it was very difficult to explain how these legacy codepaths worked.  We ultimately decided that making the behavior consistent was better.  The example that ultimately convinced me we had made the right choice was that the behavior of a library would change based on whether it was hosted by a native process or a managed one.  That seemed really bad to me.

You might say, “Why shouldn’t I just set this for every app I have?” Well, the downside of this attribute is that it turns off in-proc SxS with pre-v4 runtimes.  It locks them out of the process.  This may not matter to your scenario.  If you look at some of the runtime tools, they are using this attribute.  Even Visual Studio uses this attribute.  Don’t just blindly use it though.  If you' find yourself needing it in something other than a migration aid, or for loading pre-v4 mixed-mode assemblies (which we hope becomes more rare moving forward as people start updating the interesting mixed-mode binaries out there), I’d like to know about it.  Leave me a comment!

Hopefully, you’ve got a better handle on exactly what this attribute means, and can make a more informed decision about when it is appropriate to use.

*There are many engineering challenges around in-proc SxS and IJW assemblies.  Currently, pre-v4 IJW assemblies can only load into the runtime that is associated with the “legacy shim APIs”.  But any given IJW assembly (regardless of version) may only be loaded into a single runtime per process at this time.

My brother-in-law posted a note on Facebook that was basically one of those silly things you do and perpetuate across the internet.  I usually don’t take part in such things, but this one seemed interesting, and I’ve been thinking about ways to jumpstart my blogging again now that the embargo on all the cool stuff is lifted.  So, I thought I would do it.  Here are the rules:

• Grab the book nearest you. Right now.
• Turn to page 56.
• Find the fifth sentence.
• Post that sentence along with these instructions in a note to your wall. (this was on Facebook, so it is referring to that wall)
• Don't dig for your favorite book, the coolest, the most intellectual. Use the CLOSEST.

So, it took me a while to determine which book was the “closest”, as my position is roughly normal to the bookshelf in my office.  I finally decided to be honest and pick the one that was really closest, but I will share another that was very close, as it is a good segue into future blogs.

The first (and official) one:

Semiconductor materials at 0 K have basically the same structure as insulators – a filled valence band separated from an empty conduction band by a band gap containing no allowed energy states (Fig. 3-4).

Solid State Electronic Devices, Ben G. Streetman

The second, and more relevant/interesting one:

The shim’s algorithm for picking a version in the COM interoperability scenario is much more straightforward – the latest version installed on the machine is always used.

Customizing the Microsoft .NET Framework Common Language Runtime, Steven Pratschner

What is amazing about this second one is that this is directly related to one of the features I’ve been working on for CLR v4 (and yes, that is actually the 5th sentence on page 56).  Namely, this feature is known as “in-process side by side” (or in-proc SxS for short), and was announced publicly at PDC last month.  This feature allows you to have more than version of the CLR loaded and running in a single process.

This feature is primarily a compatibility feature, targeted precisely at the behavior noted in the quote above.  When we use the latest version, we can break existing COM objects.  Not only because of breaking changes we make (of which the number is fairly small), but because of other, more subtle behavior dependencies.

Previously, loading a CLR version into the process locked the process to that CLR version.  Any other policy than “pick the latest” results in a load order dependency problem that can result in “for sure” breaks because COM components targeting newer runtimes cannot run on old runtimes.  So, clearly, that was the best choice of policies.

Now that we support multiple runtimes in the process (v2 and above), we can make a smarter, more compatible choice about runtime activation.  The precise policies are still being worked through, so I’ll avoid stating them explicitly, but you can imagine us being able to make a much better choice about what runtime to activate to run a given managed COM component.

I’ll be posting more about this feature and it’s implications soon.  Feel free to seed my future posts with questions in the comments.  Hopefully, this is the jumpstart I needed.  As for the “game” above, feel free to do it, or ignore it.  It won’t result in any difference to your luck, financial situation, or anything else.

I've been doing some app-building with Silverlight lately and exploring the limitations of the platform in comparison to the full desktop CLR and what that means for the Silverlight "ecosystem".  Those limitations can be summed up with 3 items:

1. Strict sandbox security model
2. Reduced managed framework surface area
3. No binary compatibility with libraries targeting the desktop CLR (eliminates the number of 3rd party components you can leverage).

I think #3 will begin to become a non-issue as more component providers provide builds for Silverlight. #2 can be broken down into 2 areas:

• Full technology areas that are unavailable on Silverlight (LinqToSql, WinForms, etc)
• Reduced/pruned APIs

The second is where my interest lies, and intersects with #1 (the new security model).  We've done extensive threat modeling against our APIs as well as automated tooling that has either removed (or made internal) certain APIs, or marked them as SecurityCritical, meaning they cannot be called directly from "user" code.

In addition, the security model requires safe, verifiable user code.  Normally when developing on the desktop CLR, unless you are specifically targeting a low trust environment, you can do whatever you like.

So, for this exercise, I pulled out my trusty STDF parser (blogged here and here).  I've used this project as a test vehicle for both v2.0 and Orcas, and it's served well as a project that leverages a large cross-section of features in the Framework, from high-level stuff like Linq down to expression tree inspection and further down to LCG/RefEmit.

In short, I was able to get the parser working without too much trouble. I felt like creating a source bed that intended to target both Silverlight and the Desktop should be an attainable goal.  You just have to switch your mindset from binary compatibility to source compatibility.

I combated the reduced surface area with extension methods, which worked quite well to centralize the "overload shims" that needed to be "Silverlight-only", as well as for a general refactoring tool.  My goal is to have all the desktop vs. Silverlight differences centralized into files that are either included or excluded from the build depending on which platform I'm targeting.  I wish you could create extension properties.  That would let me close all the surface area discrepancies that aren't caused by missing/irrelevant technology areas.

I was pleased that 99% of my LCG codegen stuff "just worked".  I make heavy use of Reflection.Emit via DynamicMethod to generate my record parsers based on attributes on the record classes.  The 2 problems I ran into were:

• Visibility restrictions - The new security model won't let my dynamic methods see internals.  I had used this ability to keep the API clean.  I'm still figuring out the best approach, but it was simple enough just to expose those methods.
• Verifiability - I had a few places where I was generating unverifiable code.  Some of these were my own codegen bugs, but others were just bad assumptions on my part.

This brings me to constrained callvirt, an interesting little IL tidbit I discovered in the porting process.  Calling conventions are subtly different between reference types and value types, and it also depends on whether the given method is actually overridden in the value type (which can lead to confusing breaks when that state changes).  In the v1.x days, you always knew the type you were dealing with, so it didn't matter that much and you could generally always create the right sequence of IL to make a call.  V2.0 introduced generics, which meant that you couldn't emit unified IL for both the possibility of reference types or value types.  This meant there needed to be a way to unify the IL for the 2 cases.  This is where the constrained prefix came in.  It allows you to write unified IL that works regardless of whether you're working with a value type or a reference type (think generics constrained by an interface, or calling a method defined on System.Object like ToString()).

Anyway, I was able to fix my unverifiable code by utilizing the constrained prefix.  It also simplified my codegen logic significantly in a number of places where I had different paths based on whether I was working with a value type or not.

All in all, I was pleased with the results.  I'll be posting a sample Silverlight app using the library when I get some UI stuff figured out.

This is one of those entries that attempts to fill a void in online search for a particular topic.  I ramble on for a while to give enough context so that a search engine can match it up in a relevant manner.

I was debugging something the other day, and thought I had come across a heinous bug in the CLR.  Turns out, everything was working fine and the bug was in the app.

A program was crashing and it was a managed exception, so I attached to it with VS2008 and dug in.  The first thing I noted was that this was a retail build, so my stack was collapsed in a number of places... expected, keep moving.  Then, I noted an interesting frame on my stack (this is a contrived example, not the actual thing I saw):

>    CanonTest.exe!CanonTest.GenericType<System.__Canon,int>.SomeOperation() + 0x55 bytes   

That's weird, what is System.__Canon and what's it doing here? Surely this must be a horrible CLR bug where my method tables are being corrupted!  Internet searches seem to be confirming my thoughts.  A few others with random __Canon's showing up on the stack do look like bugs.  After a bit more reasoning, I come to the conclusion that System.__Canon must be special in some way since it follows the pattern for such things...  marked internal, pre-pended with double-underscore.

A few emails later, I had my answer.  System.__Canon is the special type that is used to identify "canonical" generic type instantiations. It typically only shows up in release builds, so you don't normally see it on the stack while debugging.  So, it's easy to assume something's wrong when you do see it.

If you'll recall, one of the really cool things about the Generics implementation is that it allows for code sharing.  Jitted methods can be shared between compatible type instantiations.  For instance, the code for SomeGenericType<string> can likely be shared with SomeGenericType<Foo> (where Foo is another reference type. We don't currently share code between value types, so you'll continue to see those on the stack such as in the example).  That shared code needs to live somewhere, so we have the concept of a "canonical" generic type instantiation that is identified by __Canon as the type parameter.

Also, in alot of debugger stack representations you'll get the GenericType`2 form rather than expanded form.  In that case, you never see __Canon.

For some more info on Generics and code sharing, see Joel Pobar's excellent blog entry on the subject.

The CTP for the MVC framework includes support for master-page, page, and user-control based views.  I thought it might be interesting to enable .ashx-based views for things like RSS generation via System.Xml.Linq, or other more "raw" view output.

As it turns out, this is fairly trivial.  The place we need to extend is the IViewFactory returned by Controller.ViewFactory.  This is the component that is responsible for creating the view when a call to RenderView is made.

The default view factory is the WebFormViewFactory, which knows how to generate views based on .master, .aspx, and .ascx views.  Since we want to add support for .ashx, we'll use WebFormViewFactory as a starting place.  We'll inherit from WebFormViewFactory and override CreateView to supply our extra .ashx lookup.

          using System; using System.Globalization; using System.IO; using System.Web; using System.Web.Compilation; using System.Web.Mvc; publicclass SpecialViewFactory
: WebFormViewFactory { staticreadonlystring[]
ViewLocationFormats = newstring[]
{ "~/Views/{1}/{0}.ashx", "~/Views/Shared/{0}.ashx"
}; ControllerContext _ControllerContext; #region IViewFactory Members protectedoverride IView
CreateView(ControllerContext controllerContext, string viewName, string masterName, object viewData)
{ _ControllerContext = controllerContext; //check to
see if there is an ashx that matches here.objectvalue = null;
controllerContext.RouteData.Values.TryGetValue("controller", outvalue); string controllerName
= valueasstring; if (controllerName
== null) { thrownew InvalidOperationException("No
route data value available for controller."); } Type viewType = null; foreach (var
loc in ViewLocationFormats) { var path = string.Format(CultureInfo.InvariantCulture,
loc, viewName, controllerName); viewType = GetCompiledType(path); if (viewType
!= null) break;
} if (viewType == null)
{ returnbase.CreateView(controllerContext,
viewName, masterName, viewData); } if (!typeof(IView).IsAssignableFrom(viewType))
{ //TODO: better exceptionthrownew InvalidOperationException("Type
not a view"); } var view = (IView)Activator.CreateInstance(viewType); var viewHandler
= view as ViewHandler; if (viewHandler
!= null) viewHandler.ViewData = viewData; _ControllerContext
= null; return view;
} private Type GetCompiledType(string path)
{ Type compiledType = null; try { if (File.Exists(_ControllerContext.HttpContext.Request.MapPath(path)))
{ compiledType = BuildManager.GetCompiledType(path); } } catch (HttpCompileException)
{ throw; } catch (HttpParseException)
{ throw; } catch (HttpException)
{ } return compiledType; } #endregion } 

GetCompiledType had to be replicated as it isn't exposed in the base class.  Note, I added a File.Exists check before I attempt to get the compiled type from the BuildManager. This was really to avoid having to deal with a bunch of first chance exceptions in the debug, although it seems likely that avoiding the exception is a good thing.  It wouldn't catch handlers that are mapped in the app dynamically or via web.config.

As you can see, I also added a ViewHandler class that my handlers can inherit from that gives them the same goodies that the other views get, but I'll leave that as an exercise for the reader to implement.

So, now the only thing remaining is to inject our special view factory into the pipeline instead of the default.  A simple way to do this is to set the ViewFactory property in the constructor of any controller that needs .ashx support. Now, you can create .ashx files and use them as views!

Next time, I'll show you how to add support for routing controller actions based on data not in the URL.

I wish I had time to come up with more concrete information (examples/code) in this post, but I don't have the time to work that stuff up.  I did think it would be useful for people searching for solutions to this problem, so here it is in all its ambiguity.

I was playing with expression tree inspection and dynamic interpretation the other day, when I hit something that I was sure was a bug.  I was inspecting an expression tree, identifying "branches" of interest, and generating lamdba expressions from them on the fly.

You might do this to break up an expression into separate units of execution to spread across multiple processors (a la PLINQ), or to replace parts of a tree requiring local execution before passing off to another layer to be transformed into another domain like SQL, or whatever.  In any case, I was doing it.

I found that if the type of the expression was a value type, I could not create a lambda expression returning object from it, even though there is an inheritance relationship.  You get a fairly straightforward, but perhaps surprising exception.

After some back and forth with the Linq team, I discovered that this was by design.  In the case of value types, the boxing operation required to make an object must be represented by a unary convert expression. The solution is to wrap such expression trees with a call to Expression.Convert(expression, typeof(object)).

An interesting pattern I've seen emerge since the early releases of Orcas is what I might refer to as "delegate properties".  What I mean by that is a property (or field, I suppose) that returns a delegate.  This pattern has some interesting implications.

First, in a language that treats delegates as directly callable objects, this pattern looks just like a method (someInstance.TheProperty(args)).  You can't tell the difference (although VS gives you different intellisense) by looking at a callsite like this. Among other things, this leads to some interesting naming issues.  Do you name it like you would a method?

Second, it opens up opportunities to do some really powerful (and slightly insane) hybrid inheritance models.  Think about a virtual delegate property that has both a getter and a setter, now think about trying to predict what that delegate will do when you call it.  It doesn't sound like something to recommend as part of a public API, but I think there are some interesting scenarios there.

If I come up with something interesting and useful, I'll let you know.

If you have been following my series on delegates, you may have experimented with open-instance delegates and perhaps found it difficult to create an open-instance delegate for a value type.

If you'll recall, an open-instance delegate has an extra first parameter, used to pass the instance used for the invocation.  What's not made explicitly clear is that this first parameter must be passed by reference.

For reference types, you've automatically got a reference, but for value types, this must be a "ref" parameter.  For instance, a delegate type used as an open-instance delegate for Int32.CompareTo would have to be defined something like:

delegate int IntCompareToDelegate(ref int instance, int other);

Otherwise, you'll get a System.ArgumentException when you try to bind the method to the delegate, giving you the ever-helpful error message: "Error binding to target method".

There are lots of underlying reasons for this, both from a calling convention perspective, as well as a side-effect perspective.  But, you can simplify it by thinking about modifications to the instance.  If you passed by value (creating a copy that the method acted on), any changes made to the instance by the method would be lost because they happened to a copy.

In most cases, value types are immutable in the framework, but you could run into issues with your own types.  And, again, this isn't the only reason for this restriction (take a look at the IL generated for a value-type method call to get some more ideas).  It's just the easiest to understand.

If you'll recall, Orcas extension methods, which are similar in concept to this, do not follow this pattern and are subject to the infamous value type copying problems.

I was helping a friend with a problem recently.  He was taking a set of serial web service calls and doing them in parallel to save time, and was not up-to-speed on the best approach for that.  Once he settled on an approach, he realized that since his web service calls were being wrapped in an abstraction layer, he didn't have the Begin/End asynchronous call methods that are provided by the proxy class.

"No problem, just wrap them in a delegate".  The compiler automatically gives you Begin/EndInvoke methods in addition to the synchronous Invoke method.  And, you're guaranteed not to mess up the implementation because it's all provided by the CLR!  Just one of those things you might forget if you find yourself in the same situation.

After my last few CLR posts, I've had a couple of private inquiries regarding the usefulness of closed static delegates.  To bring everyone up to speed, a delegate pointing to an instance method needs a "target" instance to operate on (we'll get to open instance delegates later).  A static method, needs no such target, so we can leverage the "space" used for the instance case to carry around another object of interest.  We call a delegate with a provided value for this space "closed over the first argument".

For example, let's say we have a static method that does some operation on two numbers.  For simplicity, let's just say it adds them.  Our silly class and method might look like this:

            public
            static
            class NumberFunctions
{ publicstaticdouble Add(double first, double second)
{ return first + second; } }

Normally, a delegate for this method would look like:

            public
            delegate
            double BinaryOperation(double first, double second);

But, we're going to create a closed static delegate, which means we're going to "burn" the first argument into the delegate itself, so it's not needed in the delegate signature.  Instead, we'll use the following delegate signature (I didn't spend much time thinking up these names, I hope they make sense:

            public
            delegate
            double ClosedCall(double other);

So, how do we create the delegate?  Normally, since C# (pre-Orcas) doesn't have syntax for creating closed static delegates, you are forced to use one of the Delegate.CreateDelegate overloads:

ClosedCall addToOne = (ClosedCall)Delegate.CreateDelegate(
        typeof(ClosedCall),
1.0, typeof(NumberFunctions).GetMethod("Add",
BindingFlags.Public | BindingFlags.Static)); 

Of course, we just spent 2entries looking at a helper that can do this for us (I'm not claiming this is better, I just want you to be able to see what's happening):

ClosedCall addToOne = DelegateBinder.Bind<ClosedCall>(1.0,
        typeof(NumberFunctions).GetMethod("Add",
BindingFlags.Public | BindingFlags.Static));

Now, a call to addToOne(someNumber) will yield the result of adding the supplied argument to one.  This is a contrived example, but you could imagine taking a method (perhaps generated on the fly via LCG), and "attaching" an instance to it via the first argument.  Then, being able to call it many times with different subsequent arguments, or passing it to another component that would provide the rest of the arguments.  In this way, you get the benefits of not having to keep track of an instance, without having to own the API for the instance.  Additionally, you could "chain" delegates together so that many arguments are captured in a stack of delegate calls, allowing closure-type semantics at the cost of some stack space (although since C# has closure support, you'd never really need to do that).

What's really cool is that with C# 3.0's Extension Methods feature, we now have language support for creating early-bound closed-static delegates.  If you bind a delegate to an extension method (using the regular syntax for an instance method), you will get the exact IL for creating an early-bound closed static method without our fancy helper class.  Let's see how that would look.  Let's use a different example to keep us on our toes.  Here's a helper function that creates email addresses:

            public
            static
            class StringExtensions
{ publicstaticstring MakeEmailAddressWithAlias(thisstring domain, string alias)
{ returnstring.Format("{0}@{1}",
alias, domain); } }

Notice the "this" in front of the first parameter, this tells the compiler that the method should be considered when resolving method calls for string.  We'll use one of the delegate types provided in Orcas. Now, here's how the bind looks:

            string fooDotCom = "foo.com";
Func<string, string>
makeFooDotComAddress = fooDotCom.MakeEmailAddressWithAlias; string email
= makeFooDotComAddress("bar");

So, the result is that email will be bar@foo.com.

Hopefully, through these contrived examples, you can see the scenarios that closed static methods provide, as well as learn how you can create one the easy way with extension methods in Orcas.

In my last post, I showed a nifty way of constructing "early-bound" delegates using LCG.  Here's the same helper class implemented without LCG:

            public
            static
            class DelegateBinder
{ publicstatic TDelegate
Bind<TDelegate>(object firstArg, MethodInfo
method) { return (TDelegate)Activator.CreateInstance( typeof(TDelegate),
firstArg, method.MethodHandle.GetFunctionPointer()); } } 

This one is quite a bit simpler, and extrapolating from what we learned last time, it's easy to see what's happening.  Hopefully, you are already familiar with the Activator class.  Basically, this just shows the managed call chain that produces a function pointer to a method given a MethodInfo.

I really like the LCG-based implementation, but only because of my love of DynamicMethod.  It's pretty complex, and aside from opportunities for caching, doesn't really have anything over this implementation. This one is just plain simple, and would have a single-line implementation if I hadn't put some line breaks to avoid formatting problems.  It does, however, highlight the annoyingness of having to work around the compilers' "helpfulness" when it comes to delegate construction.  If only I could just call the constructor directly.

It is worth noting that this doesn't work in the Silverlight 1.1 alpha or the compact framework (or XNA for that matter), neither of which expose RuntimeMethodHandle.GetFunctionPointer().

In a previous post about delegates, I discussed the following interesting cases of delegates:

• Closed static
• Open instance

See the previous post for the full explanation, but these basically open up some interesting dynamic scenarios.  The problem is that C# and VB do not expose syntax for constructing these in an "early-bound" fashion, that is using the special constructor on the delegate type rather than Delegate.CreateDelegate (which more or less binds via reflection).

For most scenarios this is not a huge problem, but there are some performance considerations and other issues to consider that I don't really want to dig into at the moment.  One sufficiently important scenario is testing early-bound invocation.  If your language doesn't support something, how can you test it?  Well, you can write the whole test in IL, but that is not a terribly maintainable proposition.

Another option is to only write the part you need in IL.  Unfortunately, C# doesn't allow you to write inline IL, but you can use Reflection.Emit.  And, since v2.0, you can use LCG (Lightweight Code Generation) via DynamicMethod.

The trick here is to understand how delegates are instantiated.  Delegates are just classes like any other.  They inherit from MulticastDelegate (typically).  The special part is that the runtime provides all the implementation and they have a special constructor.  Here's (approximately) the constructor signature for System.Action<T>:

            public Action(object o,
IntPtr method)

Object? IntPtr?  What the heck? Well, it's not as bizarre as you might think.  The object is simply the first argument for the invocation.  This allows binding to a particular instance ("this" for instance methods, arg 0 for static methods). The IntPtr is a pointer to the method.  "Pointers?!!?!?! in managed code?!?!" you say?  That's right, a pointer.  An object is easy enough to come by, but where do I get the pointer?  Well, the pointer can be easily retrieved via the ldftn opcode.  It loads the address of a given method (described via a token in IL, and a MethodInfo in Reflection.Emit).

Lets cut to the chase.  Here's a little class that can bind a method to a delegate type and allow you to provide the first argument (you'll need System, System.Reflection, System.Reflection.Emit using statements):

            public
            static
            class DelegateBinder
{ publicdelegate TDelegate
Binder<TDelegate>(object firstArg); publicstatic TDelegate
Bind<TDelegate>(object firstArg, MethodInfo
method) { DynamicMethod dynMethod = new DynamicMethod("PassthroughBinderImplementation", typeof(TDelegate), new Type[]
{ typeof(object)
}, typeof(DelegateBinder)); ILGenerator gen =
dynMethod.GetILGenerator(); //load the first argument gen.Emit(OpCodes.Ldarg_0); //load
the address of the method gen.Emit(OpCodes.Ldftn, method); //create
the delegate gen.Emit(OpCodes.Newobj, typeof(TDelegate).GetConstructor(new Type[]
{ typeof(object), typeof(IntPtr)
})); gen.Emit(OpCodes.Ret); return ((Binder<TDelegate>)dynMethod.CreateDelegate(typeof(Binder<TDelegate>)))(firstArg);
} } 

With this class, you can dynamically construct all the early-bound variants (ignoring variants for signature relaxation) like so:

            using System; using System.Reflection; using System.Reflection.Emit; publicdelegatestring Passthrough(string str); publicdelegatestring BoundPassthrough(); publicdelegatestring ProgramPassthrough(Program
p); publicclass Program
{ staticvoid Main(string[]
args) { Console.WriteLine("Open Static:"); Passthrough
ospt = DelegateBinder.Bind<Passthrough>(null, typeof(Program).GetMethod("StaticImplementation", new Type[]
{ typeof(string)
})); Console.WriteLine(ospt("Hello World")); Console.WriteLine("Closed
static:"); BoundPassthrough cspt = DelegateBinder.Bind<BoundPassthrough>("Hello
World", typeof(Program).GetMethod("StaticImplementation", new Type[]
{ typeof(string)
})); Console.WriteLine(cspt()); Console.WriteLine("Open
Instance:"); ProgramPassthrough oipt = DelegateBinder.Bind<ProgramPassthrough>(null, typeof(Program).GetMethod("InstanceImplementation",
Type.EmptyTypes)); Console.WriteLine(oipt(new Program("Hello
World"))); Console.WriteLine("Closed Instance:");
BoundPassthrough cipt = DelegateBinder.Bind<BoundPassthrough>(new Program("Hello
World"), typeof(Program).GetMethod("InstanceImplementation",
Type.EmptyTypes)); Console.WriteLine(cipt()); } publicstaticstring StaticImplementation(string str)
{ return str; } public Program(string payload)
{ _Payload = payload; } string _Payload; publicstring InstanceImplementation()
{ return _Payload; } }

So, there are certainly cases that will break this, most involving incompatible signature issues between the method, delegate, and the first argument.  But I didn't want to make things more complicated for an example. Besides, the point of this is not really to give you some neat tool (you'll probably never need to do this), but to give people a better idea what the compiler is doing for you when you create a delegate.

A quickie here. I had written some code in javascript to simplify testing of a fairly dense, multi-dimensional feature area.  I could define the dimensions I wanted to test declaratively in a more or less JSON fashion and then bang them together via some enumerable-like extensions that I had written.  Viola, full test matrix implemented.

Unfortunately, I learned that I would be unable to take a dependency on the scripting host at run-time.  Too bad I had already invested the time in the solution, and I would now have to code the matrix myself.  But wait, this is a dynamic language I'm dealing with!  So, in a short time, I did some object substitution, and now my javascript is a precompiler that emits C# and other support files that I then compile to do the same thing.

Nikhil has a more general tool that goes the other way.  It compiles C# into javascript for use in AJAX apps.  While this is a handy way for people unfamiliar with the power of dynamic languages to jump into the AJAX world, I'm wondering if it's really a good idea in the long term.  On the one hand, reducing the number of language dependencies in a project is good for maintainability, but choosing the less-flexible one seems like the wrong choice.  Of course javascript has no formal set of class libraries, so that's a limitation.  Hmmm.  I'll have to think about this some more.

So, my last post talked about delegates.  In it, I mentioned some compiler trickery involved in declaring events, but I didn't bother explaining it.  After reading it over again, and getting some feedback, I felt bad about glossing over what is pretty much the mainline scenario for delegates. So, what is an event?

An event is kind of like a broadcast. It enables an object to notifiy subscribers when some "event" occurs, give them relevant information about the event, and allow them to do something in response.  And, you guessed it, delegates are at the core of making this work.

Fundamentally, events are a callback mechanism, and could be implemented without delegates using anything from raw function pointers to interfaces, and the CLR doesn't keep you from doing either of those, but there's value in a consistent pattern.  In fact, the designers of the CLR felt so strongly about the value of this particular pattern, that it is part of the CLI spec (along with properties, another pattern that is implemented by other more fundamental constructs).

So, how do you make an event?  Well, in C#, you declare an event like you would declare a field whose type is some delegate and you add the "event" keyword.  So, somewhere in a type, you would have something like:

public event EventHandler Click;

Whether or not it's public depends on how you expect the event to be used.  EventHandler is a delegate with the following signature:

void EventHandler(object sender, EventArgs e);

This signature is another pattern that I'll talk about later.  For now, lets look at what the compiler does for our event declaration.  The compiler gives 3 things (if you don't count the things it already did for the delegate EventHandler):

• A private field whose type is the delegate EventHandler
• A (public in this case) method "accessor" for adding delegate callbacks: add_Click //Click comes from the event name
• A (public in this case) method "accessor" for removing delegate callbacks: remove_Click

When other code wants to hook up to your event, they use the += operator on your event.  This is really syntax sugar for calling the add_Click method.  And, conversely the -= operator calls the remove accessor.

Interestingly, you can write your own implementation for the event pattern.  You might want to do this to save size in a possibly large tree structure with lots of events at each node.  ASP.net does this with controls.  Rather than every Control having tons of fields for each event, it has a sparse dictionary of event delegates, that is only populated for events that have "subscribers".  With a tree that can easily have thousands of controls per page view, this results in a sizeable savings.  How do you do this?  Well, in C#, you use the little known syntax:

public event EventHandler Click {

add {/* do something with value in here */}

remove {/* do something with value in here */}

}

Looks like a property eh? This causes the compiler not to create the 3 things I mentioned above. Instead, it calls your add and remove accesors to do the adding and removing (via the value keyword just like properties).  In it, you can do anything you want, although it's advisable to keep the same semantics as the default implementations.

So, lets talk a little bit about what happens when an event happens and it is called.  Let's say that several other classes have registered for your event (via the += syntax or whatever the compiler supports).  Inside your class, you simply call the delegate (there's a recommended pattern for this as well).  But wait, there's more than 1 subscriber!  Remember, delegates aren't just function pointers, and they are more powerful than using interfaces alone.  If you'll recall in the last post, I said that when you create a delegate, you're really getting a MulticastDelegate, which tracks an invocation list of delegates to run. (this is why the standard event pattern returns void, otherwise, you've got the weird situation of multiple return values from what appears to be a single call).  Under normal circumstances, each delegate in the invocation list is called and execution resumes.

Soon, my ownership area will extend to include delegates.  Since I'm fairly excited about this, I thought I'd celebrate by writing a little something about them.  So, what are delegates?  A casual observer might be tempted to write off delegates as a sort of managed function pointer.  While this comparison is certainly accurate, there's much more to explaining the power of delegates.

In general, delegates are a sort of universal method dispatch mechanism.  Initially, the scenario they supported was callbacks.  Delegates are one of the things that distinguish the CLR from other VMs like Java.  Java requires the use of interfaces to implement callbacks. (I'm only calling that out as a distinction, not saying the Java way is bad. although personally I like what delegates bring to the table)  So, delegates let you wrap up a method as an object to pass around, with the expectation that it will be called from another context.

Its sort of hard to talk about delegates because the discussion is often framed by the language that's exposing them.  Currently, no managed language exposes them in the way that they are represented in IL.  In C# and VB, you declare a delegate by simply defining a method signature.  From an IL perspective, the compilers generate a class that inherits from MulticastDelegate (another story I'll get to later), with an Invoke method that matches your signature, and some various constructors to support different things.  (You also usually get the corresponding asynchronous calling pattern support methods, but I don't want to get into that)  Some other delegate-related compiler trickery is involved in declaring events, which I'll cover later.

Under the covers, a delegate [conceptually] contains 2 things:

• A target object
• A target method

Now, generally speaking, the target method is the method to be run, and the target object is the object on which the target method will be run, but there are cases where this line is blurred a bit.  For instance, when a delegate is pointing to a static method, the target object is conceptually null (internally it's not, but that's an implementation detail).  I'll get into the other cases later.

So now you're saying, "Yup, that's a delegate.  Big deal.  What's so cool about that?"  What's cool about that, my friend is that delegates are the things that power virtually all of the coolest new language features that came out in v2.0 and will be coming out, including all the dynamic language goodness like IronPython.  It's the dynamic stuff that is really exciting, so let's talk about how delegates enable dynamic languages on top of a statically typed system.

(To be fair, Jim Hugunin did his initial Iron Python work before these features were available, but they now play a big role)  One of the pieces of work done in v2.0 was called delegate relaxation.  Previously, the target method had to match the delegate signature exactly.  Now, as you might expect intuitively, the signature can be relaxed such that the target method can have "more general" parameters and return something "more specific" than the delegate's signature.  This is typically defined in terms of covariance and contravariance, terms that even people who understand them get confused.  Here's the way I usually remember it: If I could wrap the target method with a method having the delegate's signature without casting, it will work. Anyway, this feature makes delegates quite a bit more flexible.

Before I go into the other features, lets talk a little about implementation. In normal method calling in the CLR, the first parameter becomes the "this" object.  (Which is why you see ldarg.0 in IL to put it on the stack.)  So, conceptually, the target object represents the first argument for the method. (There is an implementation detail that allows static methods to be called using the same convention, which is a really elegant solution) So, by extending this idea of the target object simply being the first argument, we get a couple of interesting variants.

The first is what is called "closed" static delegates.  This allows you to specify the first argument of a static method at delegate creation rather than at the callsite.  Notice this maps quite nicely to the dynamic language concept of adding a method to an existing instance of an object.  The language runtime just needs to be able to track these extra methods as part of its method dispatch logic.

The second feature is "open" instance delegates.  This allows you to create a delegate that points to an instance method, but doesn't define the target object.  Instead, the delegate signature can have an extra first argument that will specify the target object at the callsite.  When used with LCG (DynamicMethod), this can be used to implement things like adding a method to an existing type.  Again, the language runtime merely needs to add the logic to method dispatch.

These 2 features are intriguing to me because they are not directly exposed from VB or C#.  I believe VB9 exposes these, but they are not accessible in an early bound way in C#.  You can, however, create them via Delegate.CreateDelegate() using reflection, or use Reflection.Emit to generate the corresponding IL.

Hopefully, I'll have some time in the future to do some samples of these as well as discuss more about how these improve the dynamic language support in the CLR.

I dealt with several situations in the past months where the crux of the problem was confusion over assemblyname and filename.  Let's define what we're talking about:

• Filename - The name of a file in the filesystem, such as System.dll
• Assemblyname - The name given to an assembly to establish its identity.  In this case, we'll only concern ourselves with the "simple" name. such as System

Usually, any confusion that arises between the two can be resolved by reminding people that a filesystem is just one of the places you can get an assembly from.  For instance there are APIs for getting assemblies from byte arrays.

For those that still do see it... In the managed world, the assemblyname gives identity to the code that resides in the assembly.  If you have 2 assemblies with the same assemblyname, you expect them to represent the same identity (perhaps different versions, build flavors, bitness, etc.).  If we relied on the filesystem name, the identity of the code could change just by changing the filename.  That's not the semantics we expect.

So, why does the filename matter?  Why do we recommend keeping them the same?  Some of the reasons are simple convenience.  It's nice to look at a file and know what it is without cracking it open.  If the names are different, it's like me going to a party and wearing a nametag that says, "Peter".  While there is nothing keeping me from doing it, it causes confusion.  However, another more important reason to keep them the same is that assemblies are rarely loaded by filename.  References and most dynamic loads are done by assemblyname.  You don't take a reference to System.dll, you take a reference to System.  At some point, the loader has to find an appropriate file to load to satisfy that reference.  If System's filename is Peter.dll, then it's going to have a difficult time finding it to load.  This is actually the very reason that gacutil will not let you install an assembly into the global assembly cache if the filename doesn't match the assemblyname.  However, I think it's silly that it doesn't just fix the name for you.

What about multi-module assemblies?  Well, it's the module with the assembly manifest that matters.  It's the one that should match.  Then the rest of the files need to match the assembly manifest :). But, if you're using multi-module assemblies, let me know.  I'd like to know why.

I've mentioned before that one of my ownership areas at MS is the CLR "shim".  Most people I've said that to ask me, "What's that?".  I usually reply that generally, it's mscoree.dll, to which they typically respond, "Oh yeah.  What does that do?"  In general terms, the shim is in charge of firing up the runtime in a process.  In addition, it exposes all the hosting APIs and other stuff you need to do stuff with the CLR from unmanaged code. If you look at a managed app, you will see that it has a dependency on mscoree.dll, and nothing else CLR-related.  Ater the runtime is spun up, most of the things that mscoree exposes are simply forwarded calls into the mscorwks.dll of the runtime you have installed.

What's interesting about mscoree.dll, is that it is the only piece of the runtime that doesn't run side by side.  You can have v1.1 and v2.0 installed on the machine, but you will only have one mscoree.dll.  You always have the version of mscoree.dll that corresponds to the latest version of the runtime installed on your machine (unless you have installed a patch or something that services mscoree, in which case you may have a v2.0 shim even though only v1.1 is installed on your machine).

So, naturally, backwards compatibility is extremely important in the Shim.  When you start up managed code, the Shim decides which version of the CLR to fire up based on lots of different things.  These things are all fairly well documented and all have a specific scenario they enable, but by their nature they are very confusing.

Aside: When I was job hunting, I interviewed at several companies other than Microsoft.  During some of those other interviews, I was asked questions about what runtime would be started under certain conditions. The rules are so confusing, that some of the interviewers, although all extremely smart people, had formed incorrect models of what the rules were.  Some told me I had the wrong answer to their question, when if fact it was correct. (that's not to say that I knew the correct answers to all of them.)

I'm not certain that I can clear up the confusion, but I do hope to have a series of posts in the coming months on why the shim does what it does under certain circumstances.  Then, at least you might understand what's going on when it happens.

Someone asked me the other day if you could reflect on other people's assemblies using the CLR.  My answer was, "ABSOLUTELY!!!!!"

As it turns out, they were having some problems achieving this, and they were wondering if there was some kind of security mechanism in place that was preventing reflection on 3rd party assemblies.  Here's the basic scenario.  They had a 3rd party library with a type that defined several "constants" using fields.  They needed to be able to specify a named constant via a string and return the value for the constant.  Security and performance arguments aside (they had already been considered), they simply wanted to lookup the field by name via reflection and get its value.  This can be accomplished via just a few fairly reasonable lines of code using one of the GetField family of methods on System.Type and then getting the value.  I'll leave this as an exercise for the reader so they can have the fun of wresting with the silly BindingFlags enum.

After some discussion, I learned that the trouble actually revolved around getting the Type object in the first place.  They were using Type.GetType to load the type with a namespace-qualified name as the string argument.  It was returning null (Nothing in VB).  They were validating that the string was correct using Intellisense, and concluding that since Intellisense could see the type, that Type.GetType should also "see" it (which seems like a perfectly reasonable assumption).

Type.GetType() takes a string argument specifying the type to retrieve.  When specifying types in strings, you can usually use a namespace qualified name ("[namespace.]type"), or an assembly-qualified name ("[namespace.]type, assemblyName").  If you don't provide the assembly name, the API looks through the assemblies already loaded in the AppDomain for a type that matches the name.  If an assembly name is specified, a bind occurs to the assembly and it is loaded if necessary.

In this case, there was a reference to the assembly containing the type, so Intellisense was picking it up and providing completion. The trouble was that, at runtime, the assembly had not yet been loaded into the AppDomain, so the type was unavailable.

So, the options were (in order of appropriateness in my opinion):

• Use an early-bound type - Since the type was known at compile-time, use typeof() (GetType() in VB).  This will create a compile-time, assembly-qualified type reference in the IL rather than a runtime parsing/bind/load of the type string.
• Use an assembly-qualified type string - Adding the assembly name to the type string will let the CLR know what assembly to look in (and load if necessary).  There are some subtle versioning issues with this approach, especially for strongly-named assemblies.
• Make sure the assembly is loaded prior to calling Type.GetType() - Making an early-bound call to something else in the assembly first will get it loaded into the AppDomain.  This seems like a fragile solution and I would not recommend it, although it will technically work.

The real issue here is the number of samples (especially in VB) provided by MS that use Type.GetType() with a non-qualified name (ex. "System.String").

Historically, when I gotten questions from friends and colleagues regarding performance, I've often sent them straight to Rico Mariani's blog.  It is a wonderful source of advice, guidance, rules and humor in the realm of performance.  His latest entry is another wonderful piece.

I've spoken with several individuals who have said things like, "Well, our goal was to be able to handle a request in X seconds, or handle Y amount of throughput, and we're too slow.  Any ideas?"  More than likely, they've done their design and implementation without any work to make sure they reach their performance goals, and by that time, my advice to "get rid of that dependency", or "redesign this component" is too late.  When factoring the performance goals into those design decisions would have raised the red flag immediately. 

I love his approaches because they tend to sound alot more like engineering (which is my background) than alot of guidance that tends to be thrown around.  Here's my favorite quote:

I get very worried when people say things like “Productivity and cleanliness always trump performance.”   Productivity is about creating product.  A “clean” design which fundamentally fails to address performance requirements is not an example of a productive enterprise, it is a looming disaster.  A developer productively engaged in creating a failure is uninteresting.  

Now that Rico and I work for the same company (across the street from each other) perhaps I can come up with a good excuse to meet him in person.

[UPDATE] After reading this, I thought it came off a little snobbish.  I think we've all been in the above situation.  Many times, through no fault of our own.  We're often the victims of process, bureaucracy or other external forces that often oppose success.  Please don't feel like I'm talking down to anyone.  Hindsight is 20/20.

Something that always bothered me at my previous job was having to install the framework SDK to get a copy of gacutil.exe.  I know the guys still there hate having to install the SDK on a server so they can manipulate the GAC.  Richard Lander gives some interesting information on the topic, but he doesn't go into why it's not included in the redist.

Since I've been at MS, I've learned a great deal more about the Global Assembly Cache (GAC) and the fusion APIs. Last night, I was taking a shower after cutting my hair, and the reason came to me.  That reason is... installers, or more importantly... uninstallers.

During a discussion recently, I heard an amazingly profound saying:

"It is better to fail to do, than to fail to undo"

I don't recall who said it, and they probably got it from someone else, but it is right on the money.

When programs uninstall, they have to correctly remove things they've placed in the GAC.  Let's make up an example.  Let's say I have some software company.  We've developed a magical managed library that makes it wicked easy to develop our software, so we use it in all our products.  Let's say that in our deployment model, it makes sense to deploy that library to the GAC.  So, uninstalling our software should remove it from the GAC, right?  We'll what if one of our other products is on the machine? We don't want to uninstall one and break the remaining one.

When you install an assembly into the GAC via the Fusion APIs, you do so with a traced reference.  That reference tells Fusion "who" installed it.  If 2 installers install the same assembly, it's smart enough to know not to remove the assembly until both uninstall.

GACUtil, as a management tool, enables you to  use traced references as well, but it also allows you to install without a traced references.  It also allows you to force uninstalls and do lots of other screwy things.  In other words, the tool is too powerful.  Devs need to be able to do screwy things.  Administrators need to be able to do screwy things.  Regular users don't.  Give them an install package that handles everything.  Otherwise, you're bound to have a support nightmare.

If I have time, or enough requests, we'll go into how to use the fusion APIs directly to manage the GAC.

Several people have complained that I haven't made any entries in a while.  There are several reasons for this.  Aside from being generally busy, I'm signed up to do some posts about the CLR in general, and some from my ownership areas as part of my job.  This has resulted in several "half-baked" entries ranging from hosting the runtime to the Global Assembly Cache, to some other more general CLR-related entries.

In addition, MS has had several new products in the queue that were on the verge of releasing, and I've had entries waiting for them to release.  Of course, being on the safe side, I've waited for someone else to blog first, at which point an extra post from me doesn't make much difference.

And thirdly, Jenna is growing like a weed and it's hard to find time to sit down and formulate posts when you're chasing her around the house.

So, regarding technical content.  I'd like to get an idea of what people are interested in hearing about. So shoot me an email or leave a comment with suggestions for posts regarding the CLR in general or specifically within my areas of ownership:

• The unmanaged hosting API's (CorBindToRuntime, etc.). These are what you'd use to host the CLR in your own app in order to more tightly control things, or provide additional isolation or escalation policies.
• The global assembly cache
• "automatic" CLR activation - what I mean by this is what happens when the runtime is spun up my a managed app, or via COM interop.  Things like how to decide which runtime to use, etc.

In addition, I'll try to keep the personal updates coming for those who are not looking for just technical content.

For some reason, it really bothers me when I see this pattern in code (using no particular language):

This can, of course, simply be specified as:

Now, I understand that as code is refactored, this pattern can appear due to simplifications...but still...come on. It's even worse when someVar is used only as the condition of another if block.

UPDATE: This seems to occur most often in code whose ownership has changed hands several times. New owners seem to fall into either trying to edit the code while maintaining the structure as closely as possible, or by heavily refactoring. Many times, the former choice is made, especially if the original owner is still around somewhere. Usually my first instinct when I inherit code is to massively refactor. I'll post a more in-depth view on this later.

I thought some others might find this useful.

I have been baffled for a few days why certain actions in an experimental Rails app would work fine in dev mode, and then give me mysterious HTTP 500 errors when deployed.  The Rails logs would tell me everything was just fine and there was no problem. But, there they were in Fiddler... status 500.  I haven't figured out how to get ahold of the Apache logs from my host yet, so they couldn't help me.

I finally set up Apache with FastCGI myself so I could attempt to duplicate the problem.  It was immediately apparent.  The Apache log was complaining about invalid headers in the FastCGI communication.  I was using "puts" to write out to the console when running in dev mode in order to quickly debug what was going on.  This works fine when using the Webrick standalone server, but FastCGI on Apache evidently uses stdout to do the communication between it and the fastcgi processes, and writing to stdout screws up that communication and Apache reports HTTP 500, even though Rails thinks everything's A-OK.

The lovely thing about Ruby is that I was able to fix it by redefining puts to do nothing.  Ideally, you ought to use the logging mechanisms, and I will.  But that made a great short-term fix.

During the last PDC, I drooled over the announcements regarding C# 3 and LINQ.  Now we are one step closer. Yesterday, they released a new preview of those features that bind against the current version of the framework.

Skeptical language junkies* should take a gander at Don Box's latest post that attempts to explain what the big deal is. I agree with some of the people saying that the syntax really gets in the way (and I feel that way even more after spending alot of time in Ruby lately).  They need to add a language feature that indicates whether you want the lambda or the expression, then you can fall back on type inference to keep the typing to a minimum.

Overall, I couldn't be more giddy about the way integrated query works, and being able to get your hands dirty with it now is really awesome.  We're already working with data in this way in our APIs at work, so I've been able to experiment with the language and library features directly with some of our data and it's an absolute dream.  They should not wait for the next version of the framework to give us a go-live license for this.

Now, this kind of thinking needs to be applied to the web stack and MS could have a compelling alternative to Rails.

* That is, language junkies who are skeptical, not junkies of skeptical languages.

Back the the days of .NET 1.1, you'd use a Hashtable to store values with a fast lookup using a key.  Hashtable used a single array of "buckets", which stored the key, value, and a collision index.  The index to the the bucket array is a transform of the hashcode of the key (unless there's a collision).  Long story short, there was no stable ordering to the values in a Hashtable (technically, the order is super-important to the algorithm, but it's not useful on the outside).

Fast-forward to 2.0, we have the Generic Dictionary.  It's algorithm is quite a bit different.  It has an array of Entry<Key, Value>, a private nested class similar to bucket in the Hashtable.  But, the "bucket" data is stored in a separate array of int, which holds the index into the Entry array for that hash.  When you add something to a Dictionary, it is simply added to the Entry array, and the bucket array is the one that's updated and possibly re-ordered.  Long story short, there IS meaning to the order you get from a generic Dictionary (using the enumerator).  It's the order you added them.  This subtle change adds alot of value to Dictionary in my opinion.

It was just a very interesting realization to me, and I thought someone else might find it useful.

As I mentioned earlier, I've been playing with WF.  When debugging in Visual Studio, a program running a workflow pretty much uses 99% of CPU the whole time.  I thought that was pretty scary, especially since I had alot of Delay activities in my workflow.  Well, not to worry, running the .EXE outside of devenv gives you what you expect, a very well-behaved program.

I suspect it has to do with the sweet debugging stuff that's built into the workflow designer surface.  You can add breakpoints to activities.  It's pretty cool, despite crashing on a regular basis.  Hopefully, they'll fix this before RTM.

The two main High Definition standards are 720p and 1080i.  Since there is plenty of information on the 2, I will not go into detail on them individually.  What there isn't alot of is objective, well-reasoned explanations of which one is better.  Mostly, this is because it is not a simple topic, and the winner depends largely on the context of the question.  Another reason this is so hotly debated is that people spend alot of money on HDTV's and that causes people to become zealots of whatever particular technology they've embraced. I'll try to compare the two within each of the important contexts, without personal bias.

Let's start with the fundamental question.  From a HDTV technology perspective, which is better?  The answer is, they are the same.  Before you leave a comment on how stupid I am, keep reading.  Fundamentally, we're talking about how much data can be displayed at a time.  For TV, that is limited far more by transmission bandwidth than anything else.  To maintain that bandwidth, the signal is compressed in a lossy fashion (see update below). In either format, you're going to get roughly the same "picture quality" watching TV.

Now, you're saying "shut-up, idiot.  One has got to be better.  Why won't my TV do 720p (or 1080i) if they are the same?". This is where the argument gets interesting.  The signal formats were created for "analog" TV technology, namely CRTs.  For CRTs, it's all about frequency.  (How many lines can you draw per second) 720p has more lines per second.  The problem is, you're still bandwidth limited, whatever that limit may be. Horizontal "resolution" comes down to bandwidth.  Any increase in bandwidth basically gives you more capacity for horizontal resolution, so again the two are the same.  The problem is, many HD sets are based on technologies with a "native resolution", meaning the TV is locked in to a certain resolution.  In these cases, the interlacing trick used by 1080i to gain temporal resolution at the cost of spatial resolution is useless.  [Added:] Many CRT-based HD sets cannot do 720p because it requires a higher horizontal refresh rate and the flyback circuit capable of 1080i is not capable of that speed (in short, you got screwed (myself included) to increase their profit margin). 1080i is refresh frequency comparable to 540p (if it existed as a standard).

So, the result is that most HD sets have a native resolution at (or near) 720p.  So, for those sets, a 720p signal matches the native resolution of the set and "looks" better.  There's alot of hype around 1080p sets, capable of displaying, you guessed it, a 1080p signal, which only really exists as the output of a PC at 1920x1080, which brings me to my next point.

Since basically all HD content is stored digitally, we're not just looking at lines of resolution, we've got frame sizes.  So a 720p frame is 1280x720, and you get 2 per second.  That's 1,843,200 pixels per second.  A 1080i frame is 1920x1080, but you only get 1 per second. That's 2,073,600 pixels per second.  So, compression and bandwidth aside, dealing with digital signal sources, you're getting more data (in the form of pixels per second) with 1080i than you are from 720p.  If that's the definition of better, then 1080i is better.

But, as I mentioned before for fixed resolution TV technology (anything but CRT), it takes a 1080p set to show 1080i in all it's glory.  Otherwise, 720p will probably look better due to the down-conversion resampling. Clear as mud, eh?  Well, I hope this clears it up for at least one person. [Added:] A signal will almost always look better in its native form.  Whenever there is a resampling step in up/down converting, you're going to have some degradation due to aliasing.

[Added:] I owned a 1080p DLP set for a few short days, and it was by far the prettiest thing I've seen.  it displayed both 1080i and 720p signals beautifully.  I sent it back, though.  All the upconverting in the video pipeline introduced a noticable delay that made playing video games very frustrating. :)

[UPDATE:] I had a comment asking about the lossy compression applied to video.  I'm referring to the digital compression that the video signal undergoes as part of the ATSC standard.  It uses MPEG2 compression (basically the same used for DVDs) to reduce the digital bandwidth (and therefore the analog bandwidth) that the video signal takes up so it fits in the "channel" (6MHz wide if I remember correctly).  This particular compression degrades picture quality (the amount can be controlled).  It is both a spatial and temporal compression, and gives very dramatic compression ratios.  Cable companies have more bandwidth to work with, but they usually pack the crap in as tight as they can rather than giving us better quality...jerks.

Note, there is also some "analog" compression going on as well to squeeze the ~20Mbit/sec digital signal into the 6Mhz channel.  The ATSC standard uses an 8-bit "vestigial side band" modulation (8VSB), while cable companies use a form of Quadrature Amplitude Modulation (64-QAM).  Both are beyond the scope of this article and neither of these come into play in picture quality as they are lossless forms of compression.

If you haven't guessed, I did a research paper on HDTV back in college.

[Yet another UPDATE:]  Want some hands-on proof?  Go download the 720p (146MB) and 1080i (211MB) versions of the King Kong movie trailer from Apple's site.  Compare the file sizes and then open them both, and you decide which one shows you more "data" (my definition of "better").

Another geeky post.

We built our own query system several years ago.  It used the concept of "query by example" with templates that were interpreted into SQL.  You could generate the equivalent of an "IN" clause by assigning something that implemented System.Collections.ICollection to a template property.  Recently, with the introduction of generics and the generic collections, we decided to relax that contract to IEnumerable since that's really the least common denominator of all collections.  This produced some hilarious results.  As you can imagine, anywhere where we were using strings to specify template values, they were being interpreted as collections of characters.  This manifested itself mostly by having queries return nothing, since our system properly handled Chars and treated them as strings.

But, this reminded me of a similar incident a few years ago that involved a framework that rendered objects as comma-separated values. So, instead of:

Mark, Miller

You'd get:

M, a, r, k, M, i, l, l, e, r

I thought since I had been bitten twice, I would write it down.  I usually don't forget stuff I take the time to write down.

So, the moral of the story: Remember that even though System.String implements System.Collections.IEnumerable (and now System.Collections.Generic.IEnumerable<System.Char>), you usually don't want to treat it as a collection.  You may need to special-case it.

Anyone else think of another type like this?

My friend Peter and I often use Remote Assistance to collaborate on projects.  He and I were doing some things today in Visual Studio, and while I was watching, a beautiful window appeared that showed the open files and windows and such, complete with appropriate icons. "Whoa!", I said, "What was that?!"  "What", he replied, "this?"  The window re-appeared.  Turns out, it was the Ctrl-Tab window.  For some reason, I never even thought of using it to browse the open windows. I always assumed I had to navigate the frightful tabbed interface, which just might actually be usable with that little trick.

I felt better when I typed in a type name and Intellisense didn't recongnize it.  A quick right-click->Resolve instantly added the appropriate using statement.  Peter hadn't seen that feature.  Although I guess Ctrl-Tab's been a standard for much longer than Resolve.

This past week, I've found myself writing alot of methods with the following pattern:

IEnumerable<T> DoSomething(IEnumerable<T> stream);

Then I use C#'s wonderful new iterator syntax to manipulate the stream, or simply intercept and process the data. I've found this wonderfully useful for alot of the statistical type algorithms that I implement for our project. It's been especially cool to combine this with a bit of reflection for creating "dynamic filters" based on user input. All of a sudden, all your algorithms are incredibly flexible with very little effort.

The beauty of this pattern is that it can be chained. Obviously, there is a practical limit here, but I've found it extremely useful and I thought I'd share it for those who haven't stumbled across this concept before.

Now, if we could only create iterators using anonymous delegates, then I would proceed to drown in my own drool.

[UPDATE] fixed title grammar. Sorry if that pops it back up in your aggregators, RSS readers.

I blogged a few days ago about a crazy problem I had with garbage collection in ASP.net under 2.0.  I finally tracked the problem down to Array.Sort.  I created a framework for working with large sets of data in a dynamic way.  This allows us to group data and do other complex statistical things based on fairly open-ended user input, rather than having to write special case code for each scenario.  It uses some reflection to be able to drill into objects.  When sorting, we pass in a dynamic IComparer that can drill into each object and do dynamic comparisons.  Unfortunately, many of our algorithms were dependent on sorting the data first.  This was causing a huge number of allocations due to drilling in and boxing lots of value types.  And, the nature of the sort causes that to happen multiple times per item.

Under 1.1, we took the hit on memory.  It didn't take that long, and it was soon collected.  Under 2.0, the GC seems to recognize this allocation pattern early and proactively begins collecting aggresively.  The problem seems to be that this slows down the sort tremendously, to the point where it essentially comes to a halt.  After a long time, you get a bizarre looking exception about a failed sort from deep in the framework.

What I can't figure out is, during all this time, there's still plenty of memory, and activity on other requests is not impacted horribly.

Well, I took a long hard look at some of the operations, and re-implemented them with a hashing strategy rather than relying on a sort.  As it turns out, for most cases, this is much more efficient.  Long story short, overnight stress testing turned up with clean results this morning.

This is awesome.  Joe Duffy reveals what I had suspected.  A change to the CLR that will fix the issues I've previously discussed about nullable types in version 2.0!  From his blog:

The core of this change is that the IL box instruction has been modified to recognize Nullable<T>s. For non-Nullables, behavior remains the same; but upon seeing one, it inspects its HasValue property. If HasValue is true, box peeks inside the structure, extracts the T value, and boxes that instead; otherwise, box simply leaves behind a null reference. Obviously, unbox has also been changed to allow nulls to be unboxed back into Nullable<T> structures. This had a rippling effect in the CLR codebase and also required changes to late-bound semantics to mimic the static case.

This is fantastic, and reveals just how strong Microsoft's commitment is to the development community.  I gave my feedback on this before.  I felt it was a problem that aware developers could understand and live with, but I felt that novice developers would struggle with it, and ultimately it would make the feature, and the platform less understandable and approachable.

As you can tell from my earlier post, I had pretty much decided that MS would be unable to fix it at this late stage in the game, and that would be a shame.  But, thanks to some good decision making, they've done it.  Also, the solution was quite similar to my suggestion, I'm happy to say.

Be sure to follow through and read Somasegar's post on the subject.

As if I didn't have enough fun stuff to play with, the first Windows Vista beta was released last week.  I put it on my little laptop first, where it ran very well, but I was disappointed that the new visuals didn't run on the crappy video card it has.  So, last night, I put the 64-bit version on my PC, which has a recent ATI card on it.  Everything runs great on it and it's quite beautiful.  Both installs I've done have been utterly trouble free.  It brought back pleasant memories of installing the 95 beta and waiting with anticipation through the "Starting windows for the first time" screen, which always reminded me of that cereal commercial..."taste it again... for the first time".

This evening, I'll be putting my apps back on the PC, so we'll see if I run across any problems there.

Sorry for the sparse updates.  I've been head-down on quite a few projects recently, so most of my ramblings have taken the form of internal blog entries and emails.  For the last several months, I've been pushing my co-workers to think about migrating to Whidbey (The next version of Microsoft's .Net Framework, the CLR, whatever you want to call it) sooner than later.  Most people know I'm an early adopter of new technology, and always take my suggestions as the ravings of someone infatuated over new things.  However, we're finally moving that direction for one of our biggest tools as a result of some performance optimizations I was able to do with new features provided in 2.0 (namely generics, iterators, and some new asp.net goodness).  The performance opportunities alone are return enough for the investment.

Rico shares with us an overview of these opportunities, and it's a good enough list that even a manager can see the benefits, many of them handed to you without any additional work.

A large part of my time at work recently has been addressing scalability issues in some applications.  Most of these stem from poor use of memory.  Most of these applications were proof of concept that made their way into production code for whatever reason.  More and more we identify problems and when we identify them to the developer, this is the typical exchange:

Me: You're taking too much memory here by using type Xxxxx when you could have gotten by with type Yyyyy

Dev: I have to use type Xxxxx, that's what the data is.

Sometimes, you just have to take a step back and look at your bytes.  Sometimes, you need a profiler, but sometimes you can just do some math.

For example, if I have an array of dates that correspond to events, a logical choice for a datatype is System.DateTime.  However, if you've got millions of events, System.DateTime is relatively expensive.  Especially if you consider that your event times only have resolution down to the nearest minute and only represent times within the last several weeks.  The range, or domain of DateTime (ticks since the epoch) is overkill for your circumstances. You may have been able to get away with UInt16, which would have been 2 bytes instead of 8, which is a huge reduction. (Provided that you've shown this array of DateTime to be a significant part of your memory consumption.)

I'm sure there is some official name for this, but I'll refer to it as "constrained domain", where you know something about your circumstances or usage that allows you to reduce the range or domain of a concept in order to store it as something smaller to improve memory consumption.  Remember, just because a big, rich datatype makes things easier, matches a db schema, or is named the same as the concept you need to store, doesn't mean it's a good match for your scenario.

There are, of course, other considerations to make.  The Y2k problems were the direct result of taking this concept to the extreme (of course resources were in much shorter supply then than they are today).

So, when you are looking for places to trim your memory usage, look for places where you can constrain the domain of a concept and store it in fewer bytes.  Also, if you're in CLR-land when you do this, look for boxing value types.  Remember, bytes in a System.Collections.ArrayList take up 5 bytes each, not 1. (9 bytes in 64-bit land) (Yeah generics)

Back last year, I outlined a problem I had with calling Response.End from a thread other than the thread that handled the request.  It stemmed from a progress mechanism we had implemented that had the ability to cancel long-running tasks when the user hit stop or closed the browser.  As it turns out, that wasn't the whole story.

We recently added a few reports where, the majority of the time, gathering the data takes over a couple of minutes.  While doing memory and performance optimizations, we noticed our worker process was recycling with the following message in the System event log:

Googling for this error doesn't do you much good since it could be caused by a variety of reasons.  Most of the advice I came across wasn't well thought out and made quite a few bad assumptions.  So we added some more debug logging and determined that this was happening when our canceling mechanism kicked in when a user decided they didn't really want to wait 5 minutes for the report.  I was greatly puzzled by this since this feature had been tested thoroughly and had been running in production for some time.  Looking back through the server logs, it was evident that it had been happening all along, just not very often. We had just gotten the performance on the vast majority of pages to be very good and it wasn't an issue.  The problem only came to the surface when we added the report that always takes a while.

Here's the problem.  If the handling thread is aborted, it causes a condition that IIS considers to be bad and that forces the worker process to recycle. (presumably, there is some communication that doesn't occur) This doesn't happen when Response.End() is called because it passes a special exception as the exception state to Thread.Abort.  The HttpApplication catches ThreadAbortException and checks the ExceptionState.  If it is an HttpApplication.CancelModuleException, it knows there was either a timeout, or Response.End() was called, and it cancels the Thread.Abort by calling ResetAbort which allows the thread to continue running at that point.  I thought that was pretty slick.

When I was aborting the handling thread, I was on a different thread, so I had to call Thread.Abort manually, so CancelModuleException was not being used, so the thread was ending completely and causing the recycle.  Since HttpApplication.CancelModuleException is internal (and rightly so) I could not simply use that mechanism.

The good news is that the Unload event always happens (for all practical purposes), even when the thread is being aborted.  So I added my own PageIsCancelling property to our base class Page and check it, along with the current ThreadState in Unload, and cancel any pending abort if the page is canceling.  So, the abort is contained within the callstack of the page, and the thread stays alive and all is well.  No more crazy recycling.

As an aside, it seems this is aggravated by multiple processors, which might explain how it passed testing on the dev's machine.  Although I don't have any proof of this.

The generic Nullable type in Whidbey and the C# language features it supports are great.  They've enabled me to eliminate gobs of code and simplify things incredibly. But, Joe Duffy's latest request for feedback on some of the "syntactic suger" on this feature brings up some interesting points.  He shows us several different equality tests against a nullable value type and asks if the results are intuitive.  I suspect it's a loaded question, because they are not.  Someone with a good understanding of the framework can fairly easily guess the outcome, but deep down I think they would have to admit it's not ideal.

What it boils down to is what is the purpose of the generic Nullable type.  I originally viewed it as helping to code against scenarios where values can be null like databases.  Without nullable value types, you frequently have to check to see whether you're dealing with a null value, and carry around alot of state information, and Nullable simply does that for you.  I think some of the designers of this feature see it much more broadly as a concept to unify the type system in a more performant way than simply being able to cast to object.  This requires some special casing, as is evident by the special case for type parameter constraints on generic types.  Specifying struct will not allow you to use a generic Nullable, despite the fact that it is a value type.  The problem is that the special casing stops short of completely abstracting the concepts.

The problem can readily be seen in Joe's second two tests, boxing a valueless "int?", and using the equality operator "==" against a generic argument that is instantiated as "int?" and has no value.  Both do the opposite of what you might expect if you don't think through it first.  Both do not equate to null.

I pondered this for quite a while before deciding how bad this was (and I'm still not exactly sure).  For me, this is no problem since I understand the issues at work.  For the future of the language, the outcome is not so clear.  The real question is, "Is a value-less value type different from a null reference?"  I don't think it should be.  If you cast a null string reference to object, does it retain its identity as a string? No. Can you cast it back to a string reference? Yes. So, should there be a difference in behavior with nullables? I'll let you decide.  Then there is the issue of how (and if) to effectively fix it.  Adding a conversion from Nullable to object that makes it a null reference instead of a boxed value type might do it, at least for the boxing scenario.  That's probably a pretty naive solution.  I haven't really thought that through.  For the generic situation, Nullable seems to be already special cased to some degree.  It doesn't seem like it would be too much of a stretch to generate appropriate code for Nullable.  The big problem here is that since this happens at JIT time, it's no longer a C# feature. Unfortunately, it's likely too late to address the issue in the in any real way, and that will likely set too much precedence to change it in the future if a solution could be reached.  Perhaps some FxCop rules could find these types of situations and at least flag them so they could be looked at more closely.

We have a big configuration file that maps business logic concepts to their currently used implementations so we can dynamically switch these out.  Under ASP.NET 1.0, all those implementations were in assemblies with "well-known" names (meaning we know them deterministically before compile-time).  Some of the implementations were in nested classes of user controls (we'll save the debate over whether that is a good idea for another time).  This means that the ASP.NET compiler is now in charge of them, which makes their names more difficult to discover.

Luckily, there's someone looking out for us here.  The System.Web.Compilation namespace has all kinds of goodies to help us out here, namely the BuildManager class.  It has a GetType overload that at first glance appears to do exactly what we want, unfortunately it only seems to work if the assembly in question has already been loaded.  This is not usually the case when our configuration code runs the first time.  Instead, you can use a combination of GetCompiledAssembly and good ol' Assembly.GetType.  So now, instead of knowing the assembly name, we need to know the virtual path to the compiled control. So here's a snippet that does generally what I want:

We're making the move to 64-bit at work, and one of the things that's annoying me the most is all the 32-bit shell extensions that I rely on and didn't realize it.  For example, we use Subversion for source control and use the excellent TortoiseSVN shell extension to work with it from explorer.  Since it's a 32-bit extension, it doesn't show up in the context menu.

There are two workarounds I've found for using these 32-bit extensions.

1. Use the 32-bit explorer.exe in the SysWow64 directory.  You have to have the "Launch folder windows in a separate process" option turned on, otherwise it will just see that explorer is already running and start a new 64-bit window instead of a new 32-bit process.
2. Use 32-bit IE.  There's already a shortcut to it by default in the start menu.  Just fire it up and navigate to the filesystem instead of a web page.  Voila! 32-bit extensions start showing up.  I like this method since I don't have to have a bunch of explorer processes for each window.

There's probably a cleaner way of using the 32-bit explorer, but I haven't figured it out yet.

Last night, I completed my STDF file parser that I started over the weekend.  I'm pretty happy with it.  I now fully support all of the version 4 spec.  It's written with the Feb. 2005 CTP of Whidbey.  It was primarily a good example of a "real world" application, so I wanted to use it to try out some of the spiffy new features and see how they might fit into an application that was not built for the sole purpose of showing them off.

I'm not entirely sure what I should do with it now that I've built it.  There's not much non-commercial demand for something like that that I am aware of. I might be persuaded to license it for use commercially once there is a "go live" license for Whidbey.  I'm sure it will need some testing to work out issues.  I'll probably build a viewer using it and use it at work to get the kinks out of it.  Anyway, I thought it might be useful to capture how some of these features played out in a real scenario.

Lightweight Code Generation

I used attributes to annotate my record classes in order to define the "on disk" layout of the record.  I originally put the attributes on the properties of the record, but decided to move them to the class level so that I could define the fields that are only used for parsing, and not useful after that.  At runtime, I register the record types with the parser, which uses lightweight code generation to generate converters from unknown records to the concrete records.  If you're not familiar with LCG, it is essentially Reflection.Emit without the overhead of a dynamic assembly, module, or type.  If you're unfamiliar with Reflection.Emit, it is essentially generating executable IL code on the fly, which has may benefits over generating C# or some other language and running it through the compiler with CodeDOM or soemthing else. (If you're not familiar with IL, then this entry is somewhat irrelevant) Having dealt with assembly on lots of different instruction sets from 68000 to x86 to DSPs and microcontrollers, I must say that working in IL is wonderful.  I was a bit worried about the startup time for the parser, but it seems to happen very quickly.  I'll need to experiment more to come up with the overhead of the dynamic code.  Again, I was disappointed in the lack of debugging support in LCG.

Generics

Unfortunately, generics did not work into the equation near as much as I had hoped.  I did find them very useful in places where I would normally pass a Type.  Now I can have a much stronger contract on such methods using generic methods with constraints.  I must say that I love the generic collections.  Working with strongly-typed data without having to generate the classes is very nice.  I also like being able to do a custom sort with an inline anonymous delegate with closure-type semantics rather than have the overhead of a separate class with the code in a different place.

Iterators

Iterators came into play in several places.  In the pull-based record parser, it was simple to implement IEnumerable.  I believe it only took 3 or 4 lines of code.

Delegate type inference

I don't remember what the official name of this feature is, but instead of having to do something like SomeEvent += new EventHandler(SomeMethod), you can just do SomeEvent += SomeMethod.  This seems small, but I appreciated it quite a bit.

Visual Studio Enhancements

These were a very pleasant surprise.  I hadn't thought about the exercise as testing the VS enhancements.  The snippets were the most useful improvement in this project.  Implementing properties didn't make me want to rip my hair out. Just type 'prop' and hit tab and fill in the blanks.  The built-in snippets for things like exceptions, attributes, indexers, etc. all work really well.  The one for implementing an indexer/iterator as a nested class was very interesting as well. It was also easy to create my own snippets.  I also found the strongly-typed resources to be very handy.

The debugging stuff is awesome.  I love being able to mouse over variables and drill into them dynamically.  The little popup windows for exceptions are cool, but don't really give you any good information that you couldn't get before.

I found the refactoring support to be extremely useful, although it was a little hard to shift into the mode of making sure everything compiled all the time.  Before, I would jump around alot in the code, so I would leave things in a state where it didn't compile.  It worked great once I got used to it.

For the last 2 months, I've been having a blast playing Halo 2 with my buddies online. (except Jeff, who for some reason refuses to get on despite having all the necessary ingredients) It has worked reasonably well, with the exception of some weird incompatibilities with certain people.  If they were the party leaders, I'd get the famous "We are experiencing network issues." message.  Bungie says that this is almost always caused by NAT incompatibilities.  This didn't get in the way too much because we could usually juggle around the party leader until everyone could join.

At first, I attributed this to my out-of-the-norm network configuration.  I have VoIP, so I have several routers.  When I first started playing, I had the VoIP box as the outer-most router, followed by the venerable WRT54G.  So I was doing a double NAT.  The last several weeks, I've been trying to reduce my incompatibilities.  First, I managed to set up the VoIP router inside the firewall (I have AT&T CallVantage).  This involved....absolutely nothing.  It simply worked.  I did set up QoS on the Linksys to ensure the phone would always have enough bandwidth.  This fixed some of my incompatibilities, but other issues began to crop up, like being unable to hear everyone in the game lobby sometimes.

I had been running the wifibox firmware on the Linksys, and decided to upgrade it to a more recent, official firmware. So I upgraded to the Live-certified firmware version.  This didn't help.  Then, after changing nothing, I began having problems joining people that I had never had trouble with before.  Last night, after putting the XBox in the DMZ and still having problems, I got fed up and systematically hunted down the issue.  It was definitely something with the router, which didn't make sense since it was XBox Live certified, and I was having trouble with other people who had the same router and they didn't have problems.  It just didn't add up.

So, I broke down and got a new router, and it fixed everything.  I played until 2:00 this morning without a single problem.  VoIP still works. Wireless works.  All the PCs work.  All without any additional port forwarding, DMZ settings, or other configuration besides the QoS.  It just works now.  And do you want to know the most bizarre part?  I got the exact same model of router.

My conclusion is that the wifi-box firmware screwed it up somehow.  It's the only variable left in the equation.  I'm curious if anyone else who is running 3rd party firmware has had problems like that. Perhaps I'll link to someone that I know does, and see if that generates any discussion from his previous post.

I've been toying with ECMAScript for the past week or so, mostly to update some dynamic user interface elements in a web application we have.  Every time I use it, I'm always impressed at the cool things I can do with it.  It also makes me wish that Microsoft had not messed with it as part of the CLR (JScript.NET).  Their newest version of JScript seems to merely make it into an alternate syntax, and not really take advantage of what it could do as a prototype-driven language with closure support.

I found this article that claims that it is the world's most misunderstood language.  I think I agree.  It's incredibly powerful, but very underutilized because of the confusion surrounding it.  Closures coupled with prototypes can yield some very slick code if you understand what you're doing (and I guess maybe that's the problem).  All you really need is a nice IDE to help you manage your objects and their prototype chain.

I'm also getting more into Python and Ruby, which are also very slick.  I'm just not sure how they should fit into a project.  In general, a project with fewer languages is more maintainable.  But imagine trying to build a skyscraper when your only tool is a hammer that works really well. (OK, maybe hammer is not the best analogy.  Maybe a blowtorch?  I don't know)

[UPDATE] I just had to include this link, where the author shows us how to do lot of different code reuse patterns, including multiple inheritance in Javascript.  My favorite quote:

This large set of code reuse patterns comes from a language which is considered smaller and simpler than Java

Whew, what an ordeal.  I finally got my Adobe Photoshop CS in the other day, and last night, I created a "Droplet" for batch converting all my raw files from my Canon Digital Rebel. Droplets are a really sweet feature that lets you create a little executable from recorded actions for batch operations.  So, I burned through about 2000 images, creating big jpgs suitable for uploading to my newly upgraded Flickr account.  It was getting pretty late, so I fired up an uploader an set it to upload my 1GB limit (roughly 900 pictures).

This morning, I checked my account, only to find that none of the metadata had been uploaded.  All my images appeared to have been taken on Feb 11, 2005...uh oh.  So, I set out to find out what happened.  Turns out, Photoshop saves the metadata in it's XMP format within the file. XMP is simply an rdf encoding of the data in an XML payload within the file.  It's actually pretty cool, but Flickr doesn't read this data yet.  So I set out to "fix" my pictures, since I can't upload anymore until next month and I have lots more to upload.

After looking at lots of libraries and Adobe's XMP SDK, I decided it would be easy enough to pull the data out myself.  So, I built a little app using my FlickrApi library I just created that would blast through my uploaded pictures, find the corresponding image on my local pc, pull the xmp data out of the file, and set the "date taken" on the Flickr site.  That way, I can at least organize them more easily.

It worked perfectly.  It blasted through about 900MB in less than a minute.  Look for the pictures as I tag them, annotate them, and change them from private to public.  I'll have to see if there's a way to have Photoshop preserve that data next time because I'd really like to have the rest of the metadata available.  I'll probably make my XMP parser available as well if anyone's interested.  As far as I know, there is not another managed implementation available.

Last night, I made alot of progress creating a .NET API for Flickr.  There doesn't seem to be alot of activity on the Flickr.NET project, and I don't like some of the decisions they made in the design.  They seem to have taken a direct wrapping approach to it rather than designing an API that fits in the .NET world.  It's amazing how spoiled you get when you're used to using good API's.  You start saying things like, "You mean I get an array back from this method?  Eww."  or "That's a silly name for that member."  Of course, it's probably appropriate to note that Flickr.NET seems to have been created with an emphasis on uploading.  I'm more interested in sifting through the metadata.

Anyway, I've got most of the functionality that I'm interested in implemented, and I'll be doing the rest a little at a time.  I've used the design guidelines fairly strictly, and I think I'm coming up with something that's really approachable for .NET users, and is still correlated enough to the Flickr API documentation that the parallels are easily discoverable.  You may notice some bizarreness here as I play with the different kinds of integration I have planned for my blog.

This is one of the coolest things I've seen in a long time.  Try my flickr name (marklio).  It lets you drill through my contacts and see their information.  Keeping clicking and you'll see all kinds of cool things.  It really helped me see Flickr as a web service rather than just a photo sharing site.  It gave me alot of ideas.  Things like using tag matching to automatically creating blog entry backgrounds that are relevant to the content.  I'm going to be playing with that.

Their API is really nice and well-documented.  Some people made a .NET wrapper for the API, but it's not up-to-date and I wasn't really impressed with the design.  I'll probably roll my own and just implement the things I need as I go.