Richard Lander, one of our PMs, just started a series of posts on deployment.  Specifically, he’s taking the approach of thinking about your v2 strategy while you are still working on v1 of your product.  If you’re already on v3 of your product, don’t worry.  There will still be some useful, interesting information for you. The first post just lays the groundwork for the discussion, so stay tuned for the rest.

I didn't get up an anniversary post yesterday because I was busy DOING stuff for my anniversary this year.  Unbeknownst to Becky, I arranged to take the day off. I sneakily got a late-morning massage appointment for her, and lined up a babysitter (thanks, Molly) so we could go see a movie in the afternoon.  Everything went off without a hitch.

We went to see the latest Indiana Jones movie, which we enjoyed very much despite the somewhat less than explanatory ending. We hadn't been to the theater to see a movie since Pirates II.  But, we both agreed that we aren't missing much except time alone and seeing the latest movies.

Some of my posts that I get the most recurring email/inquiries on are my various posts about the STDF (Standard Test Datalog Format) parser that I originally created as an experiment back in 2005.  After some help from a former collegue, I am pleased to announce that this is finally available on CodePlex as LinqToStdf!

It is a managed library for processing STDF files, and gives you a model to explore the data via Linq queries.  This means, you can leverage the wide variety of managed languages (C#, VB, F#, JScript, IronPython, IronRuby, Managed C++, Boo, etc.) to process the data in STDF files.  It also works in Silverlight!  It has built-in support for the V4 spec, but it's highly extensible and should be able to parse any version of the file format as well as custom records.  It can be configured to be highly strict and throw on format errors, or be robust in the face of issues that normal STDF processors would choke on to the point of being able to detect and repair corruption on the fly.

If that interests you, I'd love for you to drop by and take a deeper look at it and get involved in its ongoing development.  I've already got at least one person interested enough to contribute and ensure its success as a community project.  There is currently a "beta" release available, and hopefully we'll whip it into shape enough to call it v1.0 soon.

My hope is that this can be an adoption driver for .NET in the semiconductor industry and that through this project I can be an ambassador for the CLR in that area.

I've been playing alot with beta 2 of Silverlight 2, and I've been totally amazed at the scenarios it enables.  Working hard down inside the CLR engine, we're sometimes insulated from some of the innovation going on higher in the stack and it blows us away when we do get a chance to see it.

One of the very cool things in Silverlight is "Deep Zoom", which came from the SeaDragon project from Microsoft Research.  I decided to try it out myself on a very large panorama that I made a long time ago in New Orleans.  Unfortunately, the current toolset seems to trip over the large file size (20516x15291).  I'm trying to find out the real story behind the limitation.  All I know right now is that smaller files work.

So, I decided to try to slice it up into smaller, manageable chunks and just butt them against one another to simulate one large file.  The problem was that I couldn't find a tool to do this that didn't also trip up over the size of the file.  So, I wrote my own.  There are likely better ways to do this, this was just a quick and dirty attempt to make something that didn't totally crawl to a halt due to page faulting (or outright throw OutOfMemoryException).

The code below is what I came up with.  I use System.Drawing.Bitmap, lock the portions of the image I need, and do the data copying myself.  I ended up with this solution because GDI+ (DrawImage) seems to like to make alot of buffers (big ones in this case), and I couldn't fall back on good ol' bitblt because I couldn't get the right kind of data structures without more copying of the data.  This runs plenty fast and only takes up marginally more memory than it takes to represent the original and the size of one of the destination tiles.

Unfortunately, The MultiScaleImage does some weirdness with image that are butted up against each other, and you get as much as a whole pixel of "space" between them, depending on your zoom level. I'm still looking into other possible workarounds. So, it's wasn't ultimately useful, but I thought the code was interesting enough, and probably has academic usefulness.  So, enough talk, here's the code:

   1:  using System;

   2:  using System.Collections.Generic;

   3:  using System.Drawing;

   4:  using System.Drawing.Imaging;

   5:  using System.IO;

   6:  using System.Linq;

   7:  using System.Runtime.InteropServices;

   8:   

   9:  namespace ImageSlicer {

  10:      static class Extensions {

  11:          public static IEnumerable<int> Times(this int number) {

  12:              for (var i = 0; i < number; i++) yield return i;

  13:          }

  14:      }

  15:   

  16:      class Program {

  17:          [DllImport("msvcrt.dll", SetLastError = false)]

  18:          static unsafe extern byte* memcpy(byte* dest, byte* src, int count);

  19:   

  20:          static void Main(string[] args) {

  21:              var sourcePath = args.FirstOrDefault();

  22:              if (String.IsNullOrEmpty(sourcePath)) {

  23:                  Console.WriteLine("No source image path provided");

  24:                  ShowUsage();

  25:                  return;

  26:              }

  27:              if (!File.Exists(sourcePath)) {

  28:                  Console.WriteLine("Source image path doesn't exist");

  29:                  return;

  30:              }

  31:              var gridSizeStr = args.Skip(1).FirstOrDefault();

  32:              int gridSize = 4;

  33:              if (gridSizeStr != null && !int.TryParse(gridSizeStr, out gridSize)) {

  34:                  Console.WriteLine("Could not convert {0} to a valid grid size",

  35:                      gridSizeStr);

  36:                  return;

  37:              }

  38:              if (gridSize < 2) {

  39:                  Console.WriteLine("The grid size must be greater than 1.");

  40:                  return;

  41:              }

  42:              try {

  43:                  Console.WriteLine("Slicing {0} into a {1}x{1} grid.",

  44:                      Path.GetFileName(sourcePath), gridSize);

  45:                  Console.WriteLine("Loading...");

  46:                  using (var sourceBitmap = new Bitmap(sourcePath)) {

  47:                      Console.WriteLine("Source Image: {0}x{1}",

  48:                          sourceBitmap.Width, sourceBitmap.Height);

  49:                      var sliceWidth = sourceBitmap.Width / gridSize;

  50:                      var sliceHeight = sourceBitmap.Height / gridSize;

  51:                      Console.WriteLine("Each slice: {0}x{1}", sliceWidth, sliceHeight);

  52:                      int tile = 0;

  53:                      foreach (var row in gridSize.Times()) {

  54:                          foreach (var column in gridSize.Times()) {

  55:                              Console.WriteLine("Creating {0} of {1} ({2},{3})",

  56:                                  ++tile, gridSize * gridSize, column, row);

  57:                              using (var destBitmap = new Bitmap(sliceWidth, sliceHeight)) {

  58:                                  var sourceData = sourceBitmap.LockBits(

  59:                                      new Rectangle(column * sliceWidth,

  60:                                          row * sliceHeight, sliceWidth,

  61:                                          sliceHeight),

  62:                                      ImageLockMode.ReadOnly, PixelFormat.Format24bppRgb);

  63:                                  var destData = destBitmap.LockBits(

  64:                                      new Rectangle(0, 0, sliceWidth, sliceHeight),

  65:                                      ImageLockMode.WriteOnly, sourceData.PixelFormat);

  66:                                  unsafe {

  67:                                      byte* pSrc = (byte*)sourceData.Scan0.ToPointer();

  68:                                      byte* pDest = (byte*)destData.Scan0.ToPointer();

  69:                                      foreach (var line in sliceHeight.Times()) {

  70:                                          memcpy(pDest, pSrc, sliceWidth * 3);

  71:                                          pSrc += sourceData.Stride;

  72:                                          pDest += destData.Stride;

  73:                                      }

  74:                                  }

  75:                                  sourceBitmap.UnlockBits(sourceData);

  76:                                  destBitmap.UnlockBits(destData);

  77:                                  destBitmap.Save(String.Format("{0}_{1}_{2}.png",

  78:                                      Path.GetFileNameWithoutExtension(sourcePath),

  79:                                      column, row), ImageFormat.Png);

  80:                              }

  81:                          }

  82:                      }

  83:                  }

  84:              }

  85:              catch (Exception ex) {

  86:                  Console.WriteLine("Error processing image.");

  87:                  Console.WriteLine(ex.ToString());

  88:              }

  89:          }

  90:   

  91:          private static void ShowUsage() {

  92:              Console.WriteLine("Usage:");

  93:              Console.WriteLine("ImageSlicer.exe sourceImage [gridSize]");

  94:          }

  95:      }

  96:  }

Why not come work on the Common Language Runtime team at Microsoft?  My team is a team of SDETs (Software Development Engineer in Test) that works in the really low levels of the CLR.  We cover areas like assembly loading/fusion, AppDomains, the shim (mscoree), and interop.

As an SDET, you are in charge of all aspects of quality for the areas you own. This means you get to be involved in the design process and be literally the first person developing code using new features coming out of the team.  I've found my last year and a half here to be extremely satisfying.

This is a pretty exciting time to join the team as we're working on some long-lead items for the next version of the CLR and we're also busy shipping Silverlight 2 and .NET Framework 3.5 SP1.

You'll need strong coding, problem-solving, and communication skills.  Some background on the CLR and managed code is a plus, but not required.  If you think you've got what it takes to join the team, check out the job details and submit your resume.  I'm happy to answer questions about the team and the jobs within reason, so feel free to ping me at marklio at [you-know-where].com.

There are several openings in other areas of the team as well.  Feel free to search the other CLR jobs and find the one that's right for you.

I've been doing some app-building with Silverlight lately and exploring the limitations of the platform in comparison to the full desktop CLR and what that means for the Silverlight "ecosystem".  Those limitations can be summed up with 3 items:

1. Strict sandbox security model
2. Reduced managed framework surface area
3. No binary compatibility with libraries targeting the desktop CLR (eliminates the number of 3rd party components you can leverage).

I think #3 will begin to become a non-issue as more component providers provide builds for Silverlight. #2 can be broken down into 2 areas:

• Full technology areas that are unavailable on Silverlight (LinqToSql, WinForms, etc)
• Reduced/pruned APIs

The second is where my interest lies, and intersects with #1 (the new security model).  We've done extensive threat modeling against our APIs as well as automated tooling that has either removed (or made internal) certain APIs, or marked them as SecurityCritical, meaning they cannot be called directly from "user" code.

In addition, the security model requires safe, verifiable user code.  Normally when developing on the desktop CLR, unless you are specifically targeting a low trust environment, you can do whatever you like.

So, for this exercise, I pulled out my trusty STDF parser (blogged here and here).  I've used this project as a test vehicle for both v2.0 and Orcas, and it's served well as a project that leverages a large cross-section of features in the Framework, from high-level stuff like Linq down to expression tree inspection and further down to LCG/RefEmit.

In short, I was able to get the parser working without too much trouble. I felt like creating a source bed that intended to target both Silverlight and the Desktop should be an attainable goal.  You just have to switch your mindset from binary compatibility to source compatibility.

I combated the reduced surface area with extension methods, which worked quite well to centralize the "overload shims" that needed to be "Silverlight-only", as well as for a general refactoring tool.  My goal is to have all the desktop vs. Silverlight differences centralized into files that are either included or excluded from the build depending on which platform I'm targeting.  I wish you could create extension properties.  That would let me close all the surface area discrepancies that aren't caused by missing/irrelevant technology areas.

I was pleased that 99% of my LCG codegen stuff "just worked".  I make heavy use of Reflection.Emit via DynamicMethod to generate my record parsers based on attributes on the record classes.  The 2 problems I ran into were:

• Visibility restrictions - The new security model won't let my dynamic methods see internals.  I had used this ability to keep the API clean.  I'm still figuring out the best approach, but it was simple enough just to expose those methods.
• Verifiability - I had a few places where I was generating unverifiable code.  Some of these were my own codegen bugs, but others were just bad assumptions on my part.

This brings me to constrained callvirt, an interesting little IL tidbit I discovered in the porting process.  Calling conventions are subtly different between reference types and value types, and it also depends on whether the given method is actually overridden in the value type (which can lead to confusing breaks when that state changes).  In the v1.x days, you always knew the type you were dealing with, so it didn't matter that much and you could generally always create the right sequence of IL to make a call.  V2.0 introduced generics, which meant that you couldn't emit unified IL for both the possibility of reference types or value types.  This meant there needed to be a way to unify the IL for the 2 cases.  This is where the constrained prefix came in.  It allows you to write unified IL that works regardless of whether you're working with a value type or a reference type (think generics constrained by an interface, or calling a method defined on System.Object like ToString()).

Anyway, I was able to fix my unverifiable code by utilizing the constrained prefix.  It also simplified my codegen logic significantly in a number of places where I had different paths based on whether I was working with a value type or not.

All in all, I was pleased with the results.  I'll be posting a sample Silverlight app using the library when I get some UI stuff figured out.

I've been horrible with blogging lately.  But, I would be remiss if I  neglected to mention Peter's birthday today.  Happy Birthday, Peter.

Hopefully the thing that's been clogging my blogging pipes will be gone soon.

Some of my posts are really reactions to search queries that have previously landed on my blog.  If they did a search that got to my blog, but I know they didn't find what they were looking for, chances are they (or someone else) will do the same again.  And, if I HAVE the information they are looking for, it makes sense to just add the information, even if it's what I would consider well-known or common sense information. (common sense for software developers, that is)

One general search query I see again and again is something like "What is Action<T> for?" or "What is Func<T>?"

These are framework-provided, generic delegate types.  If you'll recall, delegates can be thought of as type-safe function pointers.  A delegate type really just captures a signature as "callable" object.  Leveraging generics to define delegate types that can capture common signatures is goodness, since they are very flexible and can be used by anyone.  This also aids in interop between different components, since a general signature is far more interopable than custom delegate types.

In v2.0, several functional-looking APIs were added that took delegates as arguments (think List<T>), so instead of adding a special delegate type for each API, several "generic" delegates were added to capture the "essence" of a signature such as Action<T> which takes T and does some action (returning void), Predicate<T> which takes T and returns bool (presumably doing some test against T), Comparer<T> which compares 2 T's, etc.

In v3.5, even more generalized functional patterns were introduced (used heavily in Linq).  And we added a bunch more Action<> "overloads" for functions returning void, and added Func<> "overloads" for functions with a return value.  (I use overload loosely since these are classes and not methods) These patterns dropped the semantic "meaning" of the delegate, and just went straight to the idea of capturing a signature.

These framework-provided delegates are useful for using in your own code rather than creating your own.  Whether you leverage the Linq-centric, super-generic Action/Func pattern, or opt to consume the more meaningful v2.0 Predicate, Comparer, etc. is up to you.