This seems to be good SQL join resource.

Shunryu Suzuki, one of the pioneers of Zen Buddhism in the United States, said this in one of his lectures:

“In the beginners mind there are many possibilities, in the experts mind there are few”

This statement was part of a talk about something a bit closer to the pulse of life than the practice of software development (namely about the practice of finding the pulse of life) but it is nevertheless a resonant concept when applied to any cross section of our human experience, software development included.

Jeff Atwood and Scott Hanselman write about this eloquently. Karl Seguin recently published an e-book about ALT.NET, and in the introduction to the book relates his experience of being able to truly grow as a programmer only after shedding the illusions that he was an expert programmer.

I’m not interested in driving that point home any further. The argument was made very well by the authors above and rehashed endlessly in the comments of their respective blogs.

However, I thought it might be interesting to describe my personal experience of coming to that realization in my own career, and how it has altered (opened up) my view of our industry.

After many years of writing software for the same company with a tight focus on the task immediately at hand I decided to tune in to some of the chatter of the social web, rather than just digging through it on an as needed basis to solve one particular problem or another. I’ve always been an avid reader of technical books but technical books, especially ones about established (or even emerging) technologies, are very focused on their subject matter. They are, as advertised, about a specific technology, not about the art and practice of programming. I’ve also read a number of the classic books on the art and practice of programming, such as The Pragmatic Programmer, Code Complete, various patterns books, books on Agile, Domain Driven Design, and so on. Those are more enlightening than a technology specific book, and they help you improve your craft in a much more general way, but by and large they weren’t reflective enough to show me how little I knew.

Then I started to pay attention to this gigantic, amazing conversation that is always unfolding in the “blogosphere,” and, slowly, groggily, I started to wake up, to understand how expansive and fluid the world of creating software actually is.

The experience can be described as spending several years climbing a mountain you were born at the foot of, all the while assuming that the mountain was the whole world. As you near the tree line, you begin to catch glimpses of something larger flickering through the gaps between the trees, and then all at once you break through the tree line and behold the entire, sprawling mountain range extending out in all directions and any illusion you may have had about mastering your environment immediately evaporates and you find yourself in awe, and maybe a little bit frightened at the unexpected vastness.

In my case I was faced, on the one hand, with the fact that the sum total of the knowledge I had acquired thus far (struggled to acquire and felt such a sense achievement for acquiring) was barely a drop in a very big bucket. On the other hand, the universe I spent most of my waking hours in just grew without warning to unimaginable size, and now I got to explore it. There are certain times in life where being humbled is the only way to continue to grow, like a rose bush that must be cut back each season before it can bloom.

The first thing I noticed when I broke through the tree line, after recovering from the sheer size of it all, was that each of the other mountains, and the valleys between them, were crawling with fellow explorers. Sure, I had passed a few people on the way up my mountain, and a few people had passed me, but all of a sudden there were thousands upon thousands of other technologists scurrying about everywhere, sporting fanny packs, pocket protectors and laptops. Most were sweating and stomping steadily up the myriad well worn paths, but others were running far ahead, hacking through underbrush, madly scribbling notes as they went,  tossing them over their shoulders by the  fistful; others gave loud, grinning sermons as they strolled along while throngs of followers cheered or broke into raucous fistfights, and still others would stop regularly to draw detailed maps of where they had been, their theories about what lay ahead of them, and very often enough take the time to give lectures and make screen-casts of their experiences pioneering in this enormous, shared adventure of being a technologist.

Personally, I find it to be quite beautiful, and tremendously inspiring.

Of course, not everyone is helpful. There are plenty of angry, impatient and elitist explorers too, yelling at one another from peak to peak, certain their mountain is Mount Sinai, the others are cheap decoys, yadda yadda. This part of the culture is best served by the religion metaphor, though, and there is no avoiding the phenomenon: humans are extremely religious creatures and programmers, to their chagrin, are not immune. How can it be helped? Human life is an absolute mystery, religion is inevitable.

And of course these countless throngs have organized into cliques, tribes, nations, hold elaborate rituals, build massive idols and sprawling temples, go to war, cross-breed, and so on. The same microcosm of human society that is repeated in every other industry, or nation, or religious sect or body politic across the planet and throughout time. Obviously the universe uses design patterns, too.

Anyway, mostly I wanted to relate my experience in order to express my newfound appreciation of our profession, and particularly the staggering social generosity of the luminaries, pioneers, and everyday developers who devote so much precious time and apparently limitless energy to clearing the paths and building the roads on which the rest of us are able to make such amazing progress. 

I will close this post with another quote from Suzuki Roshi. This one I’ll leave up to you to interpret.

“My lecture for tonight will be very short, especially after having a good dinner of noodles, which were very long. Our transmission should be a very long, long one. And our transmission is a special noodle.”

Here’s another selection of nice extension methods.

I am constantly amazed (and humbled) by the generosity of time experts in this field of very, very busy people continue to demonstrate over and over again. I don’t know if other professions are the same way, but it really makes this one a pleasure to work in. Here is a free e-book by Karl Seguin of CodeBetter.com. I haven’t read it yet, but I very much look forward to it. From the TOC it looks like it will be a very useful resource.

You can get it here.

I’ve often been caught between two opposing forces - the desire to find some excuse to use Reflection.Emit to dynamically generate behavior, because it has a sort of magical quality about it, and on the other hand being completely intimidated by the strange and foreboding mysteries of IL. I finally found an excuse compelling enough to brave my fears and stick my toe in the water, and it wasn’t too bad.

The excuse to take the plunge was, as is often the case, laziness (or perhaps impatience). I had an interface with a blue million events on it, and I wanted create some unit tests that ensured all my little events were fired the right number of times, in the right order, with the right arguments, but I didn’t want to subscribe to all the events individually and set a bunch of flags, etc. etc. (BTW - I imagine this problem has been solved by superior minds - I didn’t spend much time looking)

The sample code, along with unit tests and a Windows Forms app demonstrating possible usage, can be found here.

Here is a typical usage:

[Test]
public void Recorder_Will_Record_Events()
{
    var source = new TestEventSource();
    using (var recorder = new EventRecorder(source))
    {
      source.FireSimpleEvent();
      source.FireSimpleEvent();
      source.FireEventWithCustomArgs(“MyTestText”);
      source.FireSimpleEvent();
      source.FireEventWithCustomArgs(“NewTestText”);

      Assert.That(recorder.EventHistory.Length, Is.EqualTo(5));

      Assert.That(((TestEventArgs) recorder.EventHistory[2].Args).LoveMe, Is.EqualTo(“MyTestText”));
      Assert.That(((TestEventArgs) recorder.EventHistory[4].Args).LoveMe, Is.EqualTo(“NewTestText”));

      Assert.That(recorder.GetInvocationsFor(“SimpleEvent”).Length, Is.EqualTo(3));
      Assert.That(recorder.GetInvocationsFor(“EventWithCustomargs”).Length, Is.EqualTo(2));
     }
}

What I wanted to do was “record” all the events that were fired on a particular object during the course of some transaction, and then test that all the events that fired were the ones I was expecting, and in the appropriate order. I figured I would achieve this by creating a class called EventRecorder that could take in its constructor any other non-null object. It would then loop through each event in the objects type, subscribe to that event, and when the event fired add the name of the event and the EventArgs to a collection I could examine later. Pretty simple.

I though this would be a simple matter of using some plain ol’ reflection, like this:

 foreach (EventInfo evt in eventSource.GetType().GetEvents())
 {
 	evt.AddEventHandler(eventSource,
            new EventHandler(
		(sender,args)=>RecordEvent(evt.Name,args)));
 }

But alas, AddEventHandler expects a delegate that exactly matches the signature of the event, meaning if I have an event defined with EventHandler<CancelEventArgs>, it isn’t going to let me pass a plain EventHandler delegate. So what to do? Dynamically generate an event handler with the appropriate signature, of course, and emit a simple call to my RecordMethod. Essentially, we’ll be generating the equivalent of this:

private void HandleEvent(object sender,EventArgs<CancelEventArgs> args)
{
    this.RecordEvent(args,”NameOfTheEvent“);
}

I found an MSDN article that looked like just the ticket. The technique presented uses the DynamicMethod class, which you can use to create an ad hoc method and fill it full of op-codes to tell it what to do. In this case, we wanted to instantiate a DynamicMethod that matches the signature of the event handler we want to subscribe to. Then we simply needed to emit the IL to call the RecordEvent method and pass in the appropriate arguments. In the code below, EventSource refers to the object whose events we’re subscribing to, and was passed in to the constructor of the EventRecorder class.

/// <summary>
/// This method crates a DynamicMethod matching the signature of
/// the delegate required by the passed in EventInfo. The method
/// will simply pass the arguments on to the RecordEvent method.
/// </summary>
/// <param name=”evt”></param>
protected void BindToEvent(EventInfo evt)
{
    Type handlerType = evt.EventHandlerType;

    if (handlerType == null)
        throw new ArgumentException(“No handler could be ” +
                              ” identified for ” + evt.Name);

    Type[] handlerParams = GetDelegateParameterTypes(handlerType);

    //Not technically required, but may be helpful if an exception is thrown
    String dynamicHandlerName =
        String.Format(“EventSource_{0}”, evt.Name);

    var handlerDef = new DynamicMethod(dynamicHandlerName,
                                       typeof(void),
                                       handlerParams,
                                       typeof (EventRecorder));

    MethodInfo recordEvent =
        typeof (EventRecorder).GetMethod(
            “RecordEvent”,
             BindingFlags.Instance | BindingFlags.NonPublic);

    ILGenerator ilgen = handlerDef.GetILGenerator();

    //load ‘this’ onto the stack
    ilgen.Emit(OpCodes.Ldarg_0);

    //load the second argument (EventArgs) onto the stack
    //We’re not using the sender parameterf of the delegate,
    //so we skip emitting OpCodes.Ldarg_1
    ilgen.Emit(OpCodes.Ldarg_2);

    //Load the name of the event onto the stack
    ilgen.Emit(OpCodes.Ldstr, evt.Name);

    //Call the RecordEvent method, passing the two loaded argumetns
    //EventArgs args and String name
    ilgen.Emit(OpCodes.Callvirt, recordEvent);

    //Return control
    ilgen.Emit(OpCodes.Ret);

    //Create a delegate out of this new handler, binding to the current
    //instance as the target
    Delegate handler = handlerDef.CreateDelegate(handlerType,this);

    try
    {
        //Register the new delegeate with the event on the event source
        evt.AddEventHandler(EventSource, handler);
    }
    catch (Exception ex)
    {
        //Some COM objects don’t like this method. In production, you wouldn’t
        //eat this error like this, but for my example, I wanted to show events
        //on the WebBrowser control, and certain events of WebBrowser
        //don’t appreciate this dynamic approach. So…
    }

    //record this handler so we can get to it later
    //when we need to remove our registration from
    //the event source during dispose
    _handlers.Add(evt.Name, new EventRef(handler,evt));
}

/// <summary>
/// Extract the signature of the event’s delegate
/// </summary>
/// <param name=”d”></param>
/// <returns></returns>
private static Type[] GetDelegateParameterTypes(Type d)
{
    if (d.BaseType != typeof (MulticastDelegate))
        throw new ApplicationException(
            d.Name + ” is not a delegate.”);

    MethodInfo invoke = d.GetMethod(“Invoke”);

    ParameterInfo[] parameters = invoke.GetParameters();

    //Allocate an extra slot, see below for why
    var typeParameters = new Type[parameters.Length+1];

    for (int i = 0; i < parameters.Length; i++)
    {
        typeParameters[i+1] = parameters[i].ParameterType;
    }

    //This is an extremely important step - we need to inject the type
    //of EventRecorder because we will be using EventRecorder as the Target
    //of the dynamic delegate that we create using the DynamicMethod. When
    //you go to convert your emitted MSIL to an actual delegate using DynamicMethod.CreateDelegate,
    //the first argument of the signature MUST match the target object you are binding
    //the delegate to.
    typeParameters[0] = typeof (EventRecorder);

    return typeParameters;
}

/// <summary>
/// The internal method that gets called by the dynamically
/// generated delegates we are using to bind to the EventSource’s
/// events. This method will get called once for every event that fires
/// on the EventSource.
/// </summary>
/// <param name=”args”></param>
/// <param name=”eventName”></param>
protected virtual void RecordEvent(EventArgs args, string eventName)
{
    EventInfo evt = _handlers[eventName].Event;

    if (evt != null)
    {
        var history = new EventInvocation { Args = args, Event = evt };
        _eventHistory.Add(history);
        OnEventRecorded(history);
    }
}

IL is pretty close to the machine, relative to C#, so it’s entirely stack based. You load arguments onto the call stack, then perform an operation, such as calling a method. If you call a method, and the method has a signature with arguments, the arguments for the method signature are popped off the stack and consumed.

In this example we’re generating a dynamic method with a signature of two visible arguments - (object,MyEventArgs) - but we’re calling a method (RecordEvent) with a slightly different signature (EventArgs,string). We can discard the “object” argument of the dynamic method, which we do simply by not loading it onto the stack, which is why we don’t emit the opcode Ldarg_1, but we need to load the EventArgs and a string.

Looking at my source code it doesn’t quite add up though, because you can see I’m loading argument 0 (Ldarg_0) which looks like it should be “object” based on the dynamic method signature and argument 2 (Ldarg_2) which doesn’t even seem like it should exist, as the signature is (object,EventArgs) - only two arguments. Well, it turns out that all instance methods require a 0 position argument that is “this” - this argument is inserted by the compiler so you never see it, but when you say “this.DoSomething()” in your code, in IL method signature that gets created is DoSomething(MyType this). That means our call to “RecordEvent(EventArgs,string)” really looks like this: “RecordEvent(EventRecorder,EventArgs,string)”, which means we need to load 3 variables onto the stack, which we do:

Ldarg_0 = this

//Ldarg_1 = object - not needed

Ldarg_2 = EventArgs

Ldsstr = EventInfo.Name

NOT calling Ldarg_1 discards the object part of the dynamic method signature, which is OK, because that just represents EventSource anyway, and we have a reference to that already.

One important thing to note, though, is that when calling CreateDelegate on DynamicMethod it is critical to pass in “this” as the target - otherwise when the method you generated tries to access member variables, you’ll get illegal memory access errors. Also subtle, if you are creating a dynamic method that you intend to bind to an instance of an object, then the DynamicMethod needs to be created using the secret, full signature of MyMethod(EventRecorder,object,EventArgs) NOT just MyMethod(object,EventArgs) as you would expect. You can see this being taken care of in the GetDelegateParameterTypes method. This piece of info was not in the MSDN sample, and was a just a joy to figure out

By the way - a great way to figure out just what IL you need is to write the method you are hoping to generate, compile it, then view the IL in ILDASM.exe. Then you can either blindly translate the IL into Opcode.xxx calls, or you try to understand why it is doing what it is doing and modify according to your needs, but I probably never would have figured out the Ldarg_0 / Ldarg_2 mystery on my own - ILDASM saved my butt.

I need to find a better way of debugging this stuff, but the fact of the matters is that once you see what is going on, IL isn’t really any more complicated than anything else, it is just far more verbose and cryptic when you get it wrong

The rest of EventRecorder is nothing special. The sample includes unit tests and a simple windows forms example.

Here are some resources:

DynamicMethod.CreateDelegate Method

How to hook up a delegate using reflection

Reflection.Emit tidbits

Introduction to creating types with Reflection.Emit, DymamicMethod

A “getting-started” tutorial on how to get up and running with Simple Service Bus is available here. The guide explains various methods of configuring endpoints, how to define, send and publish messages, how to create and register message handlers, as well as how to setup the endpoint health monitoring system that is part of the framework. Much of what is presented will also apply to nServiceBus, since until you start getting into extensibility SSB hasn’t made many changes, but the syntax and configuration does differ in a number of places, so adjustments would need to be made.

This site gives a good definition of State Verification and Behavior Verification as it concerns our efforts to establish unit tests for all new functionality.

State Verification
Behavior Verification

Heres the one line definition for each if you want the 50,000 foot view.

State Verification: We inspect the state of the system under test (SUT) after it has been exercised and compare it to the expected state.

Behavior Verification: We capture the indirect outputs of the SUT as they occur and compare them to the expected behavior.

Its a subtle but important distinction in that with state verification you may run a series of method call or processes and verify the expected state of the system after all the call have been made. With behavior verification you are interested in verifying that individual method calls are made to components within the system under test including dependent components utilized by the system under test.

I have posted a comprehensive overview of Simple Service Bus, including an architectural description, in the form of a PDF document available here. A few images from this document are included below.

[UPDATE] A pdf document providing a comprehensive overview of the project is now available here.[/Update]

I have posted a new project on CodePlex, Simple Service Bus. Simple Service Bus is a derivative of Udi Dahan’s nServiceBus, which I’ve mentioned before. I’ll be posting some more extended samples and documentation for using the new library, but first I wanted to document my motivations. nServiceBus is a very solid, very useful, very easy to use piece of software for building asynchronous, messaging based service endpoints, and it’s creator is a well respected SOA guru, at least in .NET circles. I, of course, have no such pedigree, which is why I chose to start with nServiceBus as a foundation rather than starting from scratch.

So what is wrong with nServiceBus that required us to fork it? Well, nothing. My take on nServiceBus and the direction Udi Dahan seems to take the project is that it is a library designed to provide just enough rope to get the job done, but not enough rope for us developers or architects to hang ourselves from a scalability and reliability perspective as we wade into the asynchronous, messaging based waters for the first time. This is a very useful trait for a library to have. However, such an approach comes with some tradeoffs, namely when it comes to extensibility and customization, and I’m a lego brick kind of guy, maybe stemming from control issues, who knows

In service, I believe, of the protective and prescriptive nature of the project, nServiceBus has many of its most interesting parts “soldered to the motherboard” - their implementation is blended in with the default implementation of the framework itself, which means that if you want to customize how certain things work, you have to diverge from the codebase. Now, to be fair, this is only true to a certain extent. nServiceBus has several major pieces that can be swapped out, such as IBus and ITransport, but within the responsibilities of the default implementations of IBus or ITransport are a number of smaller components that can’t easily be customized.

It hardly seems necessary to go so far as to fork the project when custom IBus and ITransport implementations could have accomplished 95% of the flexibility I was going for. And, in reality, it probably wasn’t necessary - but once the thing was already on the operating table, I couldn’t resist making some preference based changes to the IBus API, and I wanted to add custom headers to the message envelope (which crosses between IBus and ITransport) and at that point there really wasn’t any turning back.

Another question would be, well, if nServiceBus didn’t have the extensibility you were after, why not choose Mass Transit? It is extensible. And, in fact, it is - but it is still fairly young, and we’ll be going into production very soon. nServiceBus has been through the fires a few times, and I figured I could add the hooks I needed without altering the fundamental design of how nServiceBus operates, which I quite like and I’ll sleep better at night because of it. My belief, though, that if it was six or eight months from now, Mass Transit would have been the direction I’d have taken. Also, I already had a project, NsbExtensions, that implemented endpoint health & performance monitoring features on top of nServiceBus, and I wanted to be able to reuse all that code.

Here are the design goals of Simple Service Bus - most but not all of which are realized but not thoroughly tested in the version currently posted to CodePlex:

* Small DLL footprint - nServiceBus (for valid architectural reasons) is split up into eleven thousand, two hundred and nine different DLL’s. That may be an exaggeration, but many of the dll’s have only one or two code files in them, with only a few lines of code each. Simple Service Bus comes in three DLL’s for a normal endpoint, and add’s a fourth to implement an Endpoint Health Monitoring Server. Also, required third party dependencies are kept to a minimum (0).

* Custom headers in the Message Envelope - I wanted to be able to attach bits of information that are necessary for the messaging system to function properly, like authentication tokens or endpoint ID’s, without having to include those pieces of irrelevant data on the otherwise domain-oriented messages themselves. This results in a somewhat cleaner set of message contracts, but it also allows you to do some content based routing or authentication checks on a message before the payload ever has to waste the CPU’s time on deserialization. If a message is going to be blocked due to lack of authorization, or re-directed to a different endpoint, or simply written to disk as part of an audit process, there is no need to send the payload through the de-serialization / messaging handling process.

* Send and Receive Pipeline Pattern for Message Processing - instead of having a hard-coded path from Transport through final Handling, Simple Service Bus provides a configurable Pipeline approach, where each interesting step in the messaging processing process is implemented as a pipeline component. This allows programmers to add, for example, a content based message router ahead of the payload deserialization component, or to add a statistics gathering component at various places within the pipeline. Additionally, a rich set of events are fired while a pipeline is processing a message, allowing fine grained performance monitoring or logging if required.

* Replaceable Subsystems - each major concern of the bus has been factored into a simple subsystem as defined by an interface. The default implementation of any subsystem can easily be replaced. For example there is a Message Dispatcher subsystem, a Subscription Manager subsystem, a Message Serialization subsystem, a Message Type to Message Handler Resolution subsystem…etc. In this way, if you want to implement a Subscription Manager that can evaluate publisher-side expressions to determine whether or not to send a message to a particular subscriber using a custom DSL or any other expression evaluator, you can easily swap out the existing subscription manager with your enhanced one. And you can do it for just one endpoint, if only one endpoint needs that special functionality.

* IoC supported, but not required. The building of objects is handled by subsystem as well, and the default implementation can be given a reference to an IoC container to build all objects and services, or it can use simple reflection if no IoC is provided. To accommodate this, a Service Locator was used rather than a pure dependency injection approach, but the services registered with the Service Locator can optionally be built with an IoC. Also, of course, all Message Handlers and Pipeline Components will be constructed on demand using an IoC if provided, or simple reflection otherwise.

* Polymorphic Message Handlers - the default Message Handler Resolution Service allows message handlers to be defined for interfaces or base classes, allowing the creation of message handlers for families or categories of messages rather than just individual messages. This also applies to subscriptions, meaning that an endpoint can, for instance, subscribe to IEventMessage, and receive all messages that implement that interface, instead of being required to subscribe to each concrete message type.

* Endpoint Health & Performance Monitoring built into the core library - all endpoints can be configured to provide status reports to a monitor service with a few lines in the configuration file.

* Simplified Configuration - this is debatable, because Udi Dahan has gone to significant lengths to partition “programmer configuration” from “administrator configuration” - the former being accomplished in code and the latter being accomplished in a configuration file. This is probably a wise, experienced based approach, but I myself like the option to configure everything in a single, simple block of XML, if XML is being used. So that is how Simple Service Bus does configuration, in a more traditional manner.

* Abstract base classes provided for easily creating custom Transports, Pipeline Components, Performance Probes, etc., abstracting much of the ugly details and allowing the system to be extended by overriding a few simple, targeted methods.

* Fully Unit Tested - this isn’t finished yet (I’m still trying to get the hang of TDD, so the tests weren’t written first in this case) but the idea is to have unit tests for each piece of the system, allowing modifications or customizations to be done in a safe, controlled manner.

*object based API. I removed the IMessage marker interface for Simple Service Bus because (perhaps mostly for aesthetic reasons) I wanted our Message Definition dll’s to be as clean and reference free as possible. So now Send, Publish, etc. operate on System.Object rather than IMessage.

And that’s about it. I fell in love with Gregor Hohpe’s “Enterprise Integration Patterns” and wanted to apply those patterns to my own evolving messaging system, so this effort is sort of centered around enabling nServiceBus to accommodate that desire in a clean, plug and play way. We’ll see how it goes.

Similar to BizTalk services, minus the workflow/identity piece but plus a fairly robust security model and web based management dashboard, Linxter(http://www.linxter.com) provides a secure, queued transport for .NET applications. Essentially the service acts as a hosted messaging queue with an extremely simple (in a good way) API and an interesting ability to optionally send file attachments with your messages, which right off the bat gives it an edge over MSMQ over HTTP in my mind. Applications using this API will request registration with a central account on startup and security settings determine if the connection is automatically granted or if an administrator has to manually grant access to your bus to the endpoint. Once access is granted, endpoints (via the Linxter SDK) can send messages and receive messages in either an “on-demand” pull model or an event based push model. The service also supports broadcasting messages to multiple endpoints.

This isn’t something that could act as an ESB, but it could certainly act as a transport for an ESB. The service doesn’t have any pricing announced and the user agreement forbids production or commercial use, but it is something to watch if such a service would be useful to you.

In our case it would be perfect for connecting client systems to our hosted solutions without having to deal with ensuring MSMQ w/HTTP support is properly installed on each endpoint machine, and ensuring the proper queues are set up, etc. Much easier deployment and maintenance for remote nodes not in our firewall. If all your endpoints are inside your firewall, a hosted queuing service may be of little value, but if you have endpoints in the wooly wild, a service like this could be invaluable, taking care of endpoint authorization, identification, and all the rest.

Here is a link to the current stage of development, which appears to be nearing the end of Beta 2