Fine, SOA may not be a City. What about a religion?

Rob Eamon has offered some compelling, well cited arguments (corrections?) in regards to a previous post of mine describing the image of SOA as a city/society that I selected from among various alternatives to use as a mental model. See the comments of that post for a full record of his take on things.

I can’t help but wonder, though, assuming his understanding is the original understanding, that the original “scope independent” definition he has laid out for the term SOA is mutating or fracturing into a different meaning, or more probably collection of meanings, as the concept is becoming more and more mainstream. Perhaps the mutation, if is real, is fueled by relative “laymen” from such out of the way places as the SMB (such as myself) as we attempt to understand SOA and become practitioners, which perhaps wasn’t common until more recently.

Or perhaps I say that entirely defensively, not wanting to believe I could have missed the mark by such a large margin.

In any case, I certainly have no other option than to concede that I was incorrect in my naive belief that I had found for myself a way of thinking about SOA that, at least in a purely conceptual fashion, could exist at an abstract level above all the tumultuous debate.

Perhaps believing in an image of SOA that encompasses all possible schools is as foolish as believing in an image of god that includes all other religions. I never get used to how much technology feels like religion to me at times.

Maybe things would be more straightforward for everyone if there were named schools of thought one could subscribe to, and then it wouldn’t be so tempting to say “This, well THIS is SOA, and that, well that is something that may resemble SOA(maybe), but in fact it is nothing of the sort.” Rather I could just say “I am a Soascopist” and you could say “I am a Soastylist” and, like religion, an argument about which is right would inevitably continue to rage. But, when the uninitiated like myself walk into a room of people shouting different definitions over each others heads, we would clearly know we needed to pick a school, learn the dogma and start shouting back, or we could state ourselves as agnostics, but we would be less inclined to try and synthesize the “truth” from all of the various contradictory opinions, which turns out to be a rather futile effort.

Then, when the holy war breaks out (and perhaps it already has), the SoaScopists, the SoaStylists and all other various sects of Soaists will take up arms together beneath a common standard and ally themselves against the RPCists. Gifted with superior agility, close strategic and financial alignment with the emperor, and extraordinarily loose coupling, the Soasists will relegate RPC to the level of an outcast technology, kept on life support on the outskirts of the kingdom via a flimsy service adapter, until some data center technician coldly pulls the plug, changes a value in a routing table, and the lights go out for good.

Of course, however it shakes out, we’ll still always prostate ourselves before the dollar, one way or another.

Amen.

(BTW, the holy war is just a joke, not my new personal metaphor for the relationship between SOA and RPC)

Create Bullet Graphs with Google Charts in 7 Easy Steps

I haven’t paid much attention to Google Charts so far, but I have enjoyed Stephen Few’s “Bullet Graphs.” A bullet graph is an extremely condensed sort of bar chart that shows the current value of a KPI, and various ranges such as unacceptable, acceptable and target, and an optional indicator line that can mean several things. Anyway, here is an example of a bullet chart (from wikipedia):

 

Bullet graph labeled.png

 

This blog post referenced below was referenced in Stephen Few’s blog this morning and describes the several easy steps required to create a bullet graph using the Google API. Being able to easily throw these things up, ad hoc, in any web page is pretty phenomenal, if you ask me. The image below is actually a Google Graph:

 

Here is the link to the guide:

DEALER DIAGNOSTICS » Blog Archive » Create Bullet Graphs with Google Charts in 7 Easy Steps

ESB’s for the Microsoft (.NET) Platform

I did a pretty exhaustive search for an ESB recently for use on a project that we’re getting underway. FYI, my selection criteria was coming from the point of view of an SMB, so we probably emphasized simplicity and ease of use over scalability and robust governance tools, for instance. Also, cost was a big factor, so the big infrastructure vendors weren’t even on the radar, even though I’m sure they could be used quite easily in a .NET environment.

Anyway, there weren’t all that many options. I guess I wasn’t surprised, but I was a little disappointed. The Java community has half a dozen or more full blown open source ESB projects, many with professional support organizations where you can buy support and consulting if you need it.

In .NET land there wasn’t anything like that. Here is what I found:

BizTalk Server + BizTalk ESB Guidance = ESB ?

Initially I assumed we’d be going down this path, so I read a really great book by Darren Jefford on the subject. BizTalk is a pretty deep piece of software, but like a lot of Microsoft frameworks it aims to be all things to all people, and we wouldn’t need 75% of what it provides and it didn’t offer the simplicity of development, maintenance and deployment I was looking for. The business rules chapter of the book opened up with a warning that the BRE in BizTalk is complicated beyond measure, yet you should learn it because it is powerful. To make it an interpretation of an ESB you apply the ESB guidance put out by Microsoft that I have heard eventually was the jumping off point for Neuron (possibly I made this up in my head), but the guidance exists and seemed to have quite a few moving parts. The installation instructions and prerequisites weren’t a walk in the park, and the whole itinerary based approach isn’t quite what we needed. We were looking for dead simple Send, Receive, Publish, Subscribe semantics in a code based API. BizTalk is a lot of things, but not those things, so the search continued.

Neudesic’s Neuron ESB

Next up we evaluated Neuron. This is really a pretty fantastic product that combines the easy API I was looking for with all the powerful translation, BAM and orchestration of BizTalk (but you need to license both products, I don’t believe Neuron offers these features separately) with a ton of flexibility and some great centralized endpoint configuration mechanisms. From what I gather they pretty much pulled BizTalk apart and exposed the functionality of its pieces (orchestrations, BAM, transformations, adapters, etc) directly via their messaging API and bypassed the MessageBox entirely, which is a pretty slick trick.  Without BizTalk, I think Neuron still offers an alternative to orchestrations using integration points with WF and like BizTalk has a pretty good set of legacy systems adapters. Still, even though Neuron is orders of magnitude less expensive than the big dollar enterprise ESB’s, it was still more than we wanted to spend, mostly because our messaging needs were so simple.

Mass Transit

This is a pretty new open source project, currently at version .1. The code is really pretty interesting, there seem to be plenty of extensibility points and the test coverage is very good. Lot’s of moving parts in this one too, but it is a rapidly evolving project. I’ve been keeping an eye on the code as it moves from .1 to .2 and the API is changing rapidly. I think this is definitely worth keeping an eye on, but as of this exact moment in time the API is a bit too volatile, as you would expect in a .1 version. Once it cooks for a while and sees some use I think it will be a nice alternative to the others on this page. Also, they have an NMS transport for integrating with ActiveMQ, although that may not be ready yet.

nServiceBus

This is a relatively mature open source project currently at version 1.8 (RC2 or something) and from accounts has been used in some pretty demanding, large scale production applications. It is a one man show, from what I can tell, run by Udi Dahan, who has given himself the moniker of “The Software Simplist” and I think that shows in his service bus. nServiceBus has the least number of moving parts and it is very easy to understand the entire code base without furrowing your brow. It is trivial to set up endpoints and get them sending, receiving, publishing and subscribing. This is the library we are tentatively using in our project. I wish, at times, it had a few more extensibility points, or were a little more modular, as it comes apart in roughly three or four big pieces. You can replace any of the big pieces, but there isn’t a whole lot of customization or modularization within the pieces. My belief is that Udi Dahan vigorously defends the simplicity of his project and avoids adding what he considers complexity for the sake of extensibility he doesn’t see the value of. By and large I would say nServiceBus codifies a particular approach to service buses and SOA in general and is deliberately designed to enforce that philosophy. Actually, I think the website says something to that effect too. One nice thing about nServiceBus is that Udi Dahan has lots of blog posts and pod casts discussing in articulate detail his SOA philosophy, which if you agree with it will you find direct support for those ideas in the project. The thing comes with a MSMQ transport, but the contrib project has an ActiveMQ transport and it sounds like an HTTP Transport will arrive shortly with a WCF/MSMQ transport coming at some indeterminate time after that. [Update] We’re now using Simple Service Bus, our own derivative of NSB, description below


Simple Service Bus

Simple Service Bus is a derivative (fork) of nServiceBus. Full details on the project can be found here, but the idea was to take the reliability and simple API found in nServiceBus and add in the extensibility and modularity required to enable us to design messaging systems in a more “building block” way, for better or worse. A pdf document providing a comprehensive overview of the project, including an architectural description / diagram, can be downloaded here. The project home page on CodePlex is here.

BizTalk Services

Microsoft is building a SaaS messaging bus with some slick features, orchestrations via WF and more significantly, I think, some sort of federated identity service using (I believe) a standards based STS service. However, it is currently in CTP and is not at all ready for production use (according tot he website). Plus, nobody knows what it will cost.  I think it is designed for partner-partner integrations and I’m not sure if it is intended for the Internet Service Bus (ISB, what they are calling BizTalk services) to be a replacement for an Enterprise Service Bus, but perhaps a compliment. Not sure on that one though. In any case, it is worth watching (and possibly playing with if you have time) but not an option for production use just yet.


Linxter ISB

Another ISB in beta, this service is similar ( I think ) to the core purpose of BizTalk services (hosted, secure messaging across the open Internet), with a slightly different feature set. I posted about Linxter here.

That’s all I could find. Personally, I think there is room for more. Resources below. Links to project/product home pages are in the titles of the products themselves.

Neuron-ESB-vs.-Microsoft-ESB-Guidance-A-Quick-Comparison

Neuron posts from Sam Gentile

NServiceBus - Makes Building Enterprise .NET Systems Easier

http://udidahan.weblogs.us/

Mass Transit author blog

Very brief review of Mass Transit code

Simple Service Bus - Overview & Architectural Description

SOA, ESBs and JBOWS, oh my!

First, I should say I am a total neophyte when it comes to SOA or ESBs. I just stumbled into this SOA party, with my fanny pack and my walking stick and my perpetually bewildered expression. But as a neophyte struggling to understand concretely what I instinctively understand is a vitally important concept or movement or architectural style or religion or whatever SOA is, I must say my bewildered expression isn’t out of place. Many blog posts or articles on the subject open with a comment on how notoriously undefinable SOA seems to be - at least to the extent that experts in the field, whoever they are, can agree on something. There are some things everyone seems to agree on, though. The most prominent of these is “SOA is not a technology.” This is often followed with a stern warning about how you can’t “buy it”, it doesn’t come “in a box”. There also seems to be consensus among those who do a lot of thinking (or at least writing) about SOA that it is about “Business-IT alignment.” That sounds like a catch phrase to me that doesn’t communicate a whole lot of useful information, but I take it to mean that the practice of SOA is a holistic practice, business wise one that seeks to understand the larger strategic goals of the organization as a whole and organize and coordinate its myriad software systems and the business processes they support in service of the Big Picture, the Big Picture ultimately and inevitably being profitability (what else is there in this glittering capitalist wonderland of ours?)

And then there is the ubiquitous prejudice (bordering on hatred) that SOA and its practitioners seem to universally hold against the poor beleaguered “Silo”. (Oh Silo, who weeps for thee?)  And finally, to finish up the things everyone seems to agree on, it seems that SOA is definitely not JBOWS.

Oh, and SOA is loosely coupled. It is definitely loosely coupled.

However, none of those aspects of SOA says much about services or architecture, except maybe the loosely coupled part. Business alignment, agility, “not technology” - those seem like outcomes or goals of a particular type of architecture rather than a description of an architecture - or even a description of an architectural style (like the mud hut or Victorian mansion). Nevertheless, SOA is about architecture, because architecture is in its name - but it is the concrete part of the thing that seems to be in such hot contention - do you need WS-*, an ESB, must you have a regipository, should messaging be asynchronous, Request Response, point to point, brokered, RESTful, Queued, should services be layered, data oriented, process oriented oh goodness, just don’t buy it in a box! Because that WON’T be it!.

Well, it is all very confusing. You kind of have to pick a camp and do your designs that way, when you’re fresh meat like me and haven’t been seasoned in the fires. I myself have selected from the shelves a nice, fresh Bus based, business unit centric sort of approach to SOA a la Udi Dahan, but am still very curious about the others, the ones with repositories and registries and that use WCF and WS-* and REST and dynamic routers and whatnot.

Anyway I was reading Sam Gentile’s educational SOA series  and it led me to reading some other thingsand I came across a metaphor for SOA that finally made it click for me in a real way, and now I’m not so perplexed about all the various hotly debated approaches because they are just (very important but equally valid) different approaches to a unified, relatively stable vision of what SOA is - because SOA is technology, of course, and it is architecture, but it isn’t a style of architecture, like mud huts VS Victorian mansions, it is a scope of architecture, like Building Design VS Urban Planning. SOA is about turning ad hoc communities of software and process into an integrated economy composed of towns that are part of larger counties that are part of larger states, and so on. SOA is about the design and execution of the master plan, the infrastructure and government and laws that all of an organizations IT entities must follow to enable peaceful, productive commerce all around. And what isn’t mandated by the law of the land is left up to the organizational units to rule themselves and their component sub communities as they please. How autonomous of them. All very hierarchical and organized. I also suspect (based on what I’ve read, mind you, not what I’ve experienced) that SOA’s often tend to grow like many cities, patches of disorganized systems agreeing to abide by a common set of rules, setting up transit systems and infrastructure between them, and these become cities which become models for other areas of the organization to follow suit.

What I like about this metaphor is that it captures many of the individual desired outcomes of SOA pretty nicely in your brain, and this is important because trying to picture what “driving increased strategic business agility and alignment into the enterprise” looks like in a way that can help you build it is a rather tall order.

For instance, perhaps each LOB application is something like an individual business or even a person who has some autonomy, makes some decisions for himself, but for the most part he must communicate in a language common to his local community and must follow the local customs, such as a specific technology stack. The larger communities to which each citizen belongs have more autonomy as a group. Maybe these are Services. They have their own culture, their own customs, and by and large they do things their way. But as a unit they must participate in the larger economy, they must import and export goods and services to other communities in an organized, coordinated way and report progress and pay taxes to the governing body and so on. Therefore at least those members of each community (endpoints) charged with conducting business on behalf of the community must,  in addition to understanding local culture and customs, also speak the national language, convert local money into a common currency, understand the global business protocols.

Clearly, you can’t “buy” something like this - you don’t march into to the wild west and order your “city” from the Sears catalog (or your favorite vendor), complete with a set of laws and a functioning government. You can buy a bunch of concrete and tools (such as WCF), or even a fully functioning subway system (such as an ESB), but the thing itself, the transformation of wild tribal cliques into a coordinated, modern, centrally governed “society” of connected systems, must be built piece by piece, tailor made, using many different kinds of physical infrastructure along the way as appropriate while implementing appropriate rules, laws, protocols, etc. For the whole rogue crowd to work together in a single, functioning digital economy you really need to gather the tribal chiefs and understand their needs and get participation and buy in because if you don’t then if not an outright revolution at least a certain amount of subversive activity is certain to undermine the functioning of your new society. No vendor can sell the solution to that.

Incidentally and off topic, this paints me a nice image for JBOWS. I like to imagine taking disparate groups of people over a wide geographical area who all speak different languages and dialects and have totally different cultures and giving each of them a telephone and a set of phrase books and expecting them conduct meaningful and efficient commerce. Throw a phone book in the mix and you have JBOGS.

So anyway, this may or may not resemble what people mean when they talk about SOA, but it is what I have managed to glean, academically speaking, from all the shrapnel that is flying around.

Scripting, Coding & Writing DSL’s in Boo - Resources

Boo is an interesting, extremely flexible CLR language with an “open compiler architecture” - which means that you can easily extend its compiler to accommodate your needs. I bought a Manning Early Access Program (unfinished, unedited, and rough around the edges) book by Ayende Rahien called Building Domain Specific Languages in Boo that, despite the writing style, which conveys an accent and is so informal that it almost borders on the insane, has already proven quite useful. The book relies on an open source library, also by Ayende Rahien (of Rhino Mocks fame) called Rhino DSL. The library itself is small, simple and effective. In addition to DSL’s, Boo (which is a statically typed, compiled language) has an optional interpreter and seems like it could be effectively used as a runtime scripting language, in addition of course to being a full blown .NET language you could write your business objects or UI in.

Anyway, below are a few resources I have come across while researching this topic in support of creating a state machine DSL referenced in my previous post:

Scripting With Boo
Binsor : Castle IOC container configuration using Boo DSL from Ayende

DSL Support
Early Access Book: Building DSLs in Boo (Ayende)

Specter Framework - writing executable object specs using a Boo DSL

An interesting example of TDD using Specter
Article on DSL’s using BOO by Ayende

A list of open source apps written in boo

Martin Fowler is writing a book on DSL’s also

Using Boo as an embedded scripting language

Visual Studio Integration via BooLangStudio

SimpleStateMachine project using a Boo DSL for configuration

SimpleStateMachine CodePlex Project

I have posted a CodePlex project here that implements a simple, customizable State Machine & miniature runtime. The point of this project was to replace Windows Workflow Foundation in one of our applications as the state machine engine. Windows Workflow foundation is a powerful, robust technology but turns out to be unsupportable overkill if all you are really after is defining and coordinating state transitions in your application. Plus, it can be rather difficult to unit test, even more difficult to version and extend once an application is deployed, and requires all sorts of baggage in the form of database infrastructure and GAC requirements, etc.

Anyway, this library does nothing else besides let you define a set of events, states, and transitions between states when triggered by events, and optionally allows for user defined actions to fire during the initialization or finalization of any or all states.

As for representing the actual state machines there are a few options: A visual designer such as WF provides. The WF designer itself is awful for representing state machines, even simple ones, as it was designed to suit sequential workflows as well and the two kinds of diagrams have different needs. It might be possible to create a visual designer that did the job well, but it would be a big task. Another option is to configure state machines in code using an object model or fluent interface. This would have worked, but it ends up looking pretty ugly after a while and you can’t change your state machines without deploying updated assemblies.

The approach I settled on was to use a DSL, using Boo and Rhino DSL, as described by Ayende Rahien in his (unfinished) book on the subject. The State Machine language itself was inspired by Martin Fowler, who appears also to be writing a book about DSL’s.

An example state machine definition looks like the sample below. It isn’t as snazzy and colorful as the visual designer version you get with WF, but in practice it is much more practical.

workflow "Telephone Call”

trigger @CallDialed
trigger @HungUp
trigger @CallConnected
trigger @LeftMessage
trigger @PlacedOnHold
trigger @TakenOffHold
trigger @PhoneHurledAgainstWall

state @OffHook:
	when @CallDialed             >> @Ringing	

state @Ringing:
	when @HungUp                 >> @OffHook
	when @CallConnected          >> @Connected

state @Connected:
	when @LeftMessage            >> @OffHook
	when @HungUp                 >> @OffHook
	when @PlacedOnHold           >> @OnHold

state @OnHold:
	when @TakenOffHold           >> @Connected
	when @HungUp                 >> @OffHook
	when @PhoneHurledAgainstWall >> @PhoneDestroyed

state @PhoneDestroyed

The .NET Framework Client Profile

is a trimmed down version of the .NET framework aimed at providing a leaner deployment package for systems that are just intended to run .NET client software. It ends up being around 23mb and should make it easier to deploy on the web to computers that might not have the latest version of .NET installed. Below is a link to a blog that has a link to a list of assemblies included in the profile :) 

POKE 53280,0: Pete Brown’s Blog : What’s in the .NET Framework Client Profile?

ASP.NET without ASP.NET - SharpTemplate.NET pre-release on CodePlex

This looks like a fantastic, flexible, simple to use template engine for .NET. I can’t imagine not finding a use for this in the near future. I haven’t dug any deeper, but LazyParser.NET may have significant promise as well.

 

ASP.NET without ASP.NET - SharpTemplate.NET pre-release on CodePlex

nServiceBus Management Extensions - new CodePlex project

I just launched a new open source project on CodePlex, NSB Management Extensions. This purpose of this project is to provide endpoint management, such as endpoint heartbeat monitoring & node performance monitoring, which exists in the initial release, as well as some more advanced services such as service/endpoint repositories, event broker with server-side subscription rules, a processing pipeline for the MSMQ Transport, etc.

The project can be found here: http://www.codeplex.com/NSBManagement

Unit Testing Custom .NET Configuration Classes

I am currently in the process of writing some custom .NET Configuration classes, such as custom ConfigurationHandler implementations, ConfigurationElementCollection, etc. Configuration classes need to be tested just like any other code, especially if they contain any behavior beyond the simple mapping of an XML property to a class property. It turns out the .NET Configuration API makes this pretty easy:

 

    [TestFixture]
    public class ConfigurationTestFixture
    {
        [Test]
        public void TestCustomConfig()
        {
            var cfg = GetTestConfiguration();
            Assert.That(cfg, Is.Not.Null);

            var mySection = cfg.GetSection(”mySection“) as mySection;
            Assert.That(mySection, Is.Not.Null);
            Assert.That(mySection.CustomProp,Is.EqualTo(”configuredValue“));
            …

        }

        private System.Configuration.Configuration GetTestConfiguration()
        {
            var map = new ExeConfigurationFileMap()
                          {
                              ExeConfigFilename =
                              Path.Combine(AppDomain.CurrentDomain.BaseDirectory,
                              "TestApp.config“)
                          };

            return ConfigurationManager.OpenMappedExeConfiguration(
                    map, ConfigurationUserLevel.None);
        }

 

For this to work you’ll need a configuration file with the appropriate XML to test your classes stored at the root of your test project with the Copy To Output Directory property set to Copy If Newer. You could use any path to create your ExeConfigurationFileMap, but for me putting the file at the root seemed the easiest solution.

image