Thoughts and musings on scalability, messaging, Windows Azure, and programming in general


The Industrial Revolution, part 3

(read part 2 here!)

"Don't do it yourself"

The primary way worker efficiency can be improved is an old principle: "Don't do it yourself".  If you want a particular worker to produce tables twice as fast, stop making him saw his own wood, but supply wood pieces in the correct sizes.  The worker can now specialize in composing tables from those precut parts, while another can specialize in turning raw materials into correctly sized wood pieces.  The next step after that is buying the wood from someone else.

In the 19th century, interchangeable parts became the name of the game.  It was no longer common for a worker or even a factory to produce everything they needed themselves from raw materials.  They could go out and purchase the parts from any other manufacturer.  The godfather of this evolution was Henry Maudslay.  Maudslay is credited with inventing the first bench micrometer, a device capable of measuring anything his factories produced to an accuracy of 3 micrometers, or 0.003 millimetres.  It gave us compatible thread and diameters on screws, nuts and bolts, and quickly became one of the biggest motors for standardization.  The end result was that factories didn't need to produce custom parts for their products any more. They could buy parts with standard measurements or have things produced for them by providing the exact measurements.  They could trust that whatever they bought, if it conformed to those measurements, it would work in their own products.

Why an 19th century inventor matters to software

Software hasn't reached this point yet, but we are on the right path.  Open standards have always been important, but in recent years more and more software companies have jumped ship from their proprietary protocols and formats to open industry-wide standards.  This goes an all levels, from how you send data through a copper wire to queueing protocols.  As a direct consequence of that, loose coupling -that holy grail of robust software- is made easier.  Service Oriented Architecture, CBSE, Microservices, Actor model, Communicating Sequential Processes, it doesn't matter what you call it.  The principe is always the same.  Build an application out of independent parts that communicate, and it will be more robust, easier to scale, and easier to maintain and improve.  Working more with open standards are now making this easier and allow us to build components on entirely different languages and paradigms, but still compose them into functional software.

The next step is to make these components so standardized that they can be switched out at will.  If I can switch out components of my application at will, I can also build software that way.  In theory, I won't ever need to know how code is written, or how the protocols function.  I could take a bunch of components, build my flow from those, and be done.  Software would go from a craft to a commodity.  This goes for distributing it to clients as well.  Everybody in corporate IT dreams of an install procedure that consist of exactly one step: clicking 'done'.  If protocols are standardized enough, that's all that will ever be required.

Standards, standards standards

All this means having a default way of communicating, so commands and queries can be issued without worrying about how they will get to destination.  That's the easy part.  The hard part is how you communicate your intent to a component.  There are no standard commands to send to a service that stores something, or that sends an invoice.  We have it for e-mail in SMTP, why not for accounting software?  What is the standard protocol to address a social network?

What we are missing is Mr. Maudslay.  There is no software equivalent to the micrometer.  There is no way to define a standard, and then run a battery of checks to see if it will behave correctly in all situations.  Unit testing and TDD help us on this path, but they aren't far-reaching enough.  The goal to shoot for should be being able to write any component, and validate that comprehensively against the intended standard protocol, without writing any tests yourself.  It also requires a standard protocol for providing standard protocols (Yo dawg...), so someone in need of a custom component can provide the exact specifications it should meet, in the same way a factory provides a set of measurements to its suppliers.
I don't know if it is even possible to build such a micrometer.  Theoretically it must be, but it requires wide adoption of a common system.  This kind of wide spread has happened in the past, but it is never easy.  My magic eight-ball is clear: "Ask again later."