The true role of domain specific languages

It is easy to be confused by the term domain specific language. It sounds like a fancy term for jargon. It is often interpreted to mean some form of specialized language. I would like to explore another role for them: as vehicles for policy statements.

In mathematics there are many examples of instances where it is easier to attack a problem by solving a more general, more uniform, problem and then specializing the result to get the desired answer.

It is very similar in programming: most programs take the form of a general mechanism paired with a policy that controls the machine. Taken seriously, you can see this effect down to the smallest example:
fact(n) where n>0 is n*fact(n-1);
fact(0) is 1
is a general machine for computing factorial; and the expression:fact(10) is a policy ‘assertion’ that specifies which particular use of the factorial machine is intended.

One important aspect of policies is that they need to be intelligible to the owner of the machine: unlike the machine itself which only needs to be trusted by the owner.

So, one critical role for a DSL is to be the policy language for the user of a mechanism.

Starting from this light leads to interesting conclusions. In particular, DSLs should be ubiquitous not rare; in particular, DSLs support the role that abstractions play in programming: by layering an appropriate syntax on top of the expression of the abstraction. It also means that programming languages should be designed to make it easy to construct and use DSLs within systems as well as externally.

A simple example: the query notation in Star, as well as in formalisms such as LINQ, may be better viewed as simple DSLs where the user is the programmer. The difference between these and more traditional DSLs is that the DSL expressions are embedded in the program rather than being separated from the code.

I think that embracing the DSL in this way should make it easier for a programming language to survive the evolution of programming itself. With an effective DSL mechanism a language ‘extension’ encoding a new language concept (for example, queries over C# or objects over C) and be done without invalidating the existing language. (The mechanisms in Star go further: it is possible to construct a DSL in Star that either augments the base language or even replaces it. We have used both approaches.)

It also explains why LISP’s macro facilities have allowed it to survive today more-or-less unchanged (nearly 60 years after being invented) whereas languages like Java and C++ have had to undergo painful transitions in order to stay relevant.

Blue sky vs applied research

So, one of the topics that has come up from time to time concerns so-called Blue-sky research versus applied research. The image of a blue sky research project is one of a researcher or small group of researchers having fun dreaming up some cool technology but with no relation to the real world.

I have always been a little uncomfortable with this distinction. The reasons are two-fold: in my experience, the time-scales associated with a project bear no direct relation to whether the research is ‘blue sky’ or ‘applied’; secondly the actual work done in a research project may be incremental or grand in both blue sky and applied. (In fact, given the general reluctance of people to fund blue sky research, they tend to be smaller and less grand than applied projects.)

So, here is a different, more grounded distinction that, I believe, is more authentic: Bottom-up versus top-down research.

Bottom-up means, of course, exploring from what you have and seeing if there are any serendipitous opportunities that make them themselves apparent. By its nature, you cannot predict the outcome of bottom-up work, but someone has to have some kind of intuition.

Top-down means problem directed. In my book, that is enough to make it applied. You are trying to solve a problem.

The reality dimension (i.e., is the research realistic or not) shows up independently for either. Some bottom-up projects are highly realistic, other top-down projects are somewhat unrealistic.

A missing piece

For some time now I have been thinking that the key to a framework for safe and effective action across the Internet was the Norms and Institutions paradigm. In that paradigm, the key concept (it seems to me) was the social act; or rather, that all human actions are rooted in the social context.

At the same time, we have been working on the SOA Reference Architecture, and I feel that it is important that SOA itself be grounded in the N&I paradigm. In the approach that we are taking, we are taking multiple views on the SOA theme. Each view is characterized by a viewpoint — literally a point of view of the architecture. One of these views is the Service as Business view, and this view is expressed in language that would be instantly familiar to anyone aware of the Norms and Institutions framework.

A challenge in such a multi-layered approach to architecture is to link the different levels together in a convincing way (ideally, in a way that is both clear and obviously supporting).

In this case, one such challenge is in linking the information exchange level of the SOA (e.g., SOAP message exchange) with the business view (e.g., buying a book). It is certainly true that any interpretation of a SOAP message as anything other than a bunch of XML is somehow extraneous to the SOAP message itself. SOAP (and related specs) simply does not have the language or conceptual framework to deal with book buying (say).

Hence the concept of a business transaction. For business types, this is a fundamental idea: we agree that you will ship me the book and I will pay for it. From the business perspective how this agreement is reached is pure IT (and boring).

For us, however, we need to relate a business transaction to an interchange of SOAP. This leads to the concept of supervenience.

One system can be said to supervene over another if (a) the first system uses the second in a replaceable manner (a different message exchange mechanism would do just as well) and (b) the second lower-level system is neutral in respect of the higher-level system. I.e., if the higher-level use is simply one possible application of the lower-level system and could be applied to other uses without fundamentally affecting its functioning.

So, the business transaction ‘system’ supervenes on the message exchange system quite nicely. Another way of saying this is that business transactions provide an effective way of capturing the meaning of SOAP-level information exchange at a level that is meaningful to people who are not IT geeks.

Safe and effective software

Someone recently asked me why I was working on the particular topics that I was interested in. I am afraid that in the heat of the moment I had a reasonable but ultimately lame answer (something about reducing friction in the marketplace).

In fact, the true answer is simpler and much more powerful. I want to be part of a ‘professional’ industry, and I believe that we are not really there yet. It is a constant source of amazement to me that there have not been any class action lawsuits against certain high profile software companies.

I like the phrase safe and effective, which describes the basic requirements for medicines of course, but should be equally applicable to software.

What would the benefits of being able to label a system safe and effective? Primarily it means that someone using the system has some assurance that the software will do what it is supposed to do, and that it wont lead you into trouble.

Of course, if you take too many aspirin, or if you misuse a software system, it can hardly fail to cause you grief: so in the end safety and effectiveness is never absolute. But we, as an industry, can do a lot better than we do today.

In the case of services, what does it take to be safe and effective?

I have written before that SOA is about action at a distance. We want to be able to use the Internet to achieve our objectives, to perform tasks. This is fundamental difference between SOA and the Web architecture (for example).

For services to be widely used it will be necessary to be able to show that they are safe and effective too.

Since it of the essence that service is about crossing ownership boundaries, about you using my system, a key piece of figuring out Safety and Effectiveness in relation to services is in being able to answer some simple questions:

1. What is the effect of using the service? (Will it cure my disease?)
2. Do the participants in the service have the appropriate rights in using and delivering the service? (Is the patient an appropriate target for the medicine?)
3. What are the potential side effects of using/offering the service? (What are the complications?)

That is the heart of the problem it seems to me. And, I firmly believe that when we can reliably answer these issues in a systematic way then we will be entitled to call ourselves a profession.

How to run a research lab

The key premise of running a research lab is that it is a form of investment. There may be many motivations for investing in research, but some of the more common ones include

• The big payoff

This is, in effect, a form of gambling. You put up a lot of money and hope that some of it will lead to a new ground-breaking profit that will allow you to clean up.

• Insurance

You want to reduce your exposure to some long-term risk that might come out of left-field and blow you away.

• Fill the pipeline

You need someone whose skills are developing new products, but not necessarily manufacturing products, to keep the pipeline full.

All of these are legitimate, although the first 2 are for ‘hi-rollers’ only: research labs are inherently expensive and these uses are particularly unlikely to pay off. When and if they do pay off then the rewards can be immense (think of Xerox Parc). But, like the lottery, you could live and die without seeing the benefit; and think that your money was wasted.

I have already argued that the third option is not really suited to a central research lab. The leader of the Business unit that owns the product family should also lead the development of new products.

There is another option not often considered, but I think critical:

• Taking care of critical success factors

Addressing technologies, marketing, etc. that are critical to the success of the company; but not inherently directly linked to particular products.

The idea is that there are topics in any business that are ‘at the heart’ of the business but not necessarily contained within a given product.

A great example of a CSF for a software company is security. Security is clearly important: a security failure can destroy a business. Security affects many (all) products but is not easily confined to a single product or technology.

Addressing security is best done from an overall/overarching perspective. Incidentally, as ay security specialist will tell you, security technology is not limited to encryption but includes policy management, architectural considerations and many other factors.

Putting a team to work to ‘own’ security would be a sound strategy for many companies. That team would take responsibility for ensuring that the company had the best security strategy and execution possible. That team is best placed centrally: for example in a central Corporate research lab.

Another good example CSF for a software company would be usability. Usability is another one of those make-or-break aspects that can lead to riches or disaster. It is also something that applies to all the products and services offered by a company. And so, a usability team may also be a wise investment; again placed in the central lab.

The pattern is that these CSFs often denote important properties that one would like to be associated with all the products and services offered by a company. And this importance is inherently connected to the relationship between the company and its customers.

Viewed this way, it is easy to see how and why a company might invest in a research lab that focuses on critical factors; and for that investment to be sustainable in bad times as well as good.

The Role of Research Part II (Innovate or die)

When I was growing up, the overwhelming reaction I got from people from all walks of life was that change was something to be avoided. I found it frustrating because, to me, it seemed that there was so many exciting things that could be happening but noone was interested.

That has changed today. Nowadays, it is accepted wisdom that you must innovate – or suffer the consequences.

Providing services, providing products

We can classify, roughly, a business into two primary styles – a service provider or a product provider. Be aware that the line between these can get pretty fuzzy. However, a key difference is in the nature of the relationship with the customer: a service provider ‘takes control’ of the relationship by offering to directly meet some requirements. A product provider is also focused on meeting requirements; but is more indirect: the product provider sells a tool to the customer that enables him or her to meet his or her requirements.

There are other differences between the two, but our focus here is on the styles and modes of innovation applicable to each. And the resulting implications for research efforts.

The primary locus of innovation for a service provider is in production – in the processes involved in delivering the service. For example, in a Systems Integration business, the primary point where innovation must happen is in the process of capturing requirements, constructing specifications, implementing systems, testing, deployment and customer management. Of these, of course, the hardest to get right, and the most critical, is the first – capturing requirements. Get that wrong and the whole exercise becomes expensive very rapidly.

For a product provider, the story is more complex. Because a product provider delivers a tool to the customer, whether its a mobile phone gadget or an Enterprise suite, there are two primary loci of innovation: in the product itself and in the process for manufacturing and marketing that product.

Product providers are often organized around the products that they develop and sell; with separate units focused on separate products. A product manager is tasked with ensuring that the products meet customers’ requirements and can be manufactured at reasonable cost. For software products, the manufacturing process itself is trivial, the main costs are in development (i.e., research), in marketing the product and in supporting customers’ use of the product. Since most of the upfront investment in a software product is in development, that is a critical point for a software product manager to focus on.

In an environment of change, especially one where change is actually accelerating, it is important that the management of a product provider is aggressively evolving the product offering as well. This means that a well managed product provider has a stream of new products coming out; each hopefully better than the previous and each better able to solve customers’ needs.

That, in turn, seems to suggest that the real character of a product provider is one of a process of developing new products! Except that the customer for this process is the company itself. The management of the product provider must be geared to the process of constructing new products. If the management is effective, then they are the best placed team to solve the problem of developing new products.

Product research is best not done in centralized labs

This one of the fundamental reasons why research labs are not good places to develop new products: that job is primary responsibility of the managers of the product and service providers! And trying to do product development in research labs puts them in conflict with the product providers.

On the other hand, anything that is not directly connected with the next product is fundamentally a distraction for the product manager. He or she is focused on putting together the pieces necessary for the next project. He will take in any number of inputs and requirements, so long as they relate directly to the product at hand.

This is one of the reasons why the product development environment is not a good locus for tool development. It is a well known fact of management that developing infrastructure tools generally slows down the development of the project the tools will be used for.

For example, consider the problem of developing a word processor. Now, imagine a time long ago when there were no object oriented programming languages. The manager of the word processor product might say: “I wish that we had an OO language, then it would be much easier to develop our word processor”. He might be right; but if he then goes on to develop resources into building that OO system, he will inevitably slow down the development of the word processor itself!

In general, the manager of the word processor team must try to minimize any development that is not directly focused on the word processor itself. Even to the point of saying that “even if an OO language would make my WP much more reliable, faster, etc., I can’t spend any resources developing the OO language because that is not my task”.

Note that this includes the deployment of tools coming outside the team: the OO language might exist but if noone in the WP group knows how to use it then that can easily slow the development cycle down.

However, and this is the kicker for the WP team manager: he has to be willing to change his product development process if he is going to remain competitive with other WP manufacturers. It is just that the natural energy for developing products is contrary to the adoption of and especially the development of general tools.

So, the take home message is that the proper place for developing new products is inside product units, and that centralized research labs are inherently poor places to develop products. Conversely, product units are not very well placed to develop infrastructure. There is a glimmer of a rational structure here.

In the next (and final) installment, I will try to outline what I think is a rational basis for organizing technical research in a commercial venture.

Participants

Today it was the turn of participants to get a hammering.

A SOA is all about people; about people getting things done. What is more, people have all kinds of relationships to each other . What, you might ask, has that got to do with SOA? A lot in my opinion.

Here is an example. Suppose that someone wants to set up a system that allows, among other things, groups, teams, and societies (such as the Society for the Preservation of Cats in the San Francisco neighborhood) to be formed and policed. There are many situations where that is already happening, just take a look at MySpace. Well, each society is going to want its own rules for governing itself. Of course, there is always an overarching set of rules that we abide by: the state and federal law; but federal law tries not to interfere too much in the way that the SPCSF is run.

Our SOA that is supporting all these societies might allow people to execute actions via the SOA: such as declaring that someone is the president, a meeting quorate and so on.

What is a service to do when it receives a request to promote a society member the new president of that society? Certainly, we could restrict the system to record keeping, but a deeper system will try to ensure that the promotion is valid. That involves knowing if the promotion action is from a valid source, whether the action is properly enacted and so on.

All of these validations, and the consequences that follow from the action (the new president can call meetings to order) are really about these other ‘out of band’ relationships between participants in the SOA system.

The same is really also true for ordinary commerce: a minor may not legally enter a contract (and therefore may not legally buy anything over the Internet). Such a fact is currently respected in the breach more than the observance; but I expect that things like SOX will change that for a lot of people. Enterprises are a lot more like mini-societies than most IT staff (and C*Os) give credit for.

We are trying to ensure that our Reference Architecture properly accounts for the possibility of these out of band relationship because they affect the performance of the system itself. Of course, it is not our job to define what the rules are; only how any set of rules might integrate with SOA-style systems.

Action at a distance

We are currently writing our first draft of the SOA Reference Architecture. Everyone is very busy doing their bit.
My current section is on the Real World Effect of using a service. The RA is really an abstract architecture: we are not focusing on things like SOAP, or any of the other 60+ Web service specifications out there. We are trying to get at the essence of makes SOA special and how it can be made to work.
It is a pretty basic aspect of services that we are trying to get something to happen: buy a book, get the weather forecast whatever. In other words: its action at a distance. I am communicating with you in the hope that we can get some mutual benefit.

This already distinguishes SOA from the Web, whose basic abstraction is to acquire a representation of a resource will be rendered locally for human consumption. Actions are not inherently about representations, they are about changing the world – one book at a time.
Action itself is a very difficult concept to get hold of. It seems to be intimately associated with intention and agenthood. Only agents can perform actions: the atmosphere may be very powerful but we do not think of it in terms of acting to create weather. Storms happen as a result of a confluence of different regions of hot and cold, wet and dry air, they are not created by a weather god stirring the pot.
Action at a distance is even weirder and harder to get hold of than simple action.
The reason is that you cant easily see it. If you had an agent that invoked the services of a service, and looked at its code, you would be hard pressed to give a clear reason for distinguishing

otherAgent.doTask()

from simply

library.doTask()

You have to know and understand the code of the agent at an entirely non-Java level: you have to know that there is another entity involved, that this agent is asking to do something, and so on. We simply do not have the concepts nailed down enough to encode them as programming language constructs.
In a way, that is one of my goals in pursuing Service Oriented Architecture. It seems to me that the light is finally beginning to dawn…

The role of research — Part I

My job is a research job, so a reasonable question is what is research in a corporate environment really for?. This is a two-part article; in this part I will concentrate on the problem. In the next part I will talk about the solution.

So, what is the problem of research?

If you ask an average person, even an average technical person, even someone who job is in research, what the role of research should be in an industrial context, my guess is that you will get an answer along the lines of

the job of research labs is to invent cool products for the business units to sell

Sometimes this is modified a bit:

the role of research labs is to support business units in whatever they want

I argue that this job description is both a poor mission for research labs and a poor use of the money spent.

First of all, consider an arbitrary business unit. Presumably, if they are doing their job correctly, they will have a constant eye out for products that they think they need – based on customer requests, gut feel, competitive landscape or all of the above.

I know that if I were a product manager in such a business unit, I would look first to my own people as a source of ideas and implementation for such missing products/functionality. I would only go somewhere else if I thought I couldn’t do it internally. But, even if I did go outside, it would be my ideas that would drive the process. After all, I own the process; it’s my responsiblity to ensure that my BU is a success.

I might go to a researcher if I had a particularly knotty problem; but most of the time the issues in developing products are not really all that hard. It is more a case of choosing the best available design compromise, tkaing into account developing technologies. The real issues in product design are to do with what customers you are targeting, what functionality you think that they will need and so on. Not rocket science generally.

So, what would make me go to the labs as a place to develop my ideas? As opposed, say, to going outside to another company or even to an ideas consultancy such as ideo? Certainly, if the labs had cool ideas I might consider them. But, as we shall see, the long term pressure on corporate research makes it an unlikely source for cool ideas.

One of the great problems with a product division using lab research directly is that their ‘product’ is hopeless from a productization perspective: I suspect that this is universally the case, but especially with software, a tool coming out of labs has to be rewritten from scratch to make it suitable. Its just too much trouble.

Also, typical researchers are very technology focused – they often ‘get the bug’ of a particular technology fix and spend their entire attention on promoting that particular hammer. For a hammer, everything is a nail. For a religious hammer, everything either is already a nail or should reconsider its position and become a nail forthwith. But that attitude is not very customer oriented; and it seriously limits the usefulness of the researcher to the BU.

Now consider from the lab’s point of view. A typical researcher is a proud animal – he/she has worked hard to get a PhD. He is used to thinking deep thoughts. He doesn’t want to have to spend time worrying about the user who spills coffee over his keyboard causing the database to suddenly start having a lot of wierd entries.

He typically doesn’t even want to think about someone who cannot progam a VCR using his smart tool. But product managers have to think about who the customer really is, not what they would like the customer to be. If the customer cannot program a VCR, that just means that the VCR has to be smart enough to not need programming. (VCR manufactures take note!)

All that productization effort is not research – it does not forward the cause of knowledge – it does not help towards getting papers or patents. It is strictly a waste of the researcher’s time and distracting from a careeer point of view.

This is not a recipe for success. In fact, I would argue that no research lab in modern times, especially a computer oriented lab, has been a successful partner for the main business when the lab’s mission is expressed in terms of developing cool products.

I have witnessed many companies struggling with their labs. In fact, I have noticed a pattern:

When a company is flush with money, they decide to establish a research arm. If the choice is paying tax (or dividend) or starting a research arm, why not spend the money and get some use from it? There are many models for research arms; many ways of funding them etc. But almost all of them start by giving the researchers carte blanche – to think great thoughts that will ensure even greater success for the parent company.

Life is good as a researcher if you have carte blanche and a bunch of like minded fellows to share lunch with.

One thing to note at this point is that most people, researchers included, are much better at solving problems than at deciding which problem to solve. If you ask a researcher what the important problems are, your answer is not necessarily any better than asking any one else. Deciding what to do is much harder than just doing it.

Then, the money starts to slow down. At the same time, the expenses of research start to go up. Its natural – you have a good idea but it needs more investment to realize the potential.

The typical reaction is for some bean counter to declare that their research facility ‘should be more accountable to the needs of the business’. Then what starts to happen is that the money that researchers get is no longer blue-sky but is more and more related to the goals of the business unit. Typically expressed in terms of products or technology that the BUs can market.

The managers of the BUs start getting pestered by researchers who need funding for their pet projects. The managers respond, typically, by treating the labs as outsourcing for their own product plans. The researchers end up becoming sub-contractors and being more and more product focused and less and less researchers.

Eventually, the BU managers will say that enough is enough – either you are part of my team (welcome aboard) or go away. The researchers might join the team (if they finally give up on their dream) or start polishing their resumes.

I personally have seen this progression in several Silicon Valley companies; as well as within my own. I have been pretty distressed by it all. Not because I think that it is wrong for research to support the goals of the company – far from it – but because I have seen that this model fails far more often than it succeeds. And because the net effect is mostly for the research labs to dissapear completely; having consumed vast resources along the way.

Something different is needed.

Something that respects the roles of the business unit, and of researchers. And that maximally leverages the natural instincts of the players.

I believe I have the answer for this; but you will have to wait for the answer