Integration Does. Not. Scale.

In times past, there was a difference between the front office of a business – designed to make a good impression – and the back office – a utilitarian place where most of the routine work got done. The first (and for a long time the predominant) use of computers in business centered around automating back office processes, so of course, IT was relegated to the back office. 

As businesses grew, various back office functions developed their own computer systems – one for purchasing, one for payroll, one for manufacturing, and so on. The manufacturing system in vogue when I was in a factory was called MRP – Material Requirements Planning. As time went on, MRP systems were expanded to the supply chain, and then to the rest of the business, where they acquired the name ERP – Enterprise Resource Planning.

Over time it became obvious that the disparate systems for each function were handling the same data in different ways, making it difficult to coordinate across functions. So IT departments worked to create a single data repository, which quite often resided in the ERP system. The ERP suite of tools expanded to include most back office processes, including customer relationship management, order processing, human resources, and financial management. 

The good news was that now all the enterprise data could be found in the single database managed by the ERP system. The bad news was that the ERP system became complex and slow. Even worse, enterprise processes had to either conform to “best practices” supported by the ERP suite or the ERP system had to be customized to support unique processes. In either case, these changes took a long time.

ERP Systems Meet Digital Organizations  

As enterprise IT focused on implementing ERP suites and developing an authoritative system of record, the Internet became a platform for a whole new category of software, spawning new business models that did not fit into the traditional processes managed by ERP systems. Here are a few examples:

1. Many software offerings that used to be sold as products are now being sold “as a service”. However, ERP systems were designed to manage the manufacture and distribution of physical products; they don’t generally manage subscription services.
2. Some companies (Google for example) give away their services and sell advertising. Other companies (such as EBay and Airbnb) create platforms that unite consumers with suppliers, often disrupting traditional industries. In a platform business, the most critical processes focus on driving network effects by facilitating interactions between buyers and sellers. Although ERP systems can manage both suppliers and customers, they usually do not focus on the interactions between them.
3. The Internet of Things (IoT) brings real time data into many processes, changing the way they are best executed. For example, predictive maintenance of heavy equipment can be scheduled based on sensor data, resulting in better outcomes for customers and thus for the enterprise. ERP suites are intended to support standard practices; they struggle to support processes that change dynamically in response to digital input.
4. Capitalizing on the availability of data generated by products, companies are moving to selling business outcomes rather than individual products (GE is an example). When you are selling engine thrust or lighting costs, rather than engines or lightbulbs, processes need to be focused on the customer context. ERP systems generally focus on internal processes.
5. ERP systems are supposed to provide a single, integrated record of important enterprise data, but that data rarely includes dynamic product performance data, information about consumer characteristics and preferences, or other information that has come to be called “Big Data”. This kind of information is becoming an extremely valuable resource, but there isn’t room in ERP databases to store and manage the massive amount of interesting data that is available.

In summary, digitization is bringing the back office much closer to the front office, providing the data for dynamic decision-making, and substituting short feedback loops and data-driven interactions for “best practices.” Since enterprise ERP suites were not built for speed or rapidly changing processes, they are increasingly being supplemented with other systems that manage critical enterprise processes.

Postmodern ERP

In the last few years, in the wake of the success of Salesforce.com, many cloud-based software services have become available. Some target the entire enterprise (NetSuite for example), but many are focused on particular areas (e.g. human resources) or particular industries (e.g. construction). These services are finding an eager audience – even in companies that have existing ERP systems. Today, about 30% of the spend for IT systems is coming from business units outside of IT [1]. If they cannot get the software they need from their IT departments, business leaders are likely to purchase cloud-based services instead.

The cloud reduces dependence on a company’s IT department, so it has become quite easy for various areas of the enterprise to independently adopt “best-of-breed” solutions specifically targeted at their needs, rather than use a single ERP suite across the enterprise. These best-of-breed systems are usually selected by line business leaders and hosted in the cloud. They tend to be faster to implement and more responsive to changing business situations than the enterprise ERP suite – partly because they are decoupled from the rest of the enterprise. Gartner calls the movement from a single ERP suite to a collection of ERP modules from multiple vendors “Postmodern ERP”[2]. 

Gartner warns that a multi-vendor ERP approach can lead to significant integration problems, and recommends that multiple vendors should not be used until the integration issues are sorted out. Of course, business leaders want to know why integration is important. IT departments typically respond that the ERP’s central database is the enterprise system-of-record; other ERP modules – financial reporting, for example – depend on this database for critical data. Without an integrated database, how will the rest of the enterprise be able to operate? How will the accounting department produce its required financial reports? 

Integration Does. Not. Scale

But hold on. There are plenty of very large companies that work remarkably well – and produce financial reports on time – without an integrated system-of-record. In fact, internet-scale companies have discovered that integration does not scale. If we go back to the year 2000, we find that Amazon.com had a traditional architecture – a big front end and a big back end – which got slower and slower as volume grew.  Eventually Amazon abandoned its integrated backend database in the early 2000’s, in favor of independent services that manage their own data and communicate with each other exclusively through clearly defined interfaces. 

If we have learned one thing from internet-scale players, it’s that true scale is not about integration, it is about federation. Amazon runs a massive order fulfillment business on a platform built out of small, independently deployable, horizontally scalable services. Each service is owned by a responsible team that decides what data the service will maintain and how that data will be exposed to other services. Netflix operates with the same architecture, as do many other internet-scale companies. In fact, adopting federated services is a proven approach for organizations that wish to scale to beyond their current limitations. 

Let’s revisit the enterprise where business units prefer to run best-of-breed ERP modules to handle the specific needs of their business. This enterprise has two choices: 

1. Integrate the various ERP modules and store their data in a single ERP database.
2. Coordinate independently-maintained enterprise data through API contracts. 

The problem with the first option is that integration creates dependencies across the enterprise. Each time a data definition in the central database is added or changed, every software module that uses the database must be updated to match the new schema. This makes the integrated database a massive dependency generator; the result is a monolithic code base where changes are slow and painful. 

Enterprises that want to move fast will select the second option. They will move to a federated architecture in which each module owns and maintains its own data, with data moving between modules via very well defined and stable interfaces. As radical as this approach may seem, internet-scale businesses have been living with services and local data stores for quite a while now, and they have found that managing interface contracts is no more difficult than managing a single, integrated database.

What Scales

Assume that every team responsible for a process can choose its own best-of-breed software module and is responsible for maintaining its own data in appropriately secure data stores. Then maintaining an authoritative source of data becomes an API problem, not a database problem. When the system-of-record for each process is contained within its own modules, new modules can be added for handling software-as-a-service, two-sided platforms, data from IoT sensors, customer outcomes or other new business model that may evolve. These modules will exchange a limited amount of data through well-defined API’s with the credit, order fulfillment, human resources, and financial modules. Internally, the new modules will collect, store, and act upon as much unstructured data and real time information as may be useful. More importantly, these modules can be updated at any time, independent of other modules in the system. In addition, they can be replicated horizontally as scale demands. 

It is the API contract, not the central database, that assures each part of the company looks at the same data in the same way. Make no mistake, these API contracts are extremely important and must be carefully vetted by each data provider with all of its consumers. API contracts take the place of database schema, and data providers must ensure that their data meets the standards of a valid system-of-record. However, changes to an API contract are handled differently than most database schema changes. Each change creates a new version of the API; both old and new versions remain valid while other software modules are gradually updated to use the new version. A wise API versioning strategy eliminates the tight coupling that makes database changes so slow and cumbersome. The reason why federation scales – while a central database approach does not scale – is because with a well-defined API’s strategy, individual modules are not dependent on other modules, so each module can be deployed independently and (usually) scaled horizontally. 

When you think of Enterprise ERP as a federation of independent modules communicating via API’s (rather than a database), the problems with multi-vendor ERP systems fade because the system-of-record is no longer a massive dependency-generator that requires lockstep deployments. With a federated approach, business leaders can move fast and experiment with different systems as they become available, and still synchronize critical enterprise data with the rest of the company. In addition, similar processes in different parts of the enterprise can use different applications to meet their unique needs without the significant tailoring expense encountered when a single ERP suite is imposed on the entire enterprise.

What about Standardization?

Won’t separate ERP modules lead to different processes in different parts of the enterprise? Yes, certainly. But the question is – under what circumstances are standard processes important? In the days of manual back office processes, there was lot of labor-intensive work: drafting, accounting, phone calls, people moving paperwork from one desk to another. Standardization in this kind of operating environment made sense and could lead to significant efficiencies. But in a digitized world, the important thing is not uniformity; it is rapid and continuous improvement in each business area. Different processes for different problems in different contexts can be a very good thing.

Jeff Bezos agrees; he believes that the only path to serious scale is to have a lot of independent agents making their own decisions about the best way to do things. This belief was a key factor in the birth of Amazon Web Services, a $10 billion business that keeps on growing. Amazon began its journey away from a big back end by creating small, cross-functional teams with end-to-end responsibility for a service. These teams designed their own processes to fit their particular environment. Amazon then developed a software architecture and data center infrastructure that allowed these teams to operate and deploy independently. The rest is history.

In Conclusion

It is time for enterprise processes become federated instead of integrated. This is not a new path – embedded software has used a similar architecture for decades. Today, almost every successful internet-scale business has adopted some type of federated approach because it is the only way to scale beyond the limitations of the enterprise. 

As digitization brings back-office teams closer to consumers and providers, they must join with their front-office colleagues and form teams that are fully capable of designing and improving a process or a line of business. These “full stack” teams should be responsible for managing their own practices, technology and data, meeting industry standards for their particular areas. They should communicate with other areas of the enterprise on demand through well-defined interfaces. 

The good news is that you can gradually migrate to a federation from almost any starting point, including an enterprise-wide ERP system. Even better, as IT moves from enforcing compliance with the company’s ERP system to brokering interface contracts and ensuring data security, it becomes a business enabler rather than a bottleneck. And best of all, responsible full stack teams that solve their own problems will create attractive jobs for talented engineers and give business units control over their own digital destiny. 

Don’t Separate Design from Implementation

I was a programmer for about fifteen years. Then I managed a factory IT department for a few years, and managed vendors delivering software for yet more years.  In all of those years (with one exception), software was delivered on time and customers were happy. Yet I never used a list of detailed requirements, let alone a backlog of stories, to figure out what should be done – not for myself, not for my department, not even for vendors.

In fact, I couldn’t imagine how one could look at a piece of paper – words – and decipher what to program. I felt that if the work to be done could be adequately written down in a detailed enough manner that code could be written from it, well, it pretty much had to be pseudocode. And if someone was going to write pseudocode, why not just write the code? It would be equally difficult, less error-prone, and much more efficient.

Software Without Stories
So if I didn’t use detailed requirements – how did I know what to code? Actually, everything had requirements, it’s just that they were high level goals and constraints, not low level directives. For example, when I was developing process control systems, the requirements were clear: the system had to control whatever process equipment the guys two floors up were designing, the product made by the process had to be consistently high quality, the operator had to find the control system convenient to use, and the plant engineer had to be able to maintain it. In addition, there was a deadline to meet and it would be career-threatening to be late. Of course there was a rough budget based on history, but when a control system was going to be used for some decades, one was never penny wise and pound foolish. With these high level goals and constraints, a small team of us proceeded to design, develop, install, and start up a sophisticated control system, with guidance from senior engineers who had been doing this kind of work for decades.

One day, after I had some experience myself, an engineering manager from upstairs came to ask me for help. He had decided to have an outside firm develop and install a process monitoring system for a plant. There was a sophisticated software system involved – the kind I could have written, except that it was too large a job for the limited number of engineers who were experienced programmers. He had chosen to contract with the outside firm on a time-and-materials basis even though his boss thought time-and-materials was a mistake. The engineering manager didn’t believe that it was possible to pre-specify the details of what was needed, but if a working system wasn’t delivered on time and on budget, he would be in deep trouble. So he gave me this job: “Keep me out of trouble by making sure that the system is delivered on time and on budget, and make sure that it does what Harold Stressman wants it to do.”

Harold was a very senior plant product engineer who wanted to capture real time process information in a database. He already had quality results in a database, and he wanted to do statistical analysis to determine which process settings gave the best results. Harold didn’t really care how the system would work, he just wanted the data. My job was to keep the engineering manager out of trouble by making sure that the firm delivered the system Harold envisioned within strict cost and schedule constraints.

The engineering manager suggested that I visit the vendor every few weeks to monitor their work. So every month for eighteen months I flew to Salt Lake City with a small group of people.  Sometimes Harold came, sometimes the engineers responsible for the sensors joined us, sometimes the plant programmers were there. We did not deliver “requirements;” we were there to review the vendor’s design and implementation. Every visit I spent the first evening pouring over the current listings to be sure I believed that the code would do what the vendor claimed it would do. During the next day and a half we covered two topics: 1) What could the system actually do today (and was this a reasonable step toward getting the data Harold needed)? and 2) Exactly how did the vendor plan to get the system done on time (and was the plan believable)?

This story has a happy ending: I kept the engineering manager out of trouble, the system paid for half of its cost in the first month, and Harold was so pleased with the system that he convinced to plant manager to hire me as IT manager.

At the plant, just about everything we did was aimed at improving plant capacity, quality, or throughput, and since we were keepers of those numbers, we could see the impact of changes immediately. The programmers in my department lived in the same small town as their customers in the warehouse and on the manufacturing floor. They played softball together at night, met in town stores and at church, had kids in the same scout troop. Believe me, we didn’t need a customer proxy to design a system. If we ever got even a small detail of any system wrong, the programmers heard about it overnight and fixed it the next day.

Bad Amateur Design
The theme running through all of my experience is that the long list of things we have come to call requirements – and the large backlog of things we have come to call stories – are actually the design of the system. Even a list of features and functions is design. And in my experience, design is the responsibility of the technical team developing the system. For example, even though I was perfectly capable of designing and developing Harold’s process monitoring system myself, I never presumed to tell the vendor’s team what features and functions the system should have. Designing the system was their job; my job was to review their designs to be sure they would solve Harold’s problem and be delivered on time.

If detailed requirements are actually design, if features and functions are design, if stories are design, then perhaps we should re-think who is responsible for this design. In most software development processes I have encountered, a business analyst or product owner has been assigned the job of writing the requirements or stories or use cases which constitute the design of the system. Quite frankly, people in these roles often lack the training and experience to do good system design, to propose alternative designs and weigh their trade-offs, to examine implementation details and modify the design as the system is being developed. All too often, detailed requirements lists and backlogs of stories are actually bad system design done by amateurs.

I suggest we might get better results if we skip writing lists of requirements and building backlogs of stories. Instead, expect the experienced designers, architects, and engineers on the development team to design the system against a set of high-level goals and constraints – with input from and review by business analysts and product managers, as well as users, maintainers, and other stakeholders.

A couple of my “old school” colleagues agree with me on this point. Fred Brooks, author of the software engineering classic “The Mythical Man Month” wrote in his recent book “The Design of Design” [1]:

“One of the most striking 20th century developments in the design disciplines is the progressive divorce of the designer from both the implementer and the user. … [As a result] instances of disastrous, costly, or embarrassing miscommunication abound.”

Tom Gilb, author of the very popular books “Principles of Software Engineering Management” and “Competitive Engineering” recently wrote [2]:

“The worst scenario I can imagine is when we allow real customers, users, and our own salespeople to dictate ‘functions and features’ to the developers, carefully disguised as ‘customer requirements’. Maybe conveyed by our product owners. If you go slightly below the surface of these false ‘requirements’ (‘means’, not ‘ends’), you will immediately find that they are not really requirements. They are really bad amateur design for the ‘real’ requirements…. 

"Let developers engineer technical solutions to meet the quantified requirements. This gets the right job (design) done by the right people (developers) towards the right requirements (higher level views of the qualities of the application).”

Separating design from implementation amounts to outsourcing the responsibility for the suitability of the resulting system to people outside the development team. The team members are then in a position of simply doing what they are told to do, rather than being full partners collaborating to create great solutions to problems that they care about.

_________________________________
Footnotes:
[1] “The Design of Design” by Fred Brooks, pp 176-77. Pearson Education, 2010
[2] "Value-Driven Development Principles and Values;" by Tom Gilb, July 2010 Issue 3, Page 18, Agile Record 2010 (www.AgileRecord.com)

Incremental Funding (Book Review)

The customer wanted an on-line currency exchange capability added to their on-line financial service offerings. They figured it would take several months to implement. But the development team suggested a different approach: start by hiring a dozen telephone operators and implement the necessary software for these folks to execute currency trades. The company gave it a try, and in six weeks the first iteration was ready. With no more than an 800 number on their web site and a rudimentary interface to the currency market, new business was being transacted and profits being made.

Over the course of the next several months, the on-line trading capability was implemented around the core module originally used by the telephone operators. Not only did the company see early revenue, but the risk of failure disappeared once trading started. In addition, system requirements were defined by observing real trades.

In the book Software by Numbers – Low Risk, High Return Development (Prentice Hall, 2004) Mark Denne and Jane Cleland-Huang make the case for incremental delivery of software. Mark Denne developed the Incremental Funding Methodology (IFM) in the 1990’s to help land a large software development contract. In an attempt to distinguish his bid from the pack, he reorganized the deliveries into units of value, and adjusted the development sequence so that the customer would realize revenue faster and in the end, receive a greater return on their investment. The customer discovered that this approach dramatically reduced their need to borrow money and gave them earlier product release with lowered risk. In what seemed like hotly competitive bidding process, Mark’s company won the bid by emphasizing “time to value” instead of development efficiency.

Software by Numbers recommends dividing a project into Minimum Marketable Features (MMF’s). These are small feature sets which deliver some identifiable value to the customer. Each MMF should have its own return on investment (ROI). By laying out the potential ROI’s of various feature sets, an optimal development sequence for the MMF’s can be determined. Early deployment of key MMF’s reduces risk while generating revenue to help fund the remainder of the project.

When business people and software developers focus on identifying and valuing marketable features, their conversation is changed. Developers are exposed to ROI and stakeholders are confronted with the realities of software development. The entire team is focused on achieving business ROI early in the development cycle. Management sees continuously measurable progress, and the team benefits from the early and compelling feedback generated by real software being used in production.

With so many benefits, why wouldn’t IFM be the preferred approach for developing software? Denne and Cleland-Huang note that many practitioners view software architecture as a monolithic whole, requiring early definition because of the extensive impact that architectural changes can have on a system. On the other hand, they argue, it is not until the details of an architecture are implemented that one can tell if the architecture is viable. Thus architecture presents us with a chicken and egg problem.

The book recommends that architectural elements which support each set of MMF’s should be developed with their respective MMF’s. In other words, architectural development should be sequenced using the same financially driven priorities as feature development. While there may be times where architectural coherency may dictate early development of features not immediately related to the current feature set, in general it is not only possible but also preferable to defer implementation of architectural elements until the features requiring these elements is being developed.

As we discover that architecture not only can evolve, but in fact, in any deployed system, the architecture will evolve over time, the view that architecture must be fixed early in the development process becomes a liability. This is a view that creates architectures that are not tolerant of the change they inevitably must undergo. Once we accept that software architectures must be designed to be change-tolerant, the barrier to early deployment of high value features is lowered.

Returning to the currency trading example, we see that early deployment of marketable features is a compelling strategy for increasing return and reducing risk. At the same time the stage is set for a better understanding of the real system requirements and improved collaboration between developers and their customers.

Reference
Software by Numbers: Low-Risk, High-Return Development by Mark Denne, Jane Cleland-Huang, Prentice Hall, 2004

Screen Beans Art, © A Bit Better Corporation

Principles of Lean Thinking

Abstract
In the 1980’s, a massive paradigm shift hit factories throughout the US and Europe. Mass production and scientific management techniques from the early 1900’s were questioned as Japanese manufacturing companies demonstrated that ‘Just-in-Time’ was a better paradigm. The widely adopted Japanese manufacturing concepts came to be known as ‘lean production’. In time, the abstractions behind lean production spread to logistics, and from there to the military, to construction, and to the service industry. As it turns out, principles of lean thinking are universal and have been applied successfully across many disciplines.

Lean principles have proven not only to be universal, but to be universally successful at improving results. When appropriately applied, lean thinking is a well-understood and well-tested platform upon which to build agile software development practices.

Introduction
Call a doctor for a routine appointment and chances are it will be scheduled a few weeks later. But one large HMO in Minnesota schedules almost all patients within a day or two of their call, for just about any kind of medical service. A while ago, this HMO decided to worked off their schedule backlogs by extending their hours, and then vary their hours slightly from week to week to keep the backlog to about a day. True, the doctors don’t have the comforting weeks-long list of scheduled patients, but in fact, they see just as many patients for the same reasons as they did before. The patients are much happier, and doctors detect medical problems far earlier than they used to.

The idea of delivering packages overnight was novel when Federal Express was started in 1971. In 1983, a new company called Lens Crafters changed the basis of competition in the eyeglasses industry by assembling prescription glasses in an hour. The concept of shipping products the same day they were ordered was a breakthrough concept when LL Bean upgraded its distribution system in the late 1980’s. Southwest Airlines, one of the few profitable airlines these days, saves a lot of money with its unorthodox method of assigning seats as people arrive at the airport. Dell maintains profitability in a cutthroat market by manufacturing to order in less than a week. Another Austin company builds custom homes in 30 days.

The common denominator behind these and many other industry-rattling success stories is lean thinking. Lean thinking looks at the value chain and asks: How can things be structured so that the enterprise does nothing but add value, and does that as rapidly as possible? All the intermediate steps, all the intermediate time and all the intermediate people are eliminated. All that’s left are the time, the people and the activities that add value for the customer.

Origins of Lean Thinking
Lean thinking got its name from a 1990’s best seller called The Machine That Changed the World : The Story of Lean Production[1]. This book chronicles the movement of automobile manufacturing from craft production to mass production to lean production. It tells the story of how Henry Ford standardized automobile parts and assembly techniques, so that low skilled workers and specialized machines could make cheap cars for the masses. The book goes on to describe how mass production provided cheaper cars than the craft production, but resulted an explosion of indirect labor: production planning, engineering, and management. Then the book explains how a small company set its sights on manufacturing cars for Japan, but it could not afford the enormous investment in single purpose machines that seemed to be required. Nor could it afford the inventory or large amount of indirect labor that seemed necessary for mass production. So it invented a better way to do things, using very low inventory and moving decision-making to production workers. Now this small company has grown into a large company, and the Toyota Production System has become known as ‘lean production’.

“The mass-producer uses narrowly skilled professionals to design products make by unskilled or semiskilled workers tending expensive, single-purpose machines. These churn out standardized products at high volume. Because the machinery costs so much and is so intolerant of disruption, the mass-producer adds many buffers – extra supplies, extra workers, and extra space – to assure smooth production…. The result: The customer gets lower costs but at the expense of variety and by means of work methods that most employees find boring and dispiriting.”[2]

Think of the centralized eyeglasses laboratory. Remember that Sears used to take two or three weeks to fill orders from its once-popular catalog. Recall the long distribution channel that used to be standard in the computer market. Think dinosaurs. Centralized equipment, huge distribution centers and lengthy distribution channels were created to realize economies of scale. They are the side effects of mass-production, passed on to other industries. What people tend to overlook is that mass-production creates a tremendous amount of work that does not directly add value. Shipping eyeglasses to a factory for one hour of processing adds more handling time by far than the processing time to make the glasses. Adding retail distribution to the cutthroat personal computer industry means that a manufacturer needs six weeks to respond to changing technology, instead of six days. Sears’ practice of building an inventory of mail orders to fill meant keeping track of stacks of orders, not to mention responding to innumerable order status queries and constant order changes.

“The lean producer, by contrast, combines the advantages of craft and mass production, while avoiding the high cost of the former and the rigidity of the later… Lean production is ‘lean’ because it uses less of everything compared with mass production – half the human effort in the factory, half the manufacturing space, half the investment in tools, half the engineering hours to develop a new product in half the time. Also, it requires keeping far less than half the inventory on site, results in many fewer defects, and produces a greater and ever growing variety of products.”[3]

While on a tour of a large customer, Michael Dell saw technicians customizing new Dell computers with their company’s ‘standard’ hardware and software. “Do you think you guys could do this for me?” his host asked. Without missing a beat, Dell replied, “Absolutely, we’d love to do that.”[4] Within a couple of weeks, Dell was shipping computers with factory-installed, customer-specific hardware and software. What took the customer an hour could be done in the factory in minutes, and furthermore, computers could be shipped directly to end-users rather than making a stop in the corporate IT department. This shortening of the value chain is the essence of lean thinking.

Companies that re-think the value chain and find ways to provide what their customers value with significantly fewer resources than their competitors can develop an unassailable competitive advantage. Sometimes competitors are simply not able to deliver the new value proposition. (Many have tired to copy Dell; few have succeeded.) Sometimes competitors do not care to copy a new concept. (Southwest Airlines has not changed the industry’s approach to seat assignments.) Sometimes the industry follows the leader, but it takes time. (Almost all direct merchandise is shipped within a day or two of receiving an order these days, but the Sears catalog has been discontinued.)

Lean Thinking in Software Development
eBay is a company which pretty much invented ‘lean’ trading by eliminating all the unnecessary steps in the trading value chain. In the mid 1990’s, basic eBay software capabilities were developed by responding daily to customer requests for improvements.[5] Customers would send an e-mail to Pierre Omidyar with a suggestion and he would implement the idea on the site that night. The most popular features of eBay, those which create the highest competitive advantage, were created in this manner.

Digital River invented the software download market in the mid 1990’s by focusing on ‘lean’ software delivery. Today Digital River routinely designs and deploys sophisticated web sites for corporate customers in a matter of a weeks, by tying the corporation’s legacy databases to standard front end components customized with a ‘look and feel’ specific to each customer.

In the mid 1990’s, Microsoft implemented corporate-wide financial, purchasing and human resource packages linked to data warehouses which can be accessed via web front-ends. Each was implemented by “a handful of seasoned IT and functional experts… (who got) the job done in the time it takes a … committee to decide on its goals.”[6]

In each of these examples, the focus of software development was on rapid response to an identified need. Mechanisms were put in place to dramatically shorten the time from problem recognition to software solution. You might call it ‘Just-in-Time’ software development.

The question is – why isn’t all software developed quickly? The answer is – rapid development must be considered important before it becomes a reality. Once speed becomes a value, a paradigm shift has to take place, changing software development practices from the mass production paradigm to lean thinking.
If your company writes reams of requirements documents (equivalent to inventory), spends hours upon hours tracking change control (equivalent to order tracking), and has an office which defines and monitors the software development process (equivalent to industrial engineering), you are operating with mass-production paradigms. Think ‘lean’ and you will find a better way.

Basic Principles of Lean Development
There are four basic principles of lean thinking which are most relevant to software development:

Add Nothing But Value (Eliminate Waste)
The first step in lean thinking is to understand what value is and what activities and resources are absolutely necessary to create that value. Once this is understood, everything else is waste. Since no one wants to consider what they do as waste, the job of determining what value is and what adds value is something that needs to be done at a fairly high level. Let’s say you are developing order tracking software. It seems like it would be very important for a customer to know the status of their order, so this would certainly add customer value. But actually, if the order is in house for less than 24 hours, the only order status that is necessary is to inform the customer that the order was received, and then that it has shipped, and let them know the shipping tracking number. Better yet, if the order can be fulfilled by downloading it on the Web, there really isn’t any order status necessary at all.

To develop breakthroughs with lean thinking, the first step is learning to see waste. If something does not directly add value, it is waste. If there is a way to do without it, it is waste. Taiichi Ohno, the mastermind of the Toyota Production System, identified seven types of manufacturing waste:

Here is how I would translate the seven wastes of manufacturing to software development:

 
Extreme Programming (XP) is a set of practices which focuses on rapid software development. It is interesting to examine how XP works to eliminate the seven wastes of software development:

‘Do It Right The First Time’
XP advocates developing software for the current need, and as more ‘stories’ (requirements) are added, the design should be ‘refactored’ to accommodate the new stories. Is it waste to refactor software? Shouldn’t developers “Do It Right the First Time?”

It is instructive to explore the origins of the slogan “Do It Right the First Time.” In the 1980’s it was very difficult to change a mass-production plant to lean production, because in mass production, workers were not expected to take responsibility for the quality of the product. To change this, the management structure of the plant had to change. “Workers respond only when there exists some sense of reciprocal obligation, a sense that management actually values skilled workers, … and is willing to delegate responsibility to [them].” [7] The slogan “Do It Right the First Time” encouraged workers to feel responsible for the products moving down the line, and encourage them to stop the line and troubleshoot problems when and where they occurred.

In the software industry, the same slogan “Do It Right the First Time,” has been misused as an excuse to apply mass-production thinking, not lean thinking to software development. Under this slogan, responsibility has been taken away from the developers who add value, which is exactly the opposite of its intended effect. “Do It Right the First Time” has been used as an excuse to insert reams of paperwork and armies of analysts and designers between the customer and the developer. In fact, the slogan is only properly applied if it gives developers more, not less, involvement in the results of their work.

A more appropriate translation of such slogans as “Zero Defects” and “Do It Right the First Time” would be “Test First”. In other words, don’t code unless you understand what the code is supposed to do and have a way to determine whether the code works. A good knowledge of the domain coupled with short build cycles and automated testing constitute the proper way for software developers to “Do It Right the First Time”.

Center On The People Who Add Value
Almost every organization claims it’s people are important, but if they truly center on those who add value, they would be able to say:

In mass-production, tasks are structured so that low skilled or unskilled workers can easily do the repetitive work, but engineers and managers are responsible for production. Workers are not allowed to modify or stop the line, because the focus is to maintain volume. One of the results of mass-production is that unskilled workers have no incentive to volunteer information about problems with the manufacturing line or ways to improve the process. Maladjusted parts get fixed at the end of the line; a poor die or improperly maintained tool is management’s problem. Workers are neither trained nor encouraged to worry about such things.

“The truly lean plant has two key organizational features: It transfers the maximum number of tasks and responsibilities to those workers actually adding value to the car on the line, and it has in place a system for detecting defects that quickly traces every problem, once discovered, to its ultimate cause.” [8]  Similarly in any lean enterprise, the focus is on the people who add value. In lean enterprises, traditional organizational structures give way to new team-oriented organizations which are centered on the flow of value, not on functional expertise.

The first experiment Taiichi Ohno undertook in developing lean production was to figure out a way to allow massive, single-purpose stamping machines to stamp out multiple parts. Formerly, it took skilled machinists hours, if not days, to change dies from one part to another. Therefore, mass production plants had many single purpose stamping machines in which the dies were almost never changed. Volume, space, and financing were not available in Japan to support such massive machines, so Ohno set about devising simple methods to change the stamping dies in minutes instead of hours. This would allow many parts of a car to be made on the same line with the same equipment. Since the workers had nothing else to do while the die was being changed, they also did the die changing, and in fact, the stamping room workers were involved in developing the methods of rapid die changeover.

Ohno transferred most of the work being done by engineers and managers in mass-production plants to the production workers. He grouped workers in small teams and trained the teams to do their own industrial engineering. Workers were encouraged to stop the line if anything went wrong, (a management job in mass-production). Before the line was re-started, the workers were expected to search for the root cause of the problem and resolve it. At first the line was stopped often, which would have been a disaster at a mass-production plant. But eventually the line ran with very few problems, because the assembly workers felt responsible to find, expose, and resolve problems as they occurred.

It is sometimes thought that a benefit of good software engineering is to allow low skilled programmers to produce code while a few high skilled architects and designers do the critical thinking. With this in mind, a project is often divided into requirements gathering, analysis, design, coding, testing, and so on, with decreasing skill presumably required at each step. A ‘standard process’ is developed for each step, so that low-skilled programmers, for example, can translate design into code simply by following the process.

This kind of thinking comes from mass-production, where skilled industrial engineers are expected to design production work for unskilled laborers. It is the antithesis of lean thinking and devalues the skills of the developers who actually write the code as surely as industrial engineers telling laborers how to do their jobs devalues the skills of production workers.

Centering on the people who add value means upgrading the skills of developers through training and apprenticeships. It means forming teams that design their own processes and address complete problems. It means that staff groups and managers exist to support developers, not to tell them what to do.

Flow Value From Demand (Delay Commitment)
The idea of flow is fundamental to lean production. If you do nothing but add value, then you should add the value in as rapid a flow as possible. If this is not the case, then waste builds up in the form of inventory or transportation or extra steps or wasted motion. The idea that flow should be ‘pulled’ from demand is also fundamental to lean production. ‘Pull’ means that nothing is done unless and until an upstream process requires it. The effect of ‘pull’ is that production is not based on forecast; commitment is delayed until demand is present to indicate what the customer really wants.

Pulling from demand can be one of the easiest ways to implement lean principles, as LL Bean and Lens Crafters and Dell found out. The idea is to fill each customer order immediately. In mass-production days, filling orders immediately meant building up lots of inventory in anticipation of customer orders. Lean production changes that. The idea is to be able to make the product so fast that it can be made to order. True, Dell and Lens Crafters and LL Bean and Toyota have to have some inventory of sub-assemblies waiting to be turned into a finished product at a moments notice. But it’s amazing how little inventory is necessary, if the process to replenish the inventory is also lean. A truly lean distribution channel only works with a really lean supply chain coupled to very lean manufacturing.

The “batch and queue” habit is very hard to break. It seems counterintuitive that doing a little bit at a time at the last possible moment will give faster, better, cheaper results. But anyone designing a control system knows that a short feedback loop is far more effective at maintaining control of a process than a long loop. The problem with batches and queues is that they hide problems. The idea of lean production is to expose problems as soon as they arise, so they can be corrected immediately. It may seem that lean systems are fragile, because they have no padding. But in fact, lean systems are quite robust, because they don’t hide unknown, lurking problems and they don’t pretend they can forecast the future.

In Lean Software Development, the idea is to maximize the flow of information and delivered value. As in lean production, maximizing flow does not mean automation. Instead, it means limiting what has to be transferred, and transferring that as few times as possible over the shortest distance with the widest communication bandwidth as late as is possible. Handing off reams of frozen documentation from one function to the next is a mass-production mentality. In Lean Software Development, the idea is to eliminate as many documents and handoffs as possible. Documents which are not useful to the customer are replaced with automated tests. These tests assure that customer value is delivered both initially and in the future when the inevitable changes are needed.

In addition to rapid, Just-in-Time information flow, Lean Software Development means rapid, Just-in-Time delivery of value. In manufacturing, the key to achieving rapid delivery is to manufacture in small batches pulled by a customer order. Similarly in software development, the key to rapid delivery is to divide the problem into small batches (increments) pulled by a customer story and customer test. The single most effective mechanism for implementing lean production is adopting Just-in-Time, pull-from-demand flow. Similarly, the single most effective mechanism for implementing Lean Development is delivering increments of real business value in short time-boxes.

In Lean Software Development, the goal is to eliminate as many documents and handoffs as possible. The emphasis is to pair a skilled development team with a skilled customer team and give them the responsibility and authority to develop the system in small, rapid increments, driven by customer priority and feedback.

Optimize across Organizations
Quite often, the biggest barrier to adopting lean practices is organizational. As products move from one department to another, a big gap often develops, especially if each department has its own set of performance measurements that are unrelated to the performance measurements of neighboring departments.
For example, let’s say that the ultimate performance measurement of a stamping room is machine productivity. This measurement motivates the stamping room to build up mounds of inventory to keep the machines running at top productivity. It does not matter that the inventory has been shown to degrade the overall performance of the organization. As long as the stamping room is measured primarily on machine productivity, it will build inventory. This is what is known as a sub-optimizing measurement, because it creates behavior which creates local optimization at the expense of overall optimization.

Sub-optimizing measurements are very common, and overall optimization is virtually impossible when they are in place. One of the biggest sub-optimizing measurements in software development occurs when project managers measured on earned value. Earned value is the cost initially estimated for the tasks which have been completed. The idea is that you had better not have spent any more than you estimated. The problem is, this requires a project manager to build up an inventory of task descriptions and estimates. Just as excess inventory in the stamping room slows down production and degrades over time, the inventory of tasks required for earned value calculations gets in the way of delivering true business value and also degrades over time. Nevertheless, if there is an earned value measurement in place, project tasks are specified and estimated, and earned value is measured. When it comes to a choice between delivering business value or earned value (and it often does), earned value usually wins out.

To avoid these problems, lean organizations are usually structured around teams that maintain responsibility for overall business value, rather than intermediate measurements such as their ability to speculate and pad estimates. Another approach is to foster a keen awareness that the downstream department is a customer, and satisfying this internal customer is the ultimate performance measurement.

The paradigm shift that is required with lean thinking is often hindered if the organization is not structured around the flow of value and focused on helping the customer pull value from the enterprise. For this reason, software development teams are best structured around delivering increments of business value, with all the necessary skills on the same team (eg. customer understanding / domain knowledge, architecture / design, system development, database administration, testing, system administration, etc.).

Software Development Contracts
Flow along the value stream is particularly difficult when multiple companies are involved. Many times I have heard the lament: “Everything you say makes sense, but it is impossible to implement in our environment, because we work under contracts with other organizations.” Indeed, the typical software development contract can be the ultimate sub-optimizing mechanism. Standard software contracts and supplier management practices have a tendency to interfere with many lean principles.

Manufacturing organizations used to have the same problem. For example, US automotive companies once believed the best way to reduce the cost of parts in an automobile was with annual competitive bidding. If the only thing that is important is cheap parts, competitive bidding may seem like the best way to achieve this goal. However, if overall company performance is more important, then better parts which integrate more effectively with the overall vehicle are more valuable. In fact, there is an direct correlation between an automotive company’s profitability and its degree of collaboration with suppliers.[9] When Chrysler moved from opportunistic to collaborative relationships with its suppliers in the late 1990’s, it’s performance improved significantly.

The software industry has some lessons to learn in the area of contractual agreements between organizations. It needs to learn how to structure collaborative relationships which maximize the overall results of both parties. A key lesson the software industry needs to learn is how to structure contracts for incremental deliveries that are not pre-defined in the contract, yet assure the customer of prompt delivery of business value appropriate to their investment. Here again, we can learn from lean production.

Lean manufacturing organizations develop a limited number of relationships with ‘trusted’ suppliers, and in turn, gain the ‘trust’ of these suppliers. What does ‘trust’ mean? “Trust [is] one party’s confidence that the other party in the exchange relationship will fulfill its promises and commitments and will not exploit its vulnerabilities.”[10] “…trust…[is] not based on greater interpersonal trust, but rather greater trust in the fairness, stability, and predictability of [the company’s] routines and processes.”[11]

It has been the practice of legal departments writing software contracts to put into contractual language all of the protections necessary to keep the other side ‘honest.’ However, the transaction costs associated with creating and monitoring such contracts are enormous. Many contracts all but demand a waterfall process, even if both companies believe this is not the best approach. It’s time that the software development industry learned the lesson of Supply Chain Management – “Extraordinary productivity gains in the production network or value chain are possible when companies are willing to collaborate in unique ways, often achieving competitive advantage by sharing resources, knowledge, and assets…. Today competition occurs between value chains and not simply between companies.”[12]

Summary and Conclusion
The lean production metaphor is a good one for software development, if it is applied in keeping with the underlying spirit of lean thinking. In the past, the application of some manufacturing concepts to software development (‘Do It Right the First Time’ comes to mind) may have lacked a deep understanding of what makes lean principles work. The underlying principles of eliminating waste, empowering front line workers, responding immediately to customer requests, and optimizing across the value chain are fundamental to lean thinking. When applied to software development, these concepts provide a broad framework for improving software development.

References

[1]The Machine That Changed the World : The Story of Lean Production, by Womack, James P., Daniel T. Jones, and Daniel Roos, New York: Rawson and Associates; 1990.

[2]The Machine That Changed the World : The Story of Lean Production, by Womack, James P., Daniel T. Jones, and Daniel Roos, New York: Rawson and Associates; 1990.

[3]The Machine That Changed the World : The Story of Lean Production, by Womack, James P., Daniel T. Jones, and Daniel Roos, New York: Rawson and Associates; 1990.

[4]Direct from Dell, by Michael Dell with Catherine Fredman, Harper Business, 1999, p 159.

[5]Q&A with eBay's Pierre Omidyar, Business Week Online, December 3, 2001.

[6]Inside Microsoft: Balancing Creativity and Discipline, Herbold, Robert J.; Harvard Business Review, January 2002.

[7]The Machine That Changed the World : The Story of Lean Production, by Womack, James P., Daniel T. Jones, and Daniel Roos, New York: Rawson and Associates; 1990.
 
[8]The Machine That Changed the World : The Story of Lean Production, by Womack, James P., Daniel T. Jones, and Daniel Roos, New York: Rawson and Associates; 1990.

[9] Collaborative Advantage, by Jeffrey H. Dyer, Oxford University Press; 2000

[10] Collaborative Advantage, by Jeffrey H. Dyer, Oxford University Press; 2000

[11] Collaborative Advantage, by Jeffrey H. Dyer, Oxford University Press; 2000

[12] Collaborative Advantage, by Jeffrey H. Dyer, Oxford University Press; 2000

Published in OOPSLA Onward! - November 2002.

Screen Beans Art, © A Bit Better Corporation

Righteous Contracts

Right"eous a. Doing that which is right; yielding to all their due; just; equitable.
[Webster’s Revised Unabridged Dictionary, 1913]

Righteous contracts. A good name for contracts whose purpose is to assure that the parties act in a just and equitable manner and yield to the other their due. Righteous contracts are those governing investments in specialized assets – assets which are very important to a business, but have no value anywhere else. For example, software developed specifically for a single company is a specialized asset, since it is useful only by the company for which it was developed. Agreements to develop specialized assets create a bilateral monopoly; that is, once the parties start working together, they have little option but to continue working together. This bilateral monopoly provides an ideal environment for opportunistic behavior on the part of both supplier and customer.

Thus the purpose of righteous contracts is to prevent opportunistic behavior, to keep one party from taking advantage of another when market forces are not in a position to do so. In a free market where there are several competent competitors, market forces control opportunism. This works for standard or commodity components, but not for specialized assets.

The traditional way to develop specialized assets has been to keep the work inside a vertical organization, where opportunism is controlled by administration. Inside a company, local optimization would presumably be prevented by someone positioned to arbitrate between departments for the overall good of the enterprise. Vertical integration allows a company to deal with uncertainty and change in a rapid and adaptive manner.

Outsourcing
Recently, however, outsourcing has become common in many companies, for very good reasons. An outside company may have lower labor costs or more specialized experience in an area that is not one of the firms core competencies. The cost of producing a service or asset can be considerably lower in an outside company. Of course, there are transaction costs associated with outsourcing, and the total cost (production costs plus transaction costs) must be lower, or vertical integration would make more sense.

Transaction costs associated with outsourcing include the cost of selecting potential suppliers, negotiating and renegotiating agreements, monitoring and enforcing the agreement, billing and tracking payments. Transaction costs also include inventory and transportation above that needed for vertical integration. In addition, there are risks associated with outsourcing, which may result in additional costs. One cost would be that of diminished communication. For example, development of any kind usually requires intense communication between various technical specialties and target users. If distance or intellectual property issues reduce the communication, it will cost more to develop the asset and the results may suffer as well. In addition, moving a specialized skill outside the company may incur lost opportunity costs.

There are two types of contracts which are used for developing specialized assets – Contracts which are executed before the development is done by the supplier, and contracts which are executed after the supplier does the work. A contract executed before work is done is known as a before-the-fact (or ex ante) contract. There are two types of before-the-fact contracts – fixed price contracts and flexible (time-and-materials) contracts. Once these contracts are executed, they set up a bilateral monopoly, fraught with opportunities for exploitation on one side or the other. Therefore, the purpose of these contract is to set up control systems to prevent exploitation.

A contract executed after work is done is called an after-the-fact (or ex post) contract. Suppose a supplier develops a system that it thinks a customer will find valuable and then tries to sell the system. In this case, control comes after the fact; the supplier makes its own decisions, and it’s reward is based on the results. Of course this is a risky proposition, so the supplier has to hedge its bets. One way to do this is to sell the system to multiple customers, basically making it into a commodity product. But this doesn’t help a company that wants suppliers to develop proprietary components for them. In order to entice suppliers to develop specialized assets prior to a contract, a company usually sets up a sole source or favored source program. If a company treats its favored suppliers well, the suppliers develop confidence that their investments will be rewarded and continue to make investments.

On the surface, after-the-fact contracts may seem implausible for software development, but in fact, they are the best solution for contracting a development project. Moreover, the best kind of development processes to use inside a company are those that mimic after-the-fact contracts. How can this be? The explanation starts by understanding why before-the-fact contracts provide poor governance for development projects.

Fixed-Price Contracts
Let’s examine the most commonly used before-the-fact contract, the fixed price contract. A key motivator for fixed price contracts is the desire of a customer to transfer risk to the supplier. This may work for simple, well-defined problems, but it is inappropriate for wicked problems.[1] If the project is complex or uncertain, a fixed price contract transfers a very high risk to the supplier. If the supplier is not equipped to deal with this risk, it will come back to haunt the customer.

Risk should be born by the party best able to manage it. If a problem is technically complex, then the supplier is most likely to be in a position to manage it. If a problem is uncertain or changing, then the customer is in the best position to manage it. Transferring the risk for such problems to the supplier is not only unfair, it is also unwise. There is no such thing as a win-loose contract. If a supplier is trapped on the wrong side of a win-loose contract, the bilateral monopoly which has been formed will trap the customer as well. Both sides loose in the end.

Fixed price contracts do not usually lower cost, because there is always at least some risk in estimating the cost. If the supplier is competent, it will include this risk in the bid. If the supplier does not understand the complexity of the problem, it is likely to underbid. The process of selecting a supplier for a fixed-price contract has a tendency to favor the most optimistic (or the most desperate) supplier. Consequently, the supplier least likely to understand the project’s complexity is most likely to be selected. Thus fixed price contracts tend to select the supplier most likely to get in trouble.

Therefore it is quite common for the customer find a supplier unable to deliver on a fixed price contract. Because the customer no longer has the option to choose another supplier, they must often come to the rescue of the supplier. Alternatively, the supplier might be able to cover its loss, but most likely it will attempt to make the loss up through change orders which add more revenue to the contract. This leads the customer to aggressively avoid any change to the contract. Faced with no other way to recoup the loss, a supplier will be motivated to find ways to deliver less than the customer really wants, either by lowering the quality or reducing the features.

The customer using fixed price contracts to transfer responsibility and risk will often find both back on their plates in the end, and if so, they will be worse off because of it.

Flexible Contracts
“Customers should prefer flexible-price contracts to fixed-price contracts where it is cheaper for the customer to deal with uncertainty than it is for the contractor to do so or where the customer is more concerned with the ability of the contractor to provide a product that works than with price,” writes Fred Thompson in the Handbook of Public Administration. (Second Edition), Rabin, Hildreth, Miller, editors, New York: Marcel Dekker, Inc., 1998.

The flexible-price contract is designed to deal with uncertainty and complexity, but it does not do away with risk, it simply shifts it from the supplier to the customer. For example, after the DOD (U.S. Department of Defense) experienced some very high profile bailouts on fixed price contracts, it began to use more flexible-price contracts is situations where the government was better able to manage the risk. Of course, with the risk transferred to the customer, the supplier has little incentive to contain costs in a flexible-price contract, a point that did not escape contract negotiators at DOD. In order to protect the public interest, DOD perfected controls imposed on the supplier.

Controlling suppliers of flexible-price contracts evolved into a discipline called project management. The waterfall lifecycle grew out of military contracts, and an early focus of PMI (Project Management Institute) was DOD contracts. Companies with DOD contracts not only hire administrators to oversee compliance with contract requirements, they also add accountants to sort out allowable and unallowable costs. Flexible-price contracts invariably have high transaction costs, due to the high cost of control.

Controls Do Not Add Value
High transaction costs would be reasonable if they added value, but in fact, transaction costs are by definition non-value-adding costs. Fred Thompson (Ibid.) notes, “Controls contribute nothing of positive value; their singular purpose lies in helping us to avoid waste. To the extent that they do what they are supposed to do, they can generate substantial savings. But it must be recognized that controls are themselves very costly.”

One way to avoid the high cost of control in flexible-price contracts is not to use them. It may be better to do development internally, where it is easier to deal with uncertainty and respond to change. The question is, on what basis should an outsourcing decision be made? Thompson (Ibid.) counsels, “The choice of institutional design should depend upon minimizing the sum of production costs and transactions costs.” He also notes, “Vertical integration occurs because it permits transaction or control costs to be minimized.”

An interesting problem with this equation is that vertical integration does not always work to minimize control costs. In fact, many organizations find themselves using DOD-like project management controls internally. It seems incongruous that control mechanisms which add cost but not value, and which were invented to prevent opportunistic behavior, would come to dominate development in the very place where they should not be needed. If the reason to develop internally is to provide flexibility in the face of uncertainty, then costly, change-resistant control practices are inappropriate. Traditional project control practices (that freeze requirements, require approval for changes, and track tasks instead of features) have a tendency to create waste, not value, when used inside a company.

After-the-fact Contracts
Let’s assume for the sake of argument that the choice has been made to outsource a complex, specialized development effort. The next question is, how can transaction costs be reduced? In the manufacturing industry, this is done with after-the-fact contracts.

Despite the obvious risks, is not uncommon for suppliers to develop specialized components for a manufacturer prior to obtaining a contract. For example, 3M Optical Systems Division used to develop optically precise lenses for specialized automotive taillights. The reward was a one year contract for a specific model. Unfortunately, after the first year, the automotive company would invariably find a cheaper way to make a similar lens, and Optical Systems would loose the business before it had recovered its investment. The division eventually decided that after-the-fact contracts with Detroit automakers were not profitable and left the business.

There are ways to make after-the-fact contracts work better. Toyota awards contracts for an entire run of a model, and uses target costing to manage costs. Thus a supplier knows that if it wins the business, it can recover its investment, while the customer is confident that the supplier will work reduce costs in line with its economic requirements. In addition, the supplier understands that it will receive favored consideration for similar components in the future.

After-the-fact contracts require two elements to work: shared risk and trust. Toyota shares the risk with a component supplier by guaranteeing the business over the life of an model. Both parties agree to work together to try to meet a target cost profile over the life of the agreement. Note that meeting future target costs is neither guaranteed nor is it the sole responsibility of the supplier. In the best relationships, technical personnel from each company work freely together without worrying about proprietary information, both to meet target costs and to develop new components not yet subject to a contract.

If both parties are pleased with the results of the first contract, they develop trust and a good working relationship, and are more likely continue to do business together. The supplier is inclined to risk more in developing new components when it has developed confidence that the investment will pay off. This kind of relationship can achieve all of the benefits of both outsourcing and vertical integration combined.

But Software is Different…
You might be saying to yourself, this is fine if there is something to be manufactured and sold many times over, like a taillight, but in software we develop a system only once, it is complex and expensive, it is subject to many changes, and if it is not properly designed and executed, huge waste might result. Where is the parallel to THIS in manufacturing?

Consider the large and expensive metal dies which stamp out vehicle body panels. The cost of developing dies can account for half of a new model’s capital investment. Consequently, a great deal of time is spent in all automotive companies working to minimize the cost of these dies. The approach in Japan is distinctly different from that in the U.S., and dramatically more effective. The best Japanese companies develop stamping dies for half the cost and in half the time as their counterparts in the West. The resulting Japanese dies will be able to stamp out a body panel in 70% of the time needed by U.S. stamping operations.

From the classic book Product Development Performance by Clark and Fujimoto, Harvard Business School Press, 1991:

Japan firms use an ‘early design, early cut’ approach, while U.S. practice is essentially “wait to design,, wait to cut.”

Because it entails making resource commitments while the body design is still subject to frequent changes, the Japanese early design, early cut approach entails significant risks of waste and duplication of resources…. Many engineering changes occur after final release of blueprints. At peak, hundreds of changes are ordered per month.

Behind the wait to design, wait to cut approach in U.S. projects is a desire to avoid expensive die rework and scrappage, which we would expect to be an inevitable consequence of the bold overlapping that characterizes the Japanese projects. However, our study revealed a quite different reality. U.S. firms, despite their conservative approach to overlapping, were spending more on engineering changes than Japanese firms. U.S. car makers reported spending as much as 30-50 percent of original die cost on rework due to engineering changes, compared to a 10-20 percent margin allowed for engineering changes by Japanese products.

The Japanese cost advantage comes not from lower wages or lower material prices, but from fundamental differences in the attitudes of designers and tool and die makers toward changes and the way changes are implemented…. In Japan, when a die is expected to exceed its cost target, die engineers and tool makers work to find ways to compensate in other areas…. Die shops in high-performing companies develop know-how techniques for absorbing engineering changes at minimum cost…. In the United States, by contrast, engineering changes have been viewed as profit opportunities by tool makers….

Suppose a body engineer decides to change the design of a panel to strengthen body-shell rigidity. The high performers tend to move quickly. The body designer immediately instructs the die shop to stop cutting the die on the milling machine. Without paperwork or formal approval, the body designer goes directly to the die shop, discusses modifications with the die engineers, checks production feasibility, and makes the agree-upon changes on the spot. Unless the changes are major, decisions are made at the working level. Traditionally, the die shop simply resumes working on the same die. Paperwork is completed after the change has been made and submitted to supervisors for approval. The cost incurred by the change is also negotiated after the fact. The attitude is “change now, negotiate later.

In companies in which die development takes a long time and changes are expensive, the engineering change process is quite different. Consider the context in which changes occur. In extreme versions of the traditional U.S. system, tool and die makers are selected in a competitive bidding process that treats ‘outside’ tool shops as providers of a commodity service. The relationship with the die maker is managed by the purchasing department, with communication taking place through intermediaries and drawings. The individuals who design the dies and body panels never interact directly whit the people who make the dies.

You would think that tool and die makers in Japan must be a department inside the automotive company. How else could it be possible for a designer to walk into a tool and die shop, stop the milling, make changes, and start up the milling again, leaving approvals and cost negotiations for later? But this is not the case. Tool and die makers are supplier companies in Japan, just as they are in the U.S. The difference lies in the attitudes of the different countries toward supplier contracts.

For Toyota in particular, a supplier is a partner. The basis of this partnership is a target cost for each area of the car. This translates into target costs for all development activities, including dies. Of course, U.S. companies have target costs for each component also, but they tend to impose the cost on the supplier without regard to feasibility. This has a tendency to create a win-loose relationship, leaving the supplier no option but to recoup costs through the change process.

In contrast, Toyota does not impose cost targets on suppliers that it does not know how to meet, and engineers from both companies work together to meet target costs. If something goes wrong and the targets cannot be met, Toyota shares the problem in an equitable manner. In this win-win environment, arms-length exchange of information through written documentation and an extensive change approval processes is unnecessary.

The Toyota Production System is founded on the premise that superior results come from eliminating anything which does not add value. Since control systems do not add value, they must be minimized, just like inventory and set-up times. Therefore supplier partnerships based on shared risk and trust are the preferred relationship. The hallmarks of these partnerships are worker-level responsibility for meeting business goals, intense communication at the technical level, a stop-the-line and fix-it-immediately attitude, and an emphasis on speed. Even for large, one-of-a-kind development projects which require highly specialized design, this approach produces dramatically superior results.

Can this work for Software Development?
Developing specialized dies is not that much different than developing specialized software. The key is to establish a partnership relationship which allows true development to take place. Development is done using a repeated cycle of design-build-test, allowing the solution to emerge. The question is, how can a contract be written to support the emergent nature of development?

Neither fixed-price nor flexible-price contracts support the nature of software development. Development always involves tradeoffs, and an organization which facilitates the best tradeoff decisions will produce the best result. Before-the-fact contracts do not support the give-and-take between developers and customers necessary to make the best tradeoffs. A developer should not have to worry about dealing with problems as they arise, but with before-the-fact contracts, this activity has to be paid for by one company or the other. Since every hour must be accounted for, the give-and-take necessary for trade-off decisions is discouraged.

What is needed is a contract approach which allows developers and customers work closely together to develop a business value for a target cost. Examples of how to do this in a vertical organization abound. There many successful examples of using Scrum for product development. Microsoft’s approach to product development is documented by Michael Cusumano in Microsoft Secrets, Simon and Schuster, 1998. The general approach is to set a clear business goal, fix resources, prioritize features, deliver working software in short cycles, and stop working on features when time runs out. This approach has a track record of delivering systems, even large ones, in a predictable timeframe for a predicable cost.

The question is, how can a contract be written to support the same approach? The answer is to move to after-the-fact contracts in which a supplier is paid for the value of the work they do. It works like this: A customer has a clearly defined business value and a target cost in mind for achieving that value. This target cost includes payments to a supplier for their contributions. The customer comes to an agreement with a supplying partner that the business value and the target cost are achievable, including the target cost for the supplier’s participation. Work proceeds without contractual guarantees that the value will be delivered or the target cost will be achieved, but both partners are committed to meet these goals.

Workers at each company use adaptive processes[2] to develop the system as a single team. They communicate intensely at the developer-customer level to make the necessary tradeoffs to achieve the value within the target cost. As working software is delivered, both supplier and customer work together using velocity charts to monitor development progress. If adjustments to the business value or the target cost structure are required, these become apparent early, when they can be addressed by limiting the feature list or extending the schedule. If this changes the business value or target cost, the parties negotiate an equitable way to share the burden or benefit.

Conclusion
Trusted-based partnerships are the first requirement to make after-the-fact contracts work. Partnerships are necessary to facilitate worker-level responsibility for meeting business goals, intense communication between developers and users to make optimal tradeoffs, daily builds and automated testing to facilitate a fix-it-immediately attitude, and a focus on early delivery of working software to create the feedback system critical to good development.

Companies that develop contracts allowing these values to flourish can expect to produce the same dramatically superior results in software development that these values produce in product development.

Lessons for Outsourcers
If your company outsources software development, consider the following:

1. Fixed Price Contracts
Fixed price contracts are risky. There is both a technical risk that the job can’t be done for the allotted cost, and the very real risk that the selection process favors less knowledgeable suppliers. If you assure that you get a competent supplier, then you can be sure the supplier will add a good margin to the cost to cover their risk. Remember that risk should be born by the party most able to manage it, so if the project is complex and changes are likely, you should assume the risk. If the project is a wicked project, you should not even consider a fixed price contract.

If you are considering a fixed price contract, you are probably interested in transferring all responsibility to your supplier. But remember, if this results in a win-loose situation, you will not win. You are going to be committed to the supplier before the cracks begin to show, and if things go wrong, you will suffer as much, if not more, than your supplier. You may have to bail them out. They will no doubt be looking to make up for loses through change orders, so you will have to control these aggressively. That means if your project is prone to uncertainty or change, you really don’t want a fixed price contract.

And finally, it is much more difficult to get what you really need under a fixed price contract. If you got the low bidder, you probably did not get the supplier most familiar with your domain. If the bid was too low, your supplier will want to cut corners. This may mean less testing, fewer features, a clumsy user interface, out-of-date technology. You are going to need to carefully limit user feedback to control changes and keep the price in line, which will make it more difficult to get what your users really want.

Traditional Control Processes
Traditional project management processes tend to emphasize scope management using requirements traceability and an authorization-based change control system. Typically cost control is provided with some variation of an earned value measurement. The first thing to realize is that all of these techniques are expensive and do not add any value to the resulting software. These practices are meant to control opportunism, and if you are concerned that your supplier might take advantage of you, they might make sense. (But try partnerships first.)

You most likely do not want to be using these practices inside your own company; they get in the way of good software development. It’s pretty well known that an iterative approach to software development, with regular user communication and feedback, is far better than the waterfall approach. However, those pesky project management practices tend to favor waterfalls. It’s a good bet that your project will be subject to change (users change their preferences, technology changes, someone forgot a critical requirement), so you want to be using adaptive processes.

2. Trust-based Partnerships
For starters, all internal development should be based on trust-based partnerships – after all, that’s why you are doing inside development in the first place! If you can’t trust someone in your own company, who can you trust?

The fastest, cheapest way to develop software with a supplier is to let their technical people make decisions based on close interaction with and regular guidance from your users. You get the best results and the happiest users this way too. This kind of relationship requires risk sharing and excellent on-going communications. In exchange for this investment, trust-based partnerships adapt well to change and uncertainty and are most likely to yield faster, better, cheaper results.

Lessons for Contractors
If your company supplies software development, consider the following:

1. Fixed Price Contracts
You owe it to your customers to educate them on the pitfalls of fixed price contracts. Make sure they understand that this will make it more difficult for you to deliver the best business value.

2. Traditional Control Processes
Don’t accept traditional control mechanisms; there are better ways. Instead, use prioritized feature sets, rapid iterations and velocity charts to monitor projects.

Never allow the customer to fix cost, schedule and features simultaneously. Preferably, you want to agree to meet cost, schedule and overall business value targets, and provide a variable feature set. If the detailed feature set is not negotiable, then at least one of the other two must be flexible.

Find out what is REALLY important to your customer in terms of business value and deliver that.

3. Trust-based Partnerships
Your top priority when negotiating the relationship is to assure that your development team will have constant user involvement and feedback. You can negotiate what this means and who represents the user, but if you don’t have access to users or a user proxy, you will have a difficult time delivering business value. And delivering business value must be your main objective.
____________________
Footnotes:

[1] A Wicked Problem is one in which each attempt at creating a solution changes the understanding of the problem. See “Wicked Projects” by Mary Poppendieck, Software Development Magazine, May, 2002, posted on this site under the title “Wicked Problems.”

[2] For a discussion of adaptive processes, see “Wicked Projects” by Mary Poppendieck, Software Development Magazine, May, 2002, posted on this site under the title “Wicked Problems.”

Screen Beans Art, © A Bit Better Corporation