When Demand Exceeds Capacity

We are getting solar panels on our roof.  Someday.  We signed a contract last November and the original estimate for installation was May. Since no one does outside work in the winter here in Minnesota, this was about two months after the earliest possible installation date. But May has come and gone, and we still do not have an installation date. Why? The electrical design of for our 40-year-old house is complex and the young solar company has limited electrical design capability. So does our mature electrical utility company. Between the two of them, our project has been held up for a long time. 

Solar systems have become widely popular since the electrical grid meltdown in Texas and the sharp rise in energy prices. So, it is not a surprise that the nascent solar industry is dealing with far more demand than the young solar companies and staid utility companies are equipped to handle. It is difficult to find enough talented people to do the complex system design, project management, and installation of solar systems. Thus, solar customers experience long delays that cost us months of potential savings, while they suffer from months of delayed revenue.  

Our solar company has an eager sales force and a handful of project managers. Once contracts are signed, sales people hand them off to project managers so they can focus on pursuing more sales. Since the rate of signed contracts greatly exceeds the rate of completed installations, the number of customers assigned to each project manager is constantly increasing. It is impossible for a project manager to manage dozens of complex solar projects at the same time, but that is what they are expected to do. 

Clearly, our solar company is unwilling to reduce its sales rate to match its project completion rate and unable to increase the completion rate to match the rate at which contracts are signed. Thus, the length of time a project takes continually increases, putting customers and project managers in a lose-lose situation. In fact, the solar industry in hardly the first industry in the world to experience far more demand that it has the capacity to supply; many other industries have navigated these waters. Typically, companies that do not learn how to manage excess demand end up failing because they generate annoyed customers faster than they generate revenue. Companies that became giants in rapidly growing industries – for example, Airbnb, AWS, Google, Netflix, and SpaceX – learned quickly to increase their delivery capacity to match the rate at which they generated sales. Successful solar companies will have to learn the same thing.

How do companies rapidly increase delivery capacity? Hiring more people might help, but it is not nearly enough. Young companies in a growth industry must figure out how to complete more jobs faster while doing less work or they will not be able to keep up with the rapid increase in demand. There are three essential strategies that successful industry giants have used to increase efficiency which other companies would do well to understand.

1. Focus on Customers Outcomes

The first strategy for increasing efficiency is to pay attention to the metrics that drive behavior. Startups tend to measure success by counting the number sales, while more efficient companies tend to measure the number of satisfied customers. When success is measured by the number of satisfied customers rather than booked revenue, several good things happen. The company learns not to accept work at a faster rate than it can complete work, because doing so does little more than stretch out the time it takes for its people to be successful while lengthening the queue of increasingly impatient customers. Work teams also learn to focus on completing installations as quickly as possible, because more installations will be seen as better performance. Finally, teams are unlikely to develop products that customers are not interested in – a very common source of wasted effort in startup companies.

It is extraordinarily difficult for an ambitious young company to turn away people that are beating a path to its door. Everyone’s instinct is to hire more salespeople and let the work pour in.  But let’s face it, the most important thing that young company needs to learn is how to COMPETE work quickly, not how to ACCEPT work quickly. If it opens the doors wide and lets everyone in, there will be so many customers milling about that workers will be distracted trying to deal with the chaos rather than focused on getting work done. The most important thing for that young company to do at this point is to stop spending so much time on managing excess work and start concentrating on how to get jobs done as fast as they arrive.

Every growth industry has gone through this learning curve. Once upon a time, mail order catalogs (Sears, for example) accepted orders as fast as they could and shipped them in about two weeks. Because of the delay, they had to deal with people changing their mind and a complex order change system was necessary. Then L.L.Bean had a novel idea – why not ship products within 24 hours of receiving the order? It worked – customers were delighted, and no order change system was needed. Today almost all fulfillment systems work on the same principle – focus on building the capacity to ship product at the same rate orders arrive. Once shipping capacity matches demand, the company can ship immediately after an order arrives, delighting customers and reducing congestion while giving customers little time to change their minds.  

2. Manage FLOW, not tasks

The second strategy for increasing efficiency is counterintuitive; it requires a shift in the company’s mental model of how work gets done. Consider our solar company. As a small startup, it thinks of every project as a separate entity and assigns a project manager to sequence the steps in an installation so all necessary work gets done. Unfortunately, this approach is not readily scalable, and even if it were, there are not enough experienced project managers available to hire. Industry leaders do things differently; they manage packages of items as they move through their system, not individual projects or tasks. 

What does it mean to manage groups of items?  Consider the shipping industry.  Until the 1950’s, ships were loaded with individual items packed carefully to keep the ship stable. Each item had to be traced through every port as it moved off the ship and onto another ship or a train or a truck. Then the shipping container was invented, and it was no longer necessary to manage individual items moving through the supply chain. Instead, shippers managed containers as they moved from origin to destination.  

Some years ago, we bought a sofa that was heavily discounted because of a missing headrest. A few months later the store called to say it was filling a container of furniture from their supplier in Norway and they could add our missing headrest for a small fee. It would have been too complex (and expensive) to order and track the headrest individually; but it was simply assigned to the container in which many items moved as a package from Norway to Minnesota. The cost of shipping was negligible. It's easy to see how containers created a reduction in complexity that dramatically reduced the cost of shipping goods and led to substantial changes in the global economy. 

How do you group and manage workflow in a growing solar company?  First, you think of a contract as a number of steps that need to be completed. Certainly, some of the steps can be done in parallel – for example solar panel layout can be done in parallel with electrical design. Other steps have prerequisites that must be completed first. You organize work so that each step has its own responsible team, its own queue, and its own completion rate. As each new contract is signed, it is coded with the various steps it must move through, including any required sequencing. Then you release the contract to the starting step (or steps). When a step is done, work should automatically move to the next step(s) until all work is complete. Your job is to manage the FLOW of work through the system rather that the progress of individual items through their various steps.

How do you manage the flow of work through a system? Start by managing the amount of time a job spends waiting for attention in the queue of each step. Queueing theory is clear: if you limit the length of a queue and send work through in FIFO (first-in-first-out) order, then time in a queue is the number of items in the queue divided by the average completion rate. For example, if you limit the items in a work queue to twenty and each item takes two days, work will wait in the queue – if it is FIFO – for no more than ten days. (Note: If a FIFO queue is not acceptable, then the queue is too long! Make it shorter.) 

Once a job reaches the end of a queue, it should flow quickly through the work step. The best approach is for a responsible team begin a job, focus on it until it is done, then go on to the next job. Focusing on one thing at a time improves the quality of the work, eliminates wasted time due to multitasking and gets everything done faster. (See Single Responsibility Teams, below.) 

If a step in the workflow is completed inside of your company, you have a lot of control over the rate at which work moves through the step (queue time + work time). Your goal is to create short, predictable completion rates for every internal step. If a step occurs outside your company (the utility company, for example, or the city permitting process) you do not have control of the time that takes, but you can measure it – you should track average time and variation. Note that it is important to separate internal steps from external steps – otherwise variations in the external completion rate will make it impossible to control internal rates of completion.

It is also important to ensure that downstream steps have capacity in their queues to handle work as it is completed upstream. If this sounds complicated, an analogy might be helpful: Consider an order coming into Amazon.  Once upon a time, orders were handled by a central database which directed each step in the processing of orders as they moved through the company.  But in the early 2000’s Amazon outgrew this process; they could not buy large enough database computers to keep up with surges in demand. Amazon needed a new approach, so it abandoned the central databases and thought of each order as moving through a series of independent services. When an order is placed, it is coded with an initial chain of necessary services (Do we need to do a credit check? Does the product require special shipping? ...) and sent on its way through the various processing steps. A service can divert the order or add additional steps (Oops, product is out of stock! ...) and downstream processes are expected to have the capacity to accept work from upstream processes. (If the product can be picked from a warehouse, it should be possible to package and ship it immediately.)

Armed with a set of steps each contract needs and the time each step will take, you, like Amazon, should be able to predict the expected installation date very early – ideally when the contract is signed. You should also have a handle on variation due to outside queues and dependencies such as parts availability and supply chain delays. You can send the contract into your workflow, track how well it is coming along and flag exceptions as soon as they occur. 

In addition, you can spot bottlenecks that need to be improved and generally manage the capacity of your organization. You can be alerted when you need to limit sales to avoid overcommitment: Do not accept work faster than the rate at which work moves through your bottleneck process. Focus on the bottleneck until it is improved, when of course a new bottleneck will appear. Find the next bottleneck and limit sales of projects that use it to the rate the bottleneck can handle. Repeat... 

By far the biggest benefit of managing the flow of groups of items is this: You will get a lot more done with a lot less effort. How can this be? When you no longer have half-done work clogging up your system, demanding attention, and distracting workers, everyone can spend much more of their time actually getting useful work done.   

3. Form Single-Responsibility Teams

The third big change you should make is to charter single-responsibility teams, organized and led by single-responsibility leaders. What is a single-responsibility team?  It is a team that has one and only one thing to do, a team that can focus all its efforts on accomplishing one thing at a time and doing that one thing well. Be careful to minimize dependencies among these teams; each team should be able to begin and complete its work independently; teams should not need to coordinate with or get permission from managers or other teams. Finally, work should not appear in the queue of a single-responsibility team until everything is in place for that work to be done, otherwise the team will have to finish someone else’s work before they can focus on their own. 

How is it possible to organize complex work into pieces that can be accomplished by single-responsibility teams? While there is no such thing as the “best” organizational structure for companies in diverse domains, it is easy to find examples that demonstrate the concept. Let’s start with AWS (Amazon Web Services). Over the past decade AWS has replaced most enterprise software companies with an array of independent services, each managed by a single-responsibility team and strung together through hardened interfaces. No one has ever seen an AWS roadmap; instead, teams identify (or are given) a customer problem and they start by writing a “press release” describing how their solution will solve the problem. Then they proceed to develop the solution, testing it with customers as often as possible. AWS has released multiple new enterprise-level services every year since 2012, each aimed at solving a specific customer problem. These services have gradually lured companies away from the complex, integrated packages of traditional vendors, often prompting AWS customers to organize their own work around autonomous, focused teams.

Now let’s look at SpaceX, which is organized very differently than AWS. SpaceX is fundamentally an engineering organization; its teams are organized around the components of booster rockets that launch things into space. There is a team for each stage of the booster (eg. payload, stage 1, stage 2, engine cluster…) with sub-teams for each sub-component of the stage (Merlin engine, landing legs…).  

You might wonder: How do teams know what to do and when to do it? How do they coordinate with other teams? How do they know they have done a good job?  SpaceX operates on the principle of responsibility, which says that each team / sub-team is expected to understand the role its component must play in the next launch and have it ready by the scheduled launch date (perhaps a couple of months away). The team, led by a responsible engineer, has one job: Ensure that the component works as well as possible during the launch and does its job as part of the overall system. 

How does SpaceX manage FLOW? For a large, multi-team development effort involving both hardware and software, managing flow is usually accomplished by scheduling a series of frequent integration events. The purpose of each event is to take a well-defined step forward in the overall development effort to learn what works, locate unintended integration effects, and discover ways to improve the system. A series of integration events sets the cadence for all teams involved in the development effort. In the case of SpaceX a test launch is scheduled every few months throughout a development cycle; importantly, launch dates Do. Not. Change. Each launch is an integration event that moves all teams forward together, creating a steady flow of real progress toward a working system. 

Of necessity, based on the structure of a booster, SpaceX teams have more dependencies than AWS teams. They also have a clear responsibility to understand and account for those dependencies. Since SpaceX teams work to meet fixed launch deadlines, their systems sometimes fail during a launch. Failures are part of the process, but teams must carefully document launch behavior to be sure that the cause of any failure is identified and eliminated. Believe it or not, this approach to development is much faster and less costly (and usually safer!) than trying to think through every failure mode and eliminate any possibility of failure before attempting a test launch. 

Does SpaceX focus on customer outcomes? The company undertook the difficult challenge of developing a reusable booster because it was clear that sending things into space was too expensive for many potential customers. Reusing boosters was an essential step in dramatically reducing the cost of a launch, significantly increasing the number of customers with access to space. This need to reduce launch costs has driven most major booster development decisions at SpaceX.

SpaceX’s reusable booster has reduced the cost of launching a kilo of payload into space by a factor of 7, while spending an order of magnitude less money on development than previous booster rockets.

Summary

A lot of companies have more work than they can possibly handle – and that is not a good thing. It creates long delays, excess complexity, and eventually, dissatisfied customers. The secret is to limit demand to capacity and then increase capacity to meet additional demand by creating a steady flow of work while teams focus on a single responsibility. You also need to stop doing work that does not contribute to great customer outcomes: Stop multitasking, stop prioritizing, get rid of backlogs, stop shepherding individual tasks. If companies as large and successful as AWS and SpaceX can organize work around customer outcomes, steady flow, and focused, single-responsibility teams – then so can you. 

Lessons from Planned Economies

Just as a market economy which relies on the collective actions of intelligent agents gives superior performance in a complex and changing economic environment, so too an agile project leadership approach which leverages the collective intelligence of a development team will give superior performance in a complex and changing business environment. However, conventional project management training focuses on using a plan as the program for action; it does not teach project leaders how to create a software development environment that fosters self-organization and learning. Since very few courses with such a focus are available today, this paper proposes a curriculum for agile software development leaders.

Planned Economies
In the middle of the 20th century, dozens of countries and millions of people believed that central planning was the best way to run their economies. Even today there are many people who can’t quite understand why market economies invariably outperform planned economies; it would seem that at least some of the planned economies should have flourished. After all, there are advantages to centralizing economic decisions: virtually full employment is possible; income can be distributed more equally; central coordination should be more efficient; directing resources into investment should spur growth. So why did planned economies fail?

There are two fundamental problems with planned economies: First, in a complex and changing economic system, it is impossible to plan for everything, so a lot of things fall between the cracks. For instance, planned economies usually suffer a shortage of spare parts, because no one plans for machines to break down. Secondary effects such as environmental impact are often ignored. Furthermore, planners do not have control of the purchase of goods, so they have to guess what consumers really want. Inaccurate forecasts are amplified by a long planning cycle, causing chronic shortages and surpluses.

The second problem with planned economies is diminished incentives for individuals. When compensation does not depend on contribution, there is little to gain from working hard. When incentives are tied to meeting targets, risk adverse managers focus on routine production to meet goals. The stronger that rewards are tied to meeting targets, the more disincentive there is for being creative or catching the things that fall between the cracks.

If we look at conventional software project management, we see practices similar to those used in planned economies, and we also see the similar results. Among projects over $3 million, less than 10% meet the conventional definition of success: on time, on budget, on scope. For projects over $6 million the number drops below 1% [1]. Interestingly, the underlying causes of failure of planned economies are the same things that cause failure in software projects, and further, the remedy is similar in both cases.

The difference between a planned and a market economy is rooted in two different management philosophies: management-as-planning/adhering and management-as-organizing/learning. Management-as-planning/adhering focuses on creating a plan that becomes a blueprint for action, then managing implementation by measuring adherence to the plan. Management-as-organizing/learning focuses on organizing work so that intelligent agents know what to do by looking at the work itself, and improve upon their implementation through a learning process.

The Planning/Adhering Model Of Project Management
Conventional wisdom holds that managing software projects is equivalent to meeting pre-planned cost, schedule and scope targets. The unquestioned dominance of cost, schedule and scope – often to the exclusion of less tangible factors such as usability or realization of purpose – draws heavily on the contract administration roots of project management. Therefore project management training and certification programs tend to focus on the management-as-planning/adherence philosophy. This philosophy has become entrenched because it seems to address two fears: a fear of scope-creep, and a fear that the cost of changes escalates significantly as development progresses.

However, management-as-planning/adherence leads to the same problems with software projects that planned economies suffered: in a complex and changing environment, it is virtually impossible for the plan to cover everything, and measuring adherence to the plan diminishes incentives for creativity and catching the things that fall between the cracks.

In the classic article ‘Managing by Whose Objectives?,’[4] Harry Levinson suggests that the biggest problem with management-by-objectives is that important intangibles which are not measurable fail to get addressed, because they are not in the action plan of any managers. Often these are secondary ‘hygiene’ factors, similar to environmental considerations in a planned economy.

In the book ‘Measuring and Managing Performance in Organizations’[5], Robert Austin makes the same point: over time, people will optimize what is measured and rewarded. Anything which is not part of the measurement plan will fade from importance. Austin points out that managers are often uncomfortable with the idea of not being able to measure everything, so they compensate through one of three techniques:

• Standardization. By creating standards for each step in a development process, it is hoped that all steps in the project can be planned and measured, and nothing will be missed.
• Specification. Specification involves constructing a detailed model of the product and/or process and planning every step in detail.
• Subdivision, functional decomposition. The Work Breakdown Structure (WBS) is the classic example of attempts to decompose a project into steps so that all steps can be planned. 

Conventional project management practices have emphasized all of these techniques to help a project manager be certain that everything is covered in the project plan. However, just as in a planned economy, these techniques are insufficient to catch everything in all but the simplest of projects. In fact, drilling down to detail early in the project has the opposite effect – it tends to create blind spots, not resolve them. By taking a depth-first rather than a breadth-first approach to planning, mistakes and omissions become more likely,[6] and these tend to be more costly because of early investment in details. Thus an early drill-down approach tends to amplify, not reduce, the cost of change.

A management-as-planning/adherence approach also tends to amplify, not reduce, scope-creep. In many software development projects, a majority of the features are seldom or never used.[1] Part of the reason for this is that asking clients at the beginning of a project what features they want, and then preventing them from changing their minds later, creates strong incentives to increase the number of features requested, just in case they are needed. While limiting scope usually provides the best opportunity for reducing software development costs, fixing scope early and controlling it rigidly tends to expand, not reduce scope.

Just as in planned economies, management-as-planning/adhering tends to have unintended consequences that produce precisely the undesirable results that the plans were supposed to prevent. The problem lies not in the planning, which is very useful, but in using the plan as a roadmap for action and measuring performance against the plan.

The Organizing/Learning Model Of Project Management
Market economies deal with the problems of planned economies by depending upon collaborating intelligent agents to make decisions within an economic framework. In market economies, it is the job of the government to organize the economic framework with such things as anti-trust laws and social safety nets. Economic activity is conducted by intelligent agents who learn from experience what is needed and how to fill the needs.

Of course, the economies of individual countries dwarf most software projects, so we might look further to find examples of management-as-organizing/learning. We will explore two domains: manufacturing and product development.

Throughout most of the 20th century, mass production in the US focused on getting things done through central planning and control, reflecting the strong influence of Frederick Taylor’s Scientific Management. The climax came when computer systems made it possible to plan the exact movement of materials and work throughout a plant. Material Requirements Planning (MRP) systems were widely expected to increase manufacturing efficiency in the 1980’s, but in fact, most MRP systems were a failure at detailed production planning. They failed for the same reasons that planned economies failed: the systems could not readily adapt to slight variations in demand or productivity. Thus they created unworkable schedules, which had to be ignored, causing the systems to become ever more unrealistic.

As the centralized MRP planning systems were failing, Just-in-Time systems appeared as a counterpoint to Scientific Management. Just-in-Time forsakes central planning in favor of collaborating teams (intelligent agents). The environment is organized in such a way that the work itself and the neighboring teams signal what needs to be done; rather than a central plan. When problems occur, the root cause is sought out and eliminated, creating an environment in which intelligent agents continually improve the overall system. In almost all manufacturing environments, implementing Just-in-Time trumps any attempt to plan detailed production activities using a MRP system. These systems succeed for the same reason a market economy succeeds: intelligent agents are better at filling in the gaps and adapting to variation than a centrally planned system.

An argument can be made that manufacturing analogies are not appropriate for software development, because manufacturing is repetitive, while projects deal with unique situations. Because of this uniqueness, the argument goes, management as planning/ adhering is the only way to maintain control a design and development environment. A look at product development practices shows that the opposite is true: creating a detailed plan and measuring adherence to that plan is actually a rather ineffective approach a complex product development project.

In the late 1980’s Detroit was shocked to discover that a typical Japanese automotive company could develop a new car in 2/3’s the time for half the cost as a typical US automaker.[7] The difference was that product development in Japan used a concurrent development process, which allows for learning cycles during the design process as well as on-going communication and negotiation among intelligent agents as design proceeds.

Just as market economies routinely outperform planned economies, concurrent development routinely outperforms sequential development. Replacing sequential (plan-up-front) engineering with concurrent (plan-as-you-go) engineering has been credited with reducing product development time by 30-70%, engineering changes by 65-90%, and time to market by 20-90%, while improving quality by 200-600%, and productivity by 20-110%.[8]

Based on experience from other domains, management-as-organizing/learning would appear to have a better chance of resulting successful software projects than the prevailing management-as-planning/adhering approach. An examination of the Agile Manifesto shows that agile software development approaches favor the management-as-organizing/learning philosophy. (See Table 1.) Therefore, we can expect that in the long run, agile software development might significantly outperform traditional software development practices. In fact, evidence is mounting that agile approaches can be very effective. [9]

Table 1. Mapping Values from the Agile Manifesto to Management Philosophies

A Curriculum For Agile Software Project Leadership
Existing training for project management appears to be largely focused on the management-as-planning/adhering philosophy. Courses seem to be aimed at obtaining certification in an approach to project management developed for other domains, such as facilities construction or military procurement. Even courses aimed specifically at software development tend to focus on work breakdown and a front-end-loaded approach to managing scope. As we have seen, this is not a good match for concurrent development or agile software development.

It would seem that at a curriculum should be available for leaders of agile software development projects, given the dismal track record of current approaches and the potential of agile software development. Project managers who know how to develop work breakdown structures and measure earned value often wonder what to do with an agile software development project. Senior managers often wonder how to improve the skills of project leaders. Courses on management-as-organizing/learning are needed to fill this void, but there seem to be few project management courses with this focus. To help make agile project leadership training more widely available, this article outlines a possible curriculum for such courses.

Change the Name: Project Leadership
Since this is new territory, we may as well start with a new name, and move away from the administrative context of the word management. All projects, agile or otherwise, benefit from good leaders; that is, people who set direction, align people, and create a motivating environment. By using the term leadership we distinguish this course from one which focuses on the management tasks of planning, budgeting, staffing, tracking, and controlling.

Setting Direction
Planning is a good thing; the ultimate success of any project depends upon the people who implement a project understanding what constitutes success. Planning becomes brittle with it decomposes the problem too fast and moves too quickly to solutions. The best approach to early planning is to move up one notch and take a broader view, rather than decompose the problem and commit to detail too early.[6] A project leader starts by understanding the purpose of the project and keeping that purpose in front of the development team at all times.

Organizing Through Iterations
The idea of management-as-organizing/learning is to structure the work so that developers can figure out what to do from the work itself. The fundamental tool for doing this is short iterations which develop working software delivering business value. Project leaders need to know how to organize iteration planning meetings and how to structure the development environment and workday so that people know what to do when they come in to work every day, without being told.

This part of the curriculum must cover such concepts as story cards, backlog lists, daily meetings, pair programming, and information radiators. It should also stress the importance of organizing worker-to-worker collaboration between those who understand what the system must do to provide value and those who understand what the code does.

Concurrent Development
Strategies for concurrent development are an important tool for project leaders, especially in organizations which are used to sequential development. General strategies include:

• sharing partially complete design information
• communicating design constraints instead of proposed solutions
• maintaining multiple options
• avoiding extra features
• developing a sense of how to absorb changes
• developing a sense of what is critically important in the domain
• developing a sense of when decisions must be made
• delaying decisions until they must be made
• developing a quick response capability

System Integrity
Project leaders must assure that the software developed under their guidance has integrity. This starts with assuring that the basic tools are in place for good software development: version control, a build process, automated testing, naming conventions, software standards, etc. Leaders must assure that the development team is guided by the true voice of the customer, so that the resulting system delivers value both initially and over time. They must assure that technical leadership establishes the basis of a sound architecture and an effective interface. They must make sure that comprehensive tests are used to provide immediate feedback, as well a framework so that refactoring can safely take place.

Leading Teams
People with experience in traditional project management who are about to lead agile software development projects might need some coaching in how to encourage a team make its own commitments, estimate its own work, and self-organize around iteration goals. Project leaders attend the daily meetings, listen to and solve team problems, serve as an intermediary between management and the team, secure necessary resources and technical expertise, resolve conflicts and keep everyone working together effectively; but they do not tell developers how to do their job. Project leaders coordinate the end-of iteration demonstration and the beginning of iteration planning, making sure that work is properly prioritized and all stakeholder interests are served.

Measurements
Feature lists or release plans, along with associated effort estimates, are often developed early in an agile project. The difference from traditional project management occurs when actual measurements begin to vary from these plans; for agile development, it is assumed that the plan is in error, and needs to be revised. By measuring the actual velocity or burndown rate, a far better picture of project health can be obtained than measuring variance from a guesstimate. Creating more accurate estimates becomes easier as developers gain experience in a domain and customers see working software. Leaders should learn how to combine reports of actual progress with increasingly accurate estimates into a tool for negotiating the scope of a project; this can be far more effective at limiting scope that the traditional method of fixing scope and controlling it with change approval mechanisms.

A useful technique for project leaders is to aggregate all measurements one level higher than normal.[5] This encourages collaboration and knowledge sharing because issues are called to the attention of a larger group of people. It also helps to avoid local optimization and the dangers of individual performance measurements. This technique is useful, for instance, for defect measurements.

Large Projects
Various techniques for synchronizing agile development across multiple teams are important for project managers to understand. Some of the techniques that might be covered are: divisible architectures, daily build and smoke test, spanning applications, loosely coupled teams that develop interfaces before modules.

Conclusion
We have often heard that the sequential, or waterfall, approach to software development projects is widely known to be ineffective, but it seems to be quite difficult to do things differently. We also heard that many, many features in most systems, possibly even a majority, are not needed or used; yet limiting scope to what is necessary seems to be an intractable problem. One solution to these problems can be found in concurrent engineering, which is widely used in product development as an alternative to sequential development. Concurrent engineering works more or less like a market economy: both depend on collaborating intelligent agents, both allow the end result emerge through communication, and both provide incentives for individuals to be creative and do everything necessary to achieve success.
_______________
References:
[1] Johnson, Jim, Chairman of The Standish Group, Keynote “ROI, It’s Your Job,” Third International Conference on Extreme Programming, Alghero, Italy, May, 26-29, 2002.

[2] Johnston, R B, and Brennan, M, “Planning or Organizing: the Implications of Theories of Activity for Management of Operations”, Omega: The International Journal of Management Science, Vol 24 no. 4 pp. 367-384, Elsevier Science, 1996.

[3] Koskela1, Lauri, “On New Footnotes to Shingo”, 9th International Group for
Lean Construction Conference, Singapore, August, 6-8 2001.

[4] Levinson, Harry, “Management by Whose Objectives?” Harvard Business Review, Vol 81, no 1, January, 2003, Reprint of 1970 article.

[5] Austin, Robert D., Measuring and Managing Performance in Organizations, 1996, Dorset Publishing House

[6] Thimbleby, Harold, “Delaying Commitment”, IEEE Software, vol 5, no 3, May, 1988

[7] Clark, Kim B, Fujimoto, Takahiro, Product Development Performance; Strategy, Organization, and Management in the World Auto Industry, Harvard Business School Press, Boston, 1991.

[8] Thomas Group Inc., National Institute of Standards & Technology Institute for Defense Analyses, from Business Week, April 30, 1990, pp 111

[9] Weber Morales, Alexandra, “Extreme Quality”, Software Development, Volume 11, No 2, February 2003.

[10] Kotter, John P, “What Leaders Really Do,” Harvard Business Review, Vol 79, no 11, December, 2001, Reprint of 1990 article.

Screen Beans Art, © A Bit Better Corporation

Righteous Contracts

Right"eous a. Doing that which is right; yielding to all their due; just; equitable.
[Webster’s Revised Unabridged Dictionary, 1913]

Righteous contracts. A good name for contracts whose purpose is to assure that the parties act in a just and equitable manner and yield to the other their due. Righteous contracts are those governing investments in specialized assets – assets which are very important to a business, but have no value anywhere else. For example, software developed specifically for a single company is a specialized asset, since it is useful only by the company for which it was developed. Agreements to develop specialized assets create a bilateral monopoly; that is, once the parties start working together, they have little option but to continue working together. This bilateral monopoly provides an ideal environment for opportunistic behavior on the part of both supplier and customer.

Thus the purpose of righteous contracts is to prevent opportunistic behavior, to keep one party from taking advantage of another when market forces are not in a position to do so. In a free market where there are several competent competitors, market forces control opportunism. This works for standard or commodity components, but not for specialized assets.

The traditional way to develop specialized assets has been to keep the work inside a vertical organization, where opportunism is controlled by administration. Inside a company, local optimization would presumably be prevented by someone positioned to arbitrate between departments for the overall good of the enterprise. Vertical integration allows a company to deal with uncertainty and change in a rapid and adaptive manner.

Outsourcing
Recently, however, outsourcing has become common in many companies, for very good reasons. An outside company may have lower labor costs or more specialized experience in an area that is not one of the firms core competencies. The cost of producing a service or asset can be considerably lower in an outside company. Of course, there are transaction costs associated with outsourcing, and the total cost (production costs plus transaction costs) must be lower, or vertical integration would make more sense.

Transaction costs associated with outsourcing include the cost of selecting potential suppliers, negotiating and renegotiating agreements, monitoring and enforcing the agreement, billing and tracking payments. Transaction costs also include inventory and transportation above that needed for vertical integration. In addition, there are risks associated with outsourcing, which may result in additional costs. One cost would be that of diminished communication. For example, development of any kind usually requires intense communication between various technical specialties and target users. If distance or intellectual property issues reduce the communication, it will cost more to develop the asset and the results may suffer as well. In addition, moving a specialized skill outside the company may incur lost opportunity costs.

There are two types of contracts which are used for developing specialized assets – Contracts which are executed before the development is done by the supplier, and contracts which are executed after the supplier does the work. A contract executed before work is done is known as a before-the-fact (or ex ante) contract. There are two types of before-the-fact contracts – fixed price contracts and flexible (time-and-materials) contracts. Once these contracts are executed, they set up a bilateral monopoly, fraught with opportunities for exploitation on one side or the other. Therefore, the purpose of these contract is to set up control systems to prevent exploitation.

A contract executed after work is done is called an after-the-fact (or ex post) contract. Suppose a supplier develops a system that it thinks a customer will find valuable and then tries to sell the system. In this case, control comes after the fact; the supplier makes its own decisions, and it’s reward is based on the results. Of course this is a risky proposition, so the supplier has to hedge its bets. One way to do this is to sell the system to multiple customers, basically making it into a commodity product. But this doesn’t help a company that wants suppliers to develop proprietary components for them. In order to entice suppliers to develop specialized assets prior to a contract, a company usually sets up a sole source or favored source program. If a company treats its favored suppliers well, the suppliers develop confidence that their investments will be rewarded and continue to make investments.

On the surface, after-the-fact contracts may seem implausible for software development, but in fact, they are the best solution for contracting a development project. Moreover, the best kind of development processes to use inside a company are those that mimic after-the-fact contracts. How can this be? The explanation starts by understanding why before-the-fact contracts provide poor governance for development projects.

Fixed-Price Contracts
Let’s examine the most commonly used before-the-fact contract, the fixed price contract. A key motivator for fixed price contracts is the desire of a customer to transfer risk to the supplier. This may work for simple, well-defined problems, but it is inappropriate for wicked problems.[1] If the project is complex or uncertain, a fixed price contract transfers a very high risk to the supplier. If the supplier is not equipped to deal with this risk, it will come back to haunt the customer.

Risk should be born by the party best able to manage it. If a problem is technically complex, then the supplier is most likely to be in a position to manage it. If a problem is uncertain or changing, then the customer is in the best position to manage it. Transferring the risk for such problems to the supplier is not only unfair, it is also unwise. There is no such thing as a win-loose contract. If a supplier is trapped on the wrong side of a win-loose contract, the bilateral monopoly which has been formed will trap the customer as well. Both sides loose in the end.

Fixed price contracts do not usually lower cost, because there is always at least some risk in estimating the cost. If the supplier is competent, it will include this risk in the bid. If the supplier does not understand the complexity of the problem, it is likely to underbid. The process of selecting a supplier for a fixed-price contract has a tendency to favor the most optimistic (or the most desperate) supplier. Consequently, the supplier least likely to understand the project’s complexity is most likely to be selected. Thus fixed price contracts tend to select the supplier most likely to get in trouble.

Therefore it is quite common for the customer find a supplier unable to deliver on a fixed price contract. Because the customer no longer has the option to choose another supplier, they must often come to the rescue of the supplier. Alternatively, the supplier might be able to cover its loss, but most likely it will attempt to make the loss up through change orders which add more revenue to the contract. This leads the customer to aggressively avoid any change to the contract. Faced with no other way to recoup the loss, a supplier will be motivated to find ways to deliver less than the customer really wants, either by lowering the quality or reducing the features.

The customer using fixed price contracts to transfer responsibility and risk will often find both back on their plates in the end, and if so, they will be worse off because of it.

Flexible Contracts
“Customers should prefer flexible-price contracts to fixed-price contracts where it is cheaper for the customer to deal with uncertainty than it is for the contractor to do so or where the customer is more concerned with the ability of the contractor to provide a product that works than with price,” writes Fred Thompson in the Handbook of Public Administration. (Second Edition), Rabin, Hildreth, Miller, editors, New York: Marcel Dekker, Inc., 1998.

The flexible-price contract is designed to deal with uncertainty and complexity, but it does not do away with risk, it simply shifts it from the supplier to the customer. For example, after the DOD (U.S. Department of Defense) experienced some very high profile bailouts on fixed price contracts, it began to use more flexible-price contracts is situations where the government was better able to manage the risk. Of course, with the risk transferred to the customer, the supplier has little incentive to contain costs in a flexible-price contract, a point that did not escape contract negotiators at DOD. In order to protect the public interest, DOD perfected controls imposed on the supplier.

Controlling suppliers of flexible-price contracts evolved into a discipline called project management. The waterfall lifecycle grew out of military contracts, and an early focus of PMI (Project Management Institute) was DOD contracts. Companies with DOD contracts not only hire administrators to oversee compliance with contract requirements, they also add accountants to sort out allowable and unallowable costs. Flexible-price contracts invariably have high transaction costs, due to the high cost of control.

Controls Do Not Add Value
High transaction costs would be reasonable if they added value, but in fact, transaction costs are by definition non-value-adding costs. Fred Thompson (Ibid.) notes, “Controls contribute nothing of positive value; their singular purpose lies in helping us to avoid waste. To the extent that they do what they are supposed to do, they can generate substantial savings. But it must be recognized that controls are themselves very costly.”

One way to avoid the high cost of control in flexible-price contracts is not to use them. It may be better to do development internally, where it is easier to deal with uncertainty and respond to change. The question is, on what basis should an outsourcing decision be made? Thompson (Ibid.) counsels, “The choice of institutional design should depend upon minimizing the sum of production costs and transactions costs.” He also notes, “Vertical integration occurs because it permits transaction or control costs to be minimized.”

An interesting problem with this equation is that vertical integration does not always work to minimize control costs. In fact, many organizations find themselves using DOD-like project management controls internally. It seems incongruous that control mechanisms which add cost but not value, and which were invented to prevent opportunistic behavior, would come to dominate development in the very place where they should not be needed. If the reason to develop internally is to provide flexibility in the face of uncertainty, then costly, change-resistant control practices are inappropriate. Traditional project control practices (that freeze requirements, require approval for changes, and track tasks instead of features) have a tendency to create waste, not value, when used inside a company.

After-the-fact Contracts
Let’s assume for the sake of argument that the choice has been made to outsource a complex, specialized development effort. The next question is, how can transaction costs be reduced? In the manufacturing industry, this is done with after-the-fact contracts.

Despite the obvious risks, is not uncommon for suppliers to develop specialized components for a manufacturer prior to obtaining a contract. For example, 3M Optical Systems Division used to develop optically precise lenses for specialized automotive taillights. The reward was a one year contract for a specific model. Unfortunately, after the first year, the automotive company would invariably find a cheaper way to make a similar lens, and Optical Systems would loose the business before it had recovered its investment. The division eventually decided that after-the-fact contracts with Detroit automakers were not profitable and left the business.

There are ways to make after-the-fact contracts work better. Toyota awards contracts for an entire run of a model, and uses target costing to manage costs. Thus a supplier knows that if it wins the business, it can recover its investment, while the customer is confident that the supplier will work reduce costs in line with its economic requirements. In addition, the supplier understands that it will receive favored consideration for similar components in the future.

After-the-fact contracts require two elements to work: shared risk and trust. Toyota shares the risk with a component supplier by guaranteeing the business over the life of an model. Both parties agree to work together to try to meet a target cost profile over the life of the agreement. Note that meeting future target costs is neither guaranteed nor is it the sole responsibility of the supplier. In the best relationships, technical personnel from each company work freely together without worrying about proprietary information, both to meet target costs and to develop new components not yet subject to a contract.

If both parties are pleased with the results of the first contract, they develop trust and a good working relationship, and are more likely continue to do business together. The supplier is inclined to risk more in developing new components when it has developed confidence that the investment will pay off. This kind of relationship can achieve all of the benefits of both outsourcing and vertical integration combined.

But Software is Different…
You might be saying to yourself, this is fine if there is something to be manufactured and sold many times over, like a taillight, but in software we develop a system only once, it is complex and expensive, it is subject to many changes, and if it is not properly designed and executed, huge waste might result. Where is the parallel to THIS in manufacturing?

Consider the large and expensive metal dies which stamp out vehicle body panels. The cost of developing dies can account for half of a new model’s capital investment. Consequently, a great deal of time is spent in all automotive companies working to minimize the cost of these dies. The approach in Japan is distinctly different from that in the U.S., and dramatically more effective. The best Japanese companies develop stamping dies for half the cost and in half the time as their counterparts in the West. The resulting Japanese dies will be able to stamp out a body panel in 70% of the time needed by U.S. stamping operations.

From the classic book Product Development Performance by Clark and Fujimoto, Harvard Business School Press, 1991:

Japan firms use an ‘early design, early cut’ approach, while U.S. practice is essentially “wait to design,, wait to cut.”

Because it entails making resource commitments while the body design is still subject to frequent changes, the Japanese early design, early cut approach entails significant risks of waste and duplication of resources…. Many engineering changes occur after final release of blueprints. At peak, hundreds of changes are ordered per month.

Behind the wait to design, wait to cut approach in U.S. projects is a desire to avoid expensive die rework and scrappage, which we would expect to be an inevitable consequence of the bold overlapping that characterizes the Japanese projects. However, our study revealed a quite different reality. U.S. firms, despite their conservative approach to overlapping, were spending more on engineering changes than Japanese firms. U.S. car makers reported spending as much as 30-50 percent of original die cost on rework due to engineering changes, compared to a 10-20 percent margin allowed for engineering changes by Japanese products.

The Japanese cost advantage comes not from lower wages or lower material prices, but from fundamental differences in the attitudes of designers and tool and die makers toward changes and the way changes are implemented…. In Japan, when a die is expected to exceed its cost target, die engineers and tool makers work to find ways to compensate in other areas…. Die shops in high-performing companies develop know-how techniques for absorbing engineering changes at minimum cost…. In the United States, by contrast, engineering changes have been viewed as profit opportunities by tool makers….

Suppose a body engineer decides to change the design of a panel to strengthen body-shell rigidity. The high performers tend to move quickly. The body designer immediately instructs the die shop to stop cutting the die on the milling machine. Without paperwork or formal approval, the body designer goes directly to the die shop, discusses modifications with the die engineers, checks production feasibility, and makes the agree-upon changes on the spot. Unless the changes are major, decisions are made at the working level. Traditionally, the die shop simply resumes working on the same die. Paperwork is completed after the change has been made and submitted to supervisors for approval. The cost incurred by the change is also negotiated after the fact. The attitude is “change now, negotiate later.

In companies in which die development takes a long time and changes are expensive, the engineering change process is quite different. Consider the context in which changes occur. In extreme versions of the traditional U.S. system, tool and die makers are selected in a competitive bidding process that treats ‘outside’ tool shops as providers of a commodity service. The relationship with the die maker is managed by the purchasing department, with communication taking place through intermediaries and drawings. The individuals who design the dies and body panels never interact directly whit the people who make the dies.

You would think that tool and die makers in Japan must be a department inside the automotive company. How else could it be possible for a designer to walk into a tool and die shop, stop the milling, make changes, and start up the milling again, leaving approvals and cost negotiations for later? But this is not the case. Tool and die makers are supplier companies in Japan, just as they are in the U.S. The difference lies in the attitudes of the different countries toward supplier contracts.

For Toyota in particular, a supplier is a partner. The basis of this partnership is a target cost for each area of the car. This translates into target costs for all development activities, including dies. Of course, U.S. companies have target costs for each component also, but they tend to impose the cost on the supplier without regard to feasibility. This has a tendency to create a win-loose relationship, leaving the supplier no option but to recoup costs through the change process.

In contrast, Toyota does not impose cost targets on suppliers that it does not know how to meet, and engineers from both companies work together to meet target costs. If something goes wrong and the targets cannot be met, Toyota shares the problem in an equitable manner. In this win-win environment, arms-length exchange of information through written documentation and an extensive change approval processes is unnecessary.

The Toyota Production System is founded on the premise that superior results come from eliminating anything which does not add value. Since control systems do not add value, they must be minimized, just like inventory and set-up times. Therefore supplier partnerships based on shared risk and trust are the preferred relationship. The hallmarks of these partnerships are worker-level responsibility for meeting business goals, intense communication at the technical level, a stop-the-line and fix-it-immediately attitude, and an emphasis on speed. Even for large, one-of-a-kind development projects which require highly specialized design, this approach produces dramatically superior results.

Can this work for Software Development?
Developing specialized dies is not that much different than developing specialized software. The key is to establish a partnership relationship which allows true development to take place. Development is done using a repeated cycle of design-build-test, allowing the solution to emerge. The question is, how can a contract be written to support the emergent nature of development?

Neither fixed-price nor flexible-price contracts support the nature of software development. Development always involves tradeoffs, and an organization which facilitates the best tradeoff decisions will produce the best result. Before-the-fact contracts do not support the give-and-take between developers and customers necessary to make the best tradeoffs. A developer should not have to worry about dealing with problems as they arise, but with before-the-fact contracts, this activity has to be paid for by one company or the other. Since every hour must be accounted for, the give-and-take necessary for trade-off decisions is discouraged.

What is needed is a contract approach which allows developers and customers work closely together to develop a business value for a target cost. Examples of how to do this in a vertical organization abound. There many successful examples of using Scrum for product development. Microsoft’s approach to product development is documented by Michael Cusumano in Microsoft Secrets, Simon and Schuster, 1998. The general approach is to set a clear business goal, fix resources, prioritize features, deliver working software in short cycles, and stop working on features when time runs out. This approach has a track record of delivering systems, even large ones, in a predictable timeframe for a predicable cost.

The question is, how can a contract be written to support the same approach? The answer is to move to after-the-fact contracts in which a supplier is paid for the value of the work they do. It works like this: A customer has a clearly defined business value and a target cost in mind for achieving that value. This target cost includes payments to a supplier for their contributions. The customer comes to an agreement with a supplying partner that the business value and the target cost are achievable, including the target cost for the supplier’s participation. Work proceeds without contractual guarantees that the value will be delivered or the target cost will be achieved, but both partners are committed to meet these goals.

Workers at each company use adaptive processes[2] to develop the system as a single team. They communicate intensely at the developer-customer level to make the necessary tradeoffs to achieve the value within the target cost. As working software is delivered, both supplier and customer work together using velocity charts to monitor development progress. If adjustments to the business value or the target cost structure are required, these become apparent early, when they can be addressed by limiting the feature list or extending the schedule. If this changes the business value or target cost, the parties negotiate an equitable way to share the burden or benefit.

Conclusion
Trusted-based partnerships are the first requirement to make after-the-fact contracts work. Partnerships are necessary to facilitate worker-level responsibility for meeting business goals, intense communication between developers and users to make optimal tradeoffs, daily builds and automated testing to facilitate a fix-it-immediately attitude, and a focus on early delivery of working software to create the feedback system critical to good development.

Companies that develop contracts allowing these values to flourish can expect to produce the same dramatically superior results in software development that these values produce in product development.

Lessons for Outsourcers
If your company outsources software development, consider the following:

1. Fixed Price Contracts
Fixed price contracts are risky. There is both a technical risk that the job can’t be done for the allotted cost, and the very real risk that the selection process favors less knowledgeable suppliers. If you assure that you get a competent supplier, then you can be sure the supplier will add a good margin to the cost to cover their risk. Remember that risk should be born by the party most able to manage it, so if the project is complex and changes are likely, you should assume the risk. If the project is a wicked project, you should not even consider a fixed price contract.

If you are considering a fixed price contract, you are probably interested in transferring all responsibility to your supplier. But remember, if this results in a win-loose situation, you will not win. You are going to be committed to the supplier before the cracks begin to show, and if things go wrong, you will suffer as much, if not more, than your supplier. You may have to bail them out. They will no doubt be looking to make up for loses through change orders, so you will have to control these aggressively. That means if your project is prone to uncertainty or change, you really don’t want a fixed price contract.

And finally, it is much more difficult to get what you really need under a fixed price contract. If you got the low bidder, you probably did not get the supplier most familiar with your domain. If the bid was too low, your supplier will want to cut corners. This may mean less testing, fewer features, a clumsy user interface, out-of-date technology. You are going to need to carefully limit user feedback to control changes and keep the price in line, which will make it more difficult to get what your users really want.

Traditional Control Processes
Traditional project management processes tend to emphasize scope management using requirements traceability and an authorization-based change control system. Typically cost control is provided with some variation of an earned value measurement. The first thing to realize is that all of these techniques are expensive and do not add any value to the resulting software. These practices are meant to control opportunism, and if you are concerned that your supplier might take advantage of you, they might make sense. (But try partnerships first.)

You most likely do not want to be using these practices inside your own company; they get in the way of good software development. It’s pretty well known that an iterative approach to software development, with regular user communication and feedback, is far better than the waterfall approach. However, those pesky project management practices tend to favor waterfalls. It’s a good bet that your project will be subject to change (users change their preferences, technology changes, someone forgot a critical requirement), so you want to be using adaptive processes.

2. Trust-based Partnerships
For starters, all internal development should be based on trust-based partnerships – after all, that’s why you are doing inside development in the first place! If you can’t trust someone in your own company, who can you trust?

The fastest, cheapest way to develop software with a supplier is to let their technical people make decisions based on close interaction with and regular guidance from your users. You get the best results and the happiest users this way too. This kind of relationship requires risk sharing and excellent on-going communications. In exchange for this investment, trust-based partnerships adapt well to change and uncertainty and are most likely to yield faster, better, cheaper results.

Lessons for Contractors
If your company supplies software development, consider the following:

1. Fixed Price Contracts
You owe it to your customers to educate them on the pitfalls of fixed price contracts. Make sure they understand that this will make it more difficult for you to deliver the best business value.

2. Traditional Control Processes
Don’t accept traditional control mechanisms; there are better ways. Instead, use prioritized feature sets, rapid iterations and velocity charts to monitor projects.

Never allow the customer to fix cost, schedule and features simultaneously. Preferably, you want to agree to meet cost, schedule and overall business value targets, and provide a variable feature set. If the detailed feature set is not negotiable, then at least one of the other two must be flexible.

Find out what is REALLY important to your customer in terms of business value and deliver that.

3. Trust-based Partnerships
Your top priority when negotiating the relationship is to assure that your development team will have constant user involvement and feedback. You can negotiate what this means and who represents the user, but if you don’t have access to users or a user proxy, you will have a difficult time delivering business value. And delivering business value must be your main objective.
____________________
Footnotes:

[1] A Wicked Problem is one in which each attempt at creating a solution changes the understanding of the problem. See “Wicked Projects” by Mary Poppendieck, Software Development Magazine, May, 2002, posted on this site under the title “Wicked Problems.”

[2] For a discussion of adaptive processes, see “Wicked Projects” by Mary Poppendieck, Software Development Magazine, May, 2002, posted on this site under the title “Wicked Problems.”

Screen Beans Art, © A Bit Better Corporation

The Leadership Paradox

“No one has yet figured out how to manage people effectively into battle; they must be led,” wrote John Kotter in ‘What Leaders Really Do’[1]. He notes that leadership is about helping people cope with change, while management is about coping with complexity. Leaders set direction, managers plan and budget. Leaders align people, managers organize and staff. Leaders motivate, managers control.

Champion
In this context, it should not be a surprise that each new product development program at 3M is led by a ‘champion’. Innovation at 3M is brought about by excited, motivated teams. They are led by a product champion who probably wrote the initial product concept, gathered management support for the program, and recruited most of the team. The champion interprets the product vision for the team, thus representing the customer who is not, after all, on site. The champion sets the pace of development and determines how decisions get made. A champion is also expected to keep working on a good idea even if the program gets killed by management.

Shusa
Similarly at Toyota, a new vehicle development program is led by a shusa or chief engineer. In contrast to the coordinating role of a new vehicle manager at US auto companies, the shusa has complete responsibility for the vehicle, and has the authority necessary to make all program decisions. The shusa has been called a ‘heavyweight program manager’[2], but this is a misnomer, because a shusa is a leader, much more than a manager.

Perhaps the correct characterization of a champion or shusa is a ‘respected leader’. The emphasis of the role is on setting direction, aligning the organization and motivating the team. At 3M, the champion is largely self-nominating, and in both companies, the role holds great stature. In both 3M and Toyota, the product produced by one of these leaders often bears their name (Fuji-san’s car or Art Fry’s Post-it® notes). These companies seem to understand that if a team is to deal with change and innovation, a ‘respected leader’ is needed; a coach is not enough and a manager (in the traditional sense) is the wrong approach.

Software Development
It has been claimed[3] that software development is similar to new product development, because it is an activity which creates something unique for a customer, something which has not been made before. In this sense, programming is not like manufacturing, which makes the same thing many times over and strives to make each instance the same.

If software development is like new product development, then how should software projects be led? There seems to be some ambivalence surrounding the role of leadership in agile methodologies, possibly stemming from the traditional role of a project manager in software development projects. Often the project manager does not have a technical background and may not feel responsible for understanding the technical aspects of the project. Usually the project manager is trained for and measured against the controls of scope, budget and schedule. This administrative role is quite different than the leadership role exercised by a champion or shusa .

If a software project requires vision and must cope with change, this is not going to come from following the rules taught in project management certification courses. If stakeholders need to be aligned and the team needs motivation, managing scope, schedule and cost is not the place to focus attention.

Scrum Master
What are the alternatives to project managers? Beck suggests the use of a ‘coach’. Schwaber and Beedle propose role of a ‘Scurm Master’: ‘The Scrum Master represents management and the team to each other.’[4] ‘… is in touch with all aspects of the project…’[5] ‘… is responsible for ensuring that impediments are promptly removed and decisions are promptly made.’[6]

A coach or Scrum Master is not quite the champion of 3M or the shusa of Toyota, but then again, there is a difference between new product development and most software development. At 3M and Toyota, the customer is not readily available and must be represented. In fact, it’s pretty well accepted at 3M that asking existing customers what they want is not the way to come up with innovations. Instead, you have to understand the problems that customers have, anticipate future needs, and invent something new to solve their problems. A 3M champion creates and communicates a vision of the new product, just as at Toyota, the shusa writes the description of the new vehicle. To do this, they need a high degree of technical expertise as well as a deep understanding of the customers.

Most agile methods depend upon the ‘customer’ to create a vision of the software under development. Sometimes this will work, but it places a large and sometimes unrealistic burden on the customer. While the champion and shusa are technical experts, the customer usually is not, so the customer’s vision of a solution may be limited. In practice, customers often don’t know what they want, and to make matters worse, there are often multiple stakeholders, making it unclear who the ‘customer’ really is.

The Scrum Master represents the customer to the team, and thus must understand who the customer is and what all the stakeholders really want. However, the Scrum Master is not chartered to come up with the technical vision of how to address the customers’ needs. In XP, the technical vision of the system is expected to evolve throughout multiple iterations.

Design vs. Vision
The biggest criticism of agile methodologies is that they do not provide for overall design prior to the beginning of development. At Toyota and 3M, the product largely emerges from the development process, but the programs start with an overall technical vision of what the product will look like in the end. In fact, the shusa has been equated to a system architect of a new car.[7] Both the shusa and the champion are leaders, not managers, but as very experienced technical leaders, they provide the overall design to the product under development, or assure that an overall design is established by the team. They judge the level of design necessary to allow development to proceed in an emergent manner, while assuring that there are no downstream surprises which should have been anticipated.

If a development process must deal with ‘wicked problems’ (see separate essay), then by definition, the solution emerges as development proceeds. Such projects need someone to make the tradeoffs between early design and emergent development. In world class new product development organizations, this role belongs to the champion or shusa, a ‘respected leader’ with the technical and customer savvy to make such tradeoffs.

Technical Lead
Experience shows that if there is not someone to make similar tradeoffs for a software system, large organizations have a tendency to err on the side of caution. A corporate architecture committee, customer committee or administrative coordinators will usually lack an overall vision of the project, and so will look for comfort in detail, even though the detail is not what is needed to proceed safely. It would be better if there were a technical lead on the program to create and be responsible for maintaining the technical vision. It would not be so good, however, if the technical lead were not a leader. A leader not only sets direction, but also aligns and motivates people. The shusa and champion are expected to secure resources, remove impediments, and lead the team into battle. A technical lead who either sits on a pedestal or on the sidelines will not be able to lead the team.

If becoming a technical guru or a PMI graduate does not make a person into a leader, then how should leaders be developed? First, an organization should understand what it means to develop leaders. The armed forces develops leaders as their primary responsibility. A Special Forces team, composed of a dozen experts and a leader, is expected to adapt continually to very demanding conditions, given only the broadest of objectives. But even the most ordinary army units are composed of teams with a leader who is expected to respond to events on the ground, adapt to changing conditions, make decisions and lead the team. Although military training is not being advocated, it might be educational to understand how the military trains its leaders.

Developing Leaders
At 3M, people with technical competence are encouraged to develop a new product concept based on solving a customer problem. In order to carry out that vision, they find they need to recruit and motivate a team and secure resources from the organization. This is one of the most popular career paths in the technical organization, and so many technical people learn how to follow this path.

Toyota and 3M have learned how to make the role of technical leader one of the most desirable in the corporation, and thus they are able to develop a large number of qualified leaders. On the contrary, project management is frequently considered an undesirable career path for software developers, so few seem interested in learning how to manage. To break this vicious circle, an organization needs to begin by considering how to make the role of project leadership attractive to technical experts. Changing job expectations from administration and management to vision and leadership would be a good start.

Finding Leaders
Where does an organization find people to fill the newly defined role of project leader? In many cases the training ground might already be in house. In the past ten years, supervisors and managers in operational areas such as manufacturing, logistics and customer service have been trained to focus on empowering the first line workers to deal with problems and improve their own processes. Thus many organizations already have leadership development programs in place, they just haven’t moved beyond the operational areas yet.

Good operations managers are likely to make good software project leaders, because they have already learned how to work with people and lead a team. If you are blessed with good operations, then look for respected supervisors and first line managers in operations who have a technical background or aptitude, and consider training them as project managers. Alternatively, but not as good, consider sending everyone who will lead a software team to a good training program for first line supervisors in an operational area. But note that it is easier to develop people who know how to lead into managers than it is to develop people who know how to manage into leaders.

You Get What You Expect And What You Inspect
The bottom line is that people respond to the expectations of their management. Software project leaders do not develop in a position where the primary expectation is change control and the primary measurement is earned value. An organization will get what it values, and the agile methodologies do us a great service in shifting our perception of value from process to people, from documentation to code, from contracts to collaboration, from plans to action.
______________________
Footnotes:

[1] ‘What Leaders Really Do’, John P Kotter, Harvard Business Review, Volume 79, Number 11. Reprint of article first printed in 1990.

[2] ‘Toyota’s Principles of Set-based Concurrent Engineering’, Durward K. Sobek II, Allen C. Ward and Jeffrey K. Liker, Sloan Management Review, Volume 40 #2, Winter 1999, and Product Development Performance: Strategy, Organization, and Management in the World Auto Industry, Clark, K.B. and T. Fujimoto, Harvard Business School Press, 1991.

[3] See, for example, Agile Software Development with Scrum, Schwaber, Ken and Mike Beedle, Prentice Hall, 2001, Chapter 6.

[4] Agile Software Development with Scrum, Schwaber, Ken and Mike Beedle, Prentice Hall, 2001, p 31

[5] Agile Software Development with Scrum, Schwaber, Ken and Mike Beedle, Prentice Hall, 2001, p 142

[6] Agile Software Development with Scrum, Schwaber, Ken and Mike Beedle, Prentice Hall, 2001, p 32

[7] ‘Toyota’s Principles of Set-based Concurrent Engineering’, Durward K. Sobek II, Allen C. Ward and Jeffrey K. Liker, Sloan Management Review, Volume 40 #2, Winter 1999

Screen Beans Art, © A Bit Better Corporation

The Impact of Logistics Innovations on Project Management

A project is a “temporary endeavor undertaken to create a unique product or service” (Project Management Institute, 1996). Projects include building a structure, developing a product, and executing a contract. Logistics “plans, implements, and controls the efficient, effective flow and storage of goods, services, and related information from the point of origin to the point of consumption” (Council of Logistics Management, 2000). Logistics is usually thought of in connection with military or manufacturing operations. However, there can be projects involved in logistics (eg. building a bridge to move troops) and logistics involved in projects (eg. supplying material to a construction site). It is not surprising, then, that these two disciplines interact and learn from each other. In this paper, great events in logistics are examined to uncover their impact on project management.

Great Logistics Feats of Antiquity
In May of 218 BC, 29 year Hannibal led about 40,000 troops, thousands of horses, and 38 elephants over the Pyrenees and Alps from Spain to Italy, a feat which had been considered impossible. During this 15 day ‘project’, his troops built jetties and rafts for the elephants to cross the Rhone River, raided towns for provisions, and struggled through avalanche-blocked mountain passes while under attack. Many of the troops and most of the elephants were lost, but even in this weakened conditioned, Hannibal defeated the Roman army that awaited his arrival. (Encyclopedia Britannica: Hannibal)

Hannibal has been singled out as a forerunner of grand strategy in military campaigns. His superb logistics while crossing the Alps is a precursor of military logistics today, a discipline in which the US military is considered the world leader. The routine logistics of military operations are interspersed with unique, temporary ‘projects’, of which Hannibal’s crossing is a prime example. The project management challenges of crossing the Alps included fixed resources, limited time, physical obstacles and troops with mixed allegiances who were ill-suited to the task. The techniques Hannibal used to meet these challenges (planning, field engineering, rapid movement, periodic re-supply, and attentiveness to troops and animals) are standard military practice today. (Encyclopedia Britannica: Strategy)

There are other great feats of antiquity which are forerunners of today’s project management practices. Building the pyramids comes immediately to mind. The Great Pyramid in Egypt was built around 2500 BC. It took about two decades and somewhere between 20,000 and 100,000 laborers to erect. Stone was cut with crude hand tools, transported without wheeled vehicles, and put into place without lifting machinery. This “masterpiece of technical skill and engineering ability” remains “perhaps the most colossal single building ever erected on the planet”. No taller building was built until the 19th century. (Encyclopedia Britannica: Giza, Pyramids of; Building Construction)

Construction projects today are not significantly different than those which built the pyramids. An architect designed the building, materials were obtained from quarries and transported to the site, experts in various ‘trades’ prepared the materials and constructed the pyramid. Built to a fixed plan, time and resources were more or less unlimited for the pyramids (unlike today).

There is another impressive feat that has been executed from the most ancient of times and continues to this day: The Event. Perhaps it is a wedding or a funeral or a coming-of-age ceremony – just about any large gathering has always required serving a lot of food to a lot of people in a very short time. Perhaps several clans gathered once a year, and there was a woman who was particularly good at organizing such events. Planning, gathering food, attending to living quarters, cooking, serving, might be a tradition or a ritual, but the event always came off best if someone behind the scenes was coordinating it all. In some sense, the most seasoned project managers throughout various cultures and times might be the event organizers who really knew how to throw a good party on a inflexible schedule with limited resources.

From Logistics to Project Management
All of the knowledge areas of project management were are developed in these three examples of logistics from antiquity: staging a celebration, managing construction, and executing a military movement. A celebration required excellent procurement, scheduling, quality and resource management. Construction projects required superior planning, managing a fixed scope over a long time, procurement and human resource management. Military movements needed excellent execution, careful management of fixed resources, precise timing, superior human resource management and continual risk assessment.

Viewed from a different perspective, advances in logistics might be thought of as starting with project management. For instance, military logistics may start with unique projects, like ferrying elephants over a river, but once the technique is developed, it can be reused to ferry other large items over a river. Feeding a large number of relatives may be a project the first time it is done, but feeding hundreds of people a sit-down dinner in fifteen minutes is routine logistics at conventions held every day. Even the pyramids were more or less standard burial structures for a period of over 500 years. (Encyclopedia Britannica: Pyramid)

The disciplines of logistics and project management overlap in such a way that it is not always possible, or even necessary to tell them apart. Projects that are repeated and standardized may become exercises in logistics, while the first implementation of a logistical solution might be considered a project. Since both disciplines often address similar problems, there is likely to be a significant transfer of ideas from one discipline to the other.

Traditionally, logistics has been the focus of much study and innovation. Since logistics is usually applied to on-going operations, improvements in logistics usually result in increasing benefits over time. If, for example, the cost of a single car can be reduced by a dollar, then in a factory making 250,000 cars annually, a quarter million dollars will be saved each year.

The author has observed that major advances in logistics usually make their way into project management in ten or fifteen years, since the same forces that spurred the paradigm shifts in logistics are usually at work in the broader economy. Below are two examples of logistics innovations moving from manufacturing to construction in one or two decades.

Standardized Parts
Two hundred years ago, the US military wished to procure 40,000 rifles. At the time, rifles were made by skilled workers who fashioned one rifle at a time. Each rifle was different from the next, so maintenance required individually fashioned parts. Unfortunately, there were not enough skilled workers in the country to make the necessary rifles, let alone maintain them.

In 1798, Eli Whitney, the inventor of the cotton gin, proposed that he could make an incredible 10,000 rifles in two years by designing machinery (templates and fixtures) to make standardized, interchangeable parts. This was a major paradigm shift for manufacturing. It took Whitney over ten years to perfect his manufacturing system, which is regarded as the birth of the tool and die industry. Ultimately the system was successful, and the rifles in the war of 1812 were produced much faster, had higher quality, could be more easily maintained, and cost a great deal less than previous rifles. (Encyclopedia Britannica: Eli Whitney; Tool and Die Making) (Taylor, 1990)

In the 1820’s, US sawmills began producing standard dimension lumber in quantity, and in the 1830’s cheap machine-made nails became available. The first ‘balloon’ frame building is thought to be a warehouse in Chicago built in 1832. Standard construction techniques rapidly evolved: 2x4 studs placed 16 inches on center, 2x10 joists spanning up to 20 feet, stability provided by ¾” sheathing through which windows and doors were cut. Machine-made nails were easily driven into the soft wood and a building could be rapidly assembled from manufactured materials by (relatively) unskilled workers. Houses are still built this way, almost two centuries later. (Encyclopedia Britannica: Building Construction)

It’s easy to visualize how standardized, interchangeable manufacturing parts influenced the development of standardized, interchangeable building parts. The entire direction of the construction industry was heavily influenced by Eli Whitney’s rifles. Although a construction project was still a unique and temporary event, it took on many of the features of standardized manufacturing.

Mass Production
In 1908, Henry Ford introduced the Model T Ford. It was so successful that Ford had to continually invent faster, cheaper ways to manufacture the car. Over the two decades of the Model T’s life, Ford perfected the assembly process, introducing the first moving assembly line in 1913. In 1927, the River Rouge plant received just enough iron ore each morning to make a day’s worth of cars; 28 hours later the ore had become the steel in a finished car. In two decades, Ford produced almost 17 million Model T’s, displacing many horses in the process. The Model T precipitated one of the most rapid and pervasive changes of the lifestyle of common people in history. (Encyclopedia Britannica: Henry Ford)

The steel coming into Ford’s plant also moved into the building industry. During the 1920’s between World War I and the Depression, steel framed high-rise buildings came to the cities of America. As these buildings went up, specialized trades became more important, since framing, sheathing, plumbing, heating, elevators, etc. each required full time specialists. (Encyclopedia Britannica: Building Construction) The various trades required someone to sequence their activities, supply materials, assure safety standards were met, and generally manage the project. Thus the construction project manager emerged to coordinate a complex series of specialized but interrelated construction tasks, each of which contributed to the completion of the building. A well-managed construction project bears quite a few similarities to an assembly line.

Just In Time
Henry Ford’s River Rouge plant was an excellent example of Just-in-Time manufacturing, but his ideas on inventory were not widely held. After World War II, operations research developed theories on inventory management which determined optimized lot sizes, reorder points, and distribution stocking levels. In the 1970’s, a radically different theory was developed in Japan, spearheaded by Toyota Motor Corporation. (Shingo, 1981) The concept that inventory should be kept at a minimum, lots should be very small, and products should be built ‘on demand’ rather than stocked, ran contrary to the current manufacturing ‘wisdom’ in the US. However, Just-in-Time models proved to have significant advantages in capital reduction, plant throughput, quality assurance and market responsiveness.

The benefits of Japanese manufacturing techniques began to dawn on the US manufacturing community in the 1980’s. Eliyahu Goldratt’s 1984 novel, The Goal, popularized Just-In-Time concepts and introduced the ‘Theory of Constraints’. (Goldratt, 1984) This theory suggested that finding and ‘feeding’ the bottleneck workstation in a manufacturing plant would allow all other processes to hold to a steady and predictable pace. Pacing manufacturing with the ‘constraint’ workstation required the manufacturing logistics community to abandon many long-held beliefs about inventory management. But the new logistics paradigm worked so well that it rapidly became standard practice in manufacturing and distribution logistics.

If the author’s thesis that logistics innovations make their way into project management in about a decade is true, then Just-In-Time concepts or their derivatives should have impacted project management in the mid 1990’s. Sure enough, in 1997, Goldratt published Critical Chain, a novel in which he applies the Theory of Constraints to Project Management. (Goldratt, 1997)

The Theory of Constraints is useful in project management when multiple projects are competing for the same resources. Similar to its application in manufacturing, Theory of Constraints project management finds and ‘feeds’ critical path tasks and bottleneck resources. Most of the ‘common wisdom’ embodied in project scheduling techniques must be re-thought for this to work in the project environment. For instance, task durations are planned at 50% probability of completion. Thus it is expected that half of the time, tasks will be late. Indeed, it has proven to be difficult to abandon the perception that it is ‘good’ for all tasks to be completed within their estimated time.

Just-In-Time and Theory of Constraints focus on optimizing the entire flow of material or work, instead of sub-optimizing individual areas. The organizational structure and reward system in the US is often set up to reward individual effort, without much regard to (or understanding of) the impact of that effort on the ‘big picture’. Since World War II, the US economy has became increasingly globalized, and countries with different cultures and reward systems are selling products in the US. Unconstrained by ‘standard practice’, these countries consider and optimize manufacturing from a different, often broader perspective. This globalization of the economy has played a large role in bringing Just-In-Time and the Theory of Constraints to logistics.

Globalization has influenced project management also. In the electronics industry, very rapid time-to-market with ‘cast-in-stone’ release dates are required for new products, and Theory of Constraints project management is being widely applied to meet these challenges.

Lessons for the Information Industry
One of the fundamental driving forces behind paradigm shifts in logistics is an overwhelming need for lower cost and less specialized labor, which leads to standardization. This is often followed by an opposing force, the desire for customization.

Standardization
In 1800, the lack of enough skilled workers to make rifles in the face of an impending war was an unacceptable situation that demanded the invention of standardized, interchangeable parts. In the early 1900’s, the demand for inexpensive transportation that allowed ordinary people to travel great distances was a key forcing function in the emergence of mass production.

It’s easy to see this drive for standardized, interchangeable parts at work in the information industry. In 1983, the IBM PC took the world by force, rapidly becoming the standard of the industry and soon outstripping IBM’s ability to maintain control. (One can imagine that Henry Ford had to invent the assembly line to avoid the same fate as IBM.)

Standardized Internet access, embodied in standard web browser capability, has driven the on-line revolution that is sweeping the country. A single cell phone frequency in Europe has spurred European cell phone use to a far greater level than in the US, which has multiple cell phone frequencies. In the face of a huge demand for a standardized document format, Microsoft jeopardized its position by releasing incompatible versions of Microsoft® Word, while Adobe® Acrobat® filled the vacuum with a standardized ‘Portable Document Format’ (PDF).

Standardization is driven by an insatiable need which cannot be satisfied at an acceptable price by the available techniques or skilled people. Today the software industry has an enormous gap between the need for programming and the available programmers. It should be obvious to students of economic history and that the software development environment is ripe for a massive switch to standardized, interchangeable, mass-produced components.

Mass Customization
As brilliant as Henry Ford was, he missed out on a major trend that overtook the success of his Model T. After two decades, people began to get tired of black (the only color of the Model T). Copying Ford’s methods, other manufacturers figured out how to make inexpensive cars, and they started adding new features. Ford’s failure to recognize the shift in customer desires caused his company to loose its overwhelmingly dominant position in the industry. (Encyclopedia Britannica: Henry Ford) In the 1970’s the same thing happened to the US car makers – their failure to respond rapidly to the huge demand for fuel economy and high quality allowed Japanese car-makers to gain a large market share.

It is not surprising that one of the latest logistics innovations is mass customization, enabled by the information revolution. Dell Computer builds computers only after receiving an order, and ships the customized computer within days. Anderson Windows custom-builds windows to fit individual residential houses. Mass customization, a key advance in new product development and logistics, requires modular architectures and standardized components that are not assembled until individual customers place orders. Sophisticated information system support, especially in order management, is necessary for successful mass customization. (Gooley, 1998)

Products are not the only things that can be customized; the Internet has abundant examples of customized marketing, customer service, and information delivery. ASPs (Application Service Providers) are rapidly positioning themselves to ‘rent’ applications through the web. ASPs sell common applications that can be rapidly configured to meet individual customer requirements. In fact, the appearance of ASPs is probably a direct result of the pressure to produce ‘standard’ software in the face of a severe programmer shortage. ASPs success will depend on how they manage to retain the advantages of standardization while moving toward mass customization.

Software project management is also being influenced by mass customization. Instead of finalizing system requirements at the beginning of a project, it is considered good practice these days to allow users and customers to modify the requirements on-going basis as the software is being developed. The concept of continuing user feedback during software development started with the spiral life cycle, originally proposed in 1988 (Boehm, 1988), and evolved to The Unified Software Development Process (Jacobson, 1999). Common wisdom today holds that IT projects should iterate toward an increasingly competent set of capabilities through incremental builds which allow continued customization of the software during its development lifecycle.

Summary and Conclusion
Logistics innovations are not accidents. They are driven by economic forces which demand a paradigm shift to keep an undesirable situation from overwhelming the economy. Once the paradigm shift occurs, the forces that caused it continue to exert pressure on other areas. Over time, the new paradigm will make its way into project management practice.

Often the economic driving force behind innovation is an overwhelming need for something in far greater quantities at far cheaper prices than current practices allow. This drives innovation in the direction of standardization. However, once standardization becomes the common paradigm, a shift in expectations usually occurs. Customers begin to expect tailored products, without increasing the cost, of course. This drives innovation in the direction of mass customization.

Another driving force behind innovation is globalization, which brings unique perspectives to logistics and project management. A fresh look at a problem by someone who doesn’t know the ‘right’ answer can lead to out-of-the-box thinking and new approaches.

By looking for trends in economic driving forces and noting innovations in logistics, project managers can predict the forces which will drive project management practices in the future.
____________________
References
Barry W. Boehm, “A Spiral Model of Software Development and Enhancement”, IEEE Computer, Volume 21, Number 5, May, 1988, Pages 61-72

Council of Logistics Management, “Definition of Logistics”, retrieved March 29, 2000 from http://www.clm1.org/

Goldratt, Eliyahu, Critical Chain, North River Press, 1997

Goldratt, Eliyahu, The Goal, North River Press, 1984

Gooley, Toby B., “Mass Customization: How logistics makes it happen”, Logistics Management and Distribution Report, April, 1998, retrieved on March 29, 2000, from ‘Logistics Online’

Jacobson, Ivar, Booch, Grady and Rumbaugh, James; The Unified Software Development Process, Addison-Wesley, 1999

Project Management Institute, Guide to the Project Management Body of Knowledge, 1996, The definition of ‘Project’ is in Section 1.2, Page 4.

Shingo, Shigeo, Study of ‘Toyota’ Production System form Industrial Engineering Viewpoint, Japan Management Association, 1981

Taylor, David, Object-Oriented Technology, A Manager’s Guide, Addison-Wesley, 1990, Reference to Eli Whitney, Pages 85-87

Published in Proceedings of PMI Seminars; Symposium 2000; Houston, Texas, September, 2000

Screen Beans Art, © A Bit Better Corporation