The Cost Center Trap

In the 1960’s, IT was largely an in-house back-office function focused on process automation and cost reduction. Today, IT plays a significant strategic and revenue role in most companies, and is deeply integrated with business functions. By 2010, over 50% of firms’ capital spending was going to IT, up from 10-15% in the 1960’s.[1] But one thing hasn't changed since the 1960’s: IT has always been considered a cost center. You are probably thinking "Why does this matter?" Trust me, cost center accounting can be a big trap.

Back in the mid 1980’s Just-in-Time (JIT) was gaining traction in manufacturing companies. JIT always drove inventories down sharply, giving companies a much faster response time when demand changed. However, accounting systems count inventory as an asset, so and any significant reduction in inventory had a negative impact on the balance sheet. Balance sheet metrics made their way into senior management metrics, so successful JIT efforts tended to make senior managers look bad. Often senior management metrics made their way down into the metrics of manufacturing organizations, and when they did, efforts to reduce inventory were half-hearted at best. A generation of accountants had to retire before serious inventory reduction was widely accepted as a good thing.[2] 

Returning to the present, being a cost center means that IT performance is judged – from an accounting perspective – solely on cost management. Frequently these accounting metrics make their way into the performance metrics of senior managers, while contributions to business performance tend to be deemphasized or absent. As the metrics of senior managers make their way down through the organization, a culture of cost control develops, with scant attention paid to improving overall business performance. Help in delivering business results is appreciated, of course, but rarely is it rewarded, and rarer still is the cost center that voluntarily accepts responsibility for business results.

Now let’s add an Agile transformation to this cost center culture. Let’s assume that the transformation is supposed to bring benefits such as faster time to market, more relevant products, better customer experiences. And let’s assume that the cost center metrics do not change, or if they do change, process metrics such as number of agile teams and speed of deployment are added. I’ll wager that very few of those agile teams are likely to focus on improving overall business performance. The incentives send a clear message: business performance is not the responsibility of a cost center.

Being in a cost center can be demoralizing. You aren’t on the A team that brings in revenue, you’re on the B team that consumes resources. No matter how well the business performs, you’ll never get credit. Your budget is unlikely to increase when times are good, but when times are tight, it will be the first to be cut. Should you have a good idea, it had better not cost anything, because you can’t spend money to make money. If you think that a bigger monitor would make you more efficient, good luck making your case. Yet if your colleagues in trading suggest larger monitors will help them generate more revenue, the big screens will show up in a flash.[3]

Let’s face it, unless there are mitigating circumstances, IT departments that started out as cost centers are going to remain cost centers even when the company attempts a digital transformation. What kind of mitigating circumstances might help IT escape the cost center trap?

1. There is serious competition from startups.
   Startups develop their software in profit centers; they haven’t learned about cost centers yet. And in a competitive battle, a profit center will beat a cost center every time.
2. IT is recognized as a strategic business driver.
   You would think that a digital transformation would be undertaken only after a company has come to realize the strategic value of digital technology, but this is not the case. IT has been treated as if it were an outside contractor for so long that it is difficult for company leaders to think of IT as a strategic business driver, integral to the company's success going forward.
   
3. A serious IT failure has had a huge impact on business results.
   When it becomes clear exactly how dependent a profit center is on a so-called cost center, people in the profit center are often motivated to share their pain with IT. Smart IT departments will use this opportunity to share the gain also.

Many people in the Agile movement preach that teams should have responsibility for the outcomes they produce and the impact of those outcomes. But responsibility starts at the top and is passed down to teams. When IT is managed as a cost center with cost objectives passed down through the hierarchy, it is almost impossible for team members from IT to assume responsibility for the business outcomes of their work. When IT metrics focus on cost control, digital transformations tend to stall.

Every ‘full stack team’ working on a digital problem should have ‘full stack responsibility’ for results, and that responsibility should percolate up to the highest managers of every person on the team.  Business results, not cost, should receive the focused attention of every member of the team, and every incentive that matters should be aimed at reinforcing this focus.

The Capitalization Dilemma

Let’s return to the surprising assertion that in 2010, over 50% of firms’ capital spending was going to IT.[1] One has to wonder what was being capitalized. Yes, there were plenty of big data centers that were no doubt capitalized, since the movement to the cloud was just beginning. But in addition to that, a whole lot of spending on software development was also being capitalized. And herein lies the seeds of another undue influence of accounting policies over IT practices.

Software development projects are normally capitalized until they are “done” – that is they reach "final operating capability" and are turned over to production and maintenance.[1] But when an organization adopts continuous delivery practices, the concept of final operating capability – not to mention maintenance – disappears. This creates a big dilemma because it's no longer clear when, or even if, software development should be capitalized. Moving expenditures from capitalized to expensed not only changes whose budget the money comes from, it can have tax consequences as well. And what happens when all that capitalized software (which, by the way, is an asset) vanishes? Just as in the days when JIT was young, continuous delivery has introduced a paradigm shift that messes up the balance sheet.

But the balance sheet problem is not the only issue; depreciation of capitalized software can wreck havoc as well. In manufacturing, the depreciation of a piece of process equipment is charged against the unit cost of products made on that equipment. The more products that are made on the equipment, the less cost each product has to bear. So there is strong incentive to keep machines running, flooding the plant with inventory that is not currently needed. In a similar manner, the depreciation of software makes it almost impossible to ignore its sunk cost, which often drives sub-optimal usage, maintenance and replacement decisions.

Capitalization of development creates a hidden bias toward large projects over incremental delivery, making it difficult to look favorably upon agile practices. Hopefully we don't have to wait for another generation of accountants to retire before delivering software rapidly, in small increments, is considered a good thing.

To summarize, the cost center trap and the capitalization dilemma both create a chain reaction:

1. Accounting drives metrics.
   ⇩
2. Metrics drive culture.
   ⇩
3. Culture eats process for lunch.

The best way to avoid this is to break the chain at the top – in step 1. Stop letting accounting drive metrics. Alternatively, if accounting metrics persist at the senior management level, then break the chain at step 2 – do not pass accounting metrics down the reporting chain; do not let them drive culture. When teams focus on improving the performance of the overall business, accounting metrics should move in the right direction on their own; if they don't then clearly something is wrong with the accounting metrics.

Beware of Proxies

This year Jeff Bezos's annual letter to Amazon shareholders[4] listed four essentials that help big companies preserve the vitality of a startup: customer obsession, a skeptical view of proxies, the eager adoption of external trends, and high-velocity decision making. These seem pretty clear, except maybe the second one: a skeptical view of proxies. Just what are proxies? Bezos explains:

“A common example is process as proxy. Good process serves you so you can serve customers. But if you’re not watchful, the process can become the thing. This can happen very easily in large organizations. The process becomes the proxy for the result you want. You stop looking at outcomes and just make sure you’re doing the process right. Gulp.”

“Another example: market research and customer surveys can become proxies for customers – something that’s especially dangerous when you’re inventing and designing products.”

Here are some common proxies we find in software development:

Accounting metrics are proxies, and not very good ones at that, because they encourage local sub-optimization. 

Project metrics – cost, schedule, and scope – are proxies. Worse, these proxies are rarely validated against actual outcomes.  

“The Business” is a proxy for customers. Generally speaking, so is the product owner.

Proxies should be resisted, Bezos argues, if you want a vibrant startup culture in your company. But without proxies, how do you manage the dynamic and increasingly important IT organization? You make a habit of measuring what really matters - skip the proxies and focus on outcomes and impact.

In his excellent book, “a Seat at the Table,”[5]  Mark Schwartz proposes that IT governance and oversight should begin with strategic business objectives and produce investment themes that accomplish these objectives. IT leaders fund teams to produce desirable outcomes that will have impact on the strategic objectives. Note that these outcomes are not proxies, they are real, measurable progress toward the strategic objective. Regular reviews of teams’ progress -- quantified by these measurable outcomes -- provides leaders with insight, flexibility and an appropriate level of control. At the same time, detailed decisions are made by the people closest to customers after careful investigation, experimentation and learning.

Schwartz concludes: "this approach can focus IT planning, reduce risk, eliminate waste, and provide a supportive environment for teams engaged in creating value."[5] What's not to like?

______________________
Footnotes:

[1] From “What is Digital Intelligence” by Sunil Mithas and F. Warren McFarlan, IEEE Computing Edge, November 2017. Pg.9.

[2] The 1962 book “The Structure of Scientific Revolutions” by Thomas Kuhn discussed how significant paradigm shifts in science do not take hold until a generation of scientists brought up with the old paradigm finally retire.

[3] Thanks to Nick Larsen. Does Your Employer See Software Development as a Cost Center or a Profit Center?

[4] Jeff Bezos - Letter to Shareholders - April 12, 2017

[5] "A Seat at the Table" by Mark Schwartz

The Scaling Dilemma

“One of the most scalable organizations in human history was the Roman army. Its defining unit: The squad – eight guys. The number of guys that could fit in a tent,” says Chris Fry, who knows a bit about scaling. He led software development at Salesforce.com during its years of hyper growth, and is now SVP of Engineering at Twitter. Fry found that the way to build a scalable organization is to focus on the basic building blocks – small, stable, multidisciplinary teams that are expected to independently tackle problems, make decisions, and get things done. Fry advises, “When it comes to building a deeply efficient engineering organization, there are several things you can do to move the needle:

• Build strong teams first. Assign them problems later.
• Keep teams together.
• Go modular. Remove dependencies.
• Establish a short, regular ship cycle.”[1]

Amazon.com works on the same principle. Its basic unit is the two-pizza team – a team small enough to be fed with two pizzas. When Amazon needed to scale, two-pizza teams were chartered to build Amazon Web Services, one service per team. A team includes everyone needed to design, deliver, and support the service – from specifications to operations. If a service is too large for a two-pizza team, Amazon prefers split the service into smaller pieces rather than to combine teams to deal with the larger service, because this preserves the dynamic interactions of small teams.

What Could Go Wrong?

If it works for Salesforce, Amazon and Twitter, surely it will work for you… So you form a lot of small, independent, multidisciplinary teams and you are careful to give them all clear goals. What could go wrong with that?

August 27, Scene 1

“Hi Owen, how’s it going?”

“Just great! We’re going to make our target. We got the last piece done overnight. We’re working on integration testing right now. We’ll have it ready to release tomorrow. Lucky for me, because this month I really need that bonus.”

August 27, Scene 2

“Hey George, I hear you had a shutdown last night.”

“Yeah, we got things up fast, but it still counts against our shutdown limit. It’s the last one we can afford this month, if we want our bonuses. So there won’t be any more.”

“Do you know what caused it?”

“The usual. A new release from development. A naive piece of code, the kind of thing you can’t test for. Anyway, it’s fixed.”

“And you’ve made sure they won’t make that mistake again?”

“Nah, they don’t want to listen to us. We’re just not going to put up any more releases until September. We’re not going to miss our target.”

“But I hear they have another release almost ready to go.”

“Over my dead body.”

The Dilemma

It is a beautiful thing when the building block squads of an organization gel into high performance teams and can be counted on to meet challenging goals. But tricky part is, once you create these strong independent teams, how do you get them to work together? How do teams maintain their autonomous character while working in concert with an increasingly large network of other teams? How do you make sure that each team has a clear goal, but none of the goals are in conflict?

In a lean environment, the leader’s role is to set up strong teams, to be sure, but it is also to devise a system – let’s call it a goal system – which assigns goals to teams. This is no easy task. Teams need clear, meaningful goals; they have to be the right goals; teams must have the capacity, capability and autonomy to achieve their goals; and most important, the goals of various teams cannot conflict with each other.

One thing we know for certain is that local goals create local optimization. So it’s clear that the start of a goal system is a system-level, unifying goal. Something that conveys the purpose of the work, the why. Some way to confirm that progress is being made – at the team level – toward achieving the overall purpose. Of course, this is a lot easier said than done.

The Goal System

Many companies use projects to set up a goal system. A project manager lets the team know what the project goal is and what everyone needs to do to reach the goal. If there are several competing projects, a Project Management Office (PMO) is added to manage the project portfolio and distribute corporate goals among projects. However, there are problems with the project approach. You don’t often see stable teams in a project company, because people are usually assigned at project start and reassigned at the end. Worse, people are often assigned to multiple projects with competing demands on team members’ time. Finally, since most projects are conceived of as a relatively large batch of work, project teams tend to be quite a bit larger than a squad of eight to fourteen people. So the basic building blocks of scale – small, intact, multidisciplinary teams – are rarely found in a project environment.

One of the things Scrum has contributed to the practice of software development is the idea that small autonomous teams perform much better than large project teams or single-discipline teams that work in sequence. So Scrum provides the building blocks of scale, but unfortunately, it does not contain a scalable system for choosing team goals, making sure they contribute to organizational goals and are in sync with the goals of other teams. So we need to look elsewhere for ways to set up a goal system.

The Theory of Constraints

People with a lean mindset might look to the Theory of Constraints (TOC) for guidance on choosing and communicating team goals because it has a good track record for directing the efforts of multiple teams toward a single goal, at least in manufacturing.[2] TOC starts with the assumption that in any system there will always be a constraint that gets in the way of achieving the system goal, and the way to keep teams working toward the overall system goal is to be sure that everyone is focused on getting more work through the system constraint. Let’s see how TOC might be applied to developing a software system.

When the Constraint is Technical

The first step – after clarifying the overall system purpose and goal – is to find the biggest constraint to achieving that goal. For purpose of discussion, let’s choose one of the most typical technical constraints encountered in delivering a software system: the integration of various components of the system without the introduction of defects or unintended consequences. In fact, project organizations typically allocate a third or more of the project time to release overhead – including integration, testing, fixing defects, and deployment – with the largest portion going to finding and fixing problems discovered during integration. When integration is the system constraint, TOC tells us that the most important focus for development teams should be removing this constraint.

Agile approaches to software development recommend the frequent delivery of working software to customers. When this recommendation is followed literally (software is released to end customers frequently), the integration constraint is regularly exposed and has to be confronted. One of the earliest agile approaches, Extreme Programming (XP), includes technical practices such as Test Driven Development and Continuous Integration that help make frequent releases practical. Continuous Delivery[3], which expands on these practices, has gained widespread favor as the agile approach which explicitly focuses on the integration constraint. In Continuous Delivery we find actionable advice on how to tackle the integration problem with techniques that scale across large networks of teams.

The objective of Continuous Delivery is to dramatically increase the number of times integration occurs while decreasing the amount of time it takes to negligible levels. This has the same effect that just-in-time flow does in manufacturing: the impact of defects is reduced to near zero because they are discovered immediately before they can propagate or hide. The problem is, a much wider swath of an organization needs to get involved in Continuous Delivery than is typically found on a development team. The system architecture has to be devisable, the marketing department has to figure out how to deal with frequent deliveries, the development and operations departments have to work closely together.

The Theory of Constraints can help here. If the constraint is integration, TOC recommends that we measure the rate at which work moves through the constraint – in this case the rate at which completely integrated and tested software is released to production – and make improvement of this rate the goal of every team involved in the system. It turns out that a throughput measurement on the system constraint is a great metric for team goals because it is easy to measure, provides immediate feedback, and is structured to result in improved system-wide performance. If everyone working on a system is trying to stabilize and improve the rate at which tested, integrated software is successfully released to end customers, then teams across the system will naturally have to work together and will find that their goals are compatible.

When the Constraint is Knowledge

More often than not, however, technical issues are not the biggest constraint in system development and the fundamental problem is not an integration problem, it is a design problem or a fitness-for-use problem. Far too often we end up with a system that doesn’t work well or is difficult to use. In this case, the biggest constraint in developing software systems is the way in which we decide what to build and parse that decision amongst the teams doing the work.

Project organizations spend a good deal of time deciding what to do and turning these decisions into goals or requirements. However, this activity is front-loaded into the beginning of the project, while verification that the project requirements are correct waits until the project is complete – too late to make changes. Most project organizations consider deciding what to build to be an execution problem rather than a constraint. They would say: It’s unfortunate that project goals and requirements are sometimes wrong, but the way to fix this is to work harder on getting them right before the next project.

Organizations with a lean mindset would frame the problem differently. They are likely to identify the biggest barrier to making good decisions as incomplete knowledge, and thus the biggest constraint of the system would be the rate at which knowledge is generated. So a lean organization would focus on the feedback loop between customers and development teams. They would decrease the length of the feedback loop, increase the speed of the feedback loop, and remove barriers to the free flow of information inside the feedback loop. These organizations would say: If we can test our assumptions and designs more quickly we will learn faster and make better decisions.[4]

No More Politics

One of the signs that teams have conflicting goals is negative politics. A good way to eliminate political wrangling is to get teams to work together toward a unifying goal and to show team members the impact of their actions on that goal – in real time.

August 27, Alternate Scene

“Hi George, I noticed a little glitch in the last night’s graph so I thought I’d check to see if anything went wrong.”

“Oh hi, Owen. Glad you stopped by. As a matter of fact, we did have a shutdown last night, but we were lucky and caught it right away and got things back up fast enough we didn’t lose any customers.”

“So what caused it, do you think?”

“It was a database lockout. It seems to happen a lot about six to eight hours after a new release. When we find it we do a workaround that lasts until the next release.”

“Whoa! That means it’s something in the code that isn’t getting caught in testing.”

“Well, yeah! There’re a lot of problems in code that only come out during production.”

“Um, well, we worked late last night to get that feature ready that Chris asked for. Do you think we can do another release tomorrow?”

“Well Owen, let’s take a look at the graph. You see last night there was a downward spike here, where we closed down the site and no one could use the app. But we got it up so fast that almost everyone was still around and we could just reconnect them. It was the middle of the night here, so we mostly had browsers in Europe and Asia. We don’t have many customers there – yet. But what if we hadn’t caught it so fast? In five minutes we would have lost maybe half of the people using the app. What if it had happened during the daytime here? That spike would have been an order of magnitude bigger, and we would have lost a lot more people. It’s one of our busiest seasons, right before school starts. Do you really think we should take such a risk?”

“But George, we have to do it sometime. Sooner is better than later.”

“Not the way I see it, Owen. The fewer releases we put out, the fewer customers we’re likely to disrupt.”

“Okay, I see your point, but we’ve got to fix that problem so we can release frequently, because new features are what drives that graph up higher.”

“Not if the system crashes, they aren’t.”

“Whatever. We still have to fix the problem.”

“Yeah, so how do you propose we do that?”

“How about a side-by-side release with a trigger that knocks out the new system if it gets flaky?”

“Easy for you to say, Owen. You don’t have to make it happen.”

“But if I could make it happen, would you let me?”

“Sure, why not? But it won’t be easy.”

“How about I get with the team and tell them that in order to have a release, we have to write some failure detection and recovery code, and babysit the next release around the clock to be sure it works.”

“Can’t hurt to give it a try.”

The Unifying Goal

A network of strong teams is the first step to scale. The second step is to set up a system that distributes goals to teams in a way that avoids goal conflict. One good way to do this is to find a goal that is the final arbitrator of ‘good,’ make it visible to all teams in real time, and hold teams responsible for it.

The purpose of throughput accounting in the Theory of Constraints is to create just such a unifying goal. Simply stated, throughput accounting provides a measurement of the rate at which an organization achieves its purpose. This rate is effectively the same as the rate of throughput at the system constraint, so teams working to improve throughput at the constraint or to improve overall system impact are working toward the same goal. Either way, a single unifying goal allows individual teams to act autonomously, confident that they are not working at cross purposes with other teams.

What if you cannot find a unifying goal that represents the system constraint, or if a team’s work has no apparent impact on that goal? Over time it would be better to move to a decoupled architecture so that individual teams can have an impact on the overall system goal. But in the meantime, each team should monitor the impact of its work on its immediate customers. The question is not: Did the team complete its work? The proper question is: Did the team give its immediate customers what was needed, when it was needed, in a state that allowed the downstream teams to perform their work well?

Finally, remember that scaling is a two way street. If you think that scaling gives you problems as a leader, imagine the problems it brings to other people in the organization. It’s fun to work at a small company where everyone knows what’s important and works together to make the company successful. But as the company grows, people at all levels are in danger of losing their line of sight from what they are doing to the company’s success. This limits their ability to act with initiative to bring about that success, and thus undermines their engagement. You can’t scale unless you keep all of the bright minds in the company engaged and working together to help the company grow.
______________________________

Footnotes

[1] From First Round Review: Unlocking the Power of Stable Teams with Twitter's SVP of Engineering.

[2] I was reminded of this as I read Tame the Flow by Steve Tendon and Wolfram Müller, which provides a nice summary of how the Theory of Constraints in general, and throughput accounting in particular, can generate much better team goals than either the work-bin WIP limits of Kanban or the cost/profit focus of traditional cost accounting.

[3] See Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble and Dave Farley (2010), and Lean Enterprise: Adopting Continuous Delivery, DevOps, and Lean Startup at Scale by Jez Humble , Barry O’Reilly, and Joanne Molesky (2014).

[4] It’s interesting to note that Continuous Delivery is an excellent way to address the knowledge constraint as well as the technical constraint of software development.

A Tale of Two Terminals

By any measure, Terminal 3 at Beijing Capital Airport is impressive. Built in less than four years and officially opened barely four months before the Olympics, the massive terminal has received numerous awards for both its stunning design and its comfortable atmosphere. And it escaped the start-up affliction of many new airport terminals when it commenced full operations on March 26, 2008 without any notable problems.

The next day, half way around the world, Heathrow Terminal 5 opened for business. At one-third the size of Beijing Terminal 3, the new London terminal had taken twice as long to build and cost twice as much. Proud executives at British Airlines and BAA (British Airports Authority) exuded confidence in a flawless opening, but that was not to be. Instead, hundreds of flights were canceled in the first few days of operation, and about 28,000 bags went missing the first weekend. The chaotic opening of Heathrow Terminal 5 was such an embarrassment that it triggered a government investigation.

The smooth opening of Beijing Terminal 3 was not an anomaly – Terminal 2 at Shanghai Pudong International Airport opened the same day, also without newsworthy incident. Given the timing just before the Beijing Olympic games, it was clear that China was keenly interested in projecting an image of competence to the traveling public. But of course, the UK was equally interested in showcasing its proficiency, and the British executives clearly expected that the opening of Heathrow Terminal 5 would go smoothly. So the question to ponder is this: How did the Chinese airports manage two uneventful terminal openings? Did they do something different, or were the problems in London just bad luck?

It’s not like testing was overlooked at Heathrow Terminal 5. In fact, a simulation of the terminal’s systems was developed and all of the technical systems were tested exhaustively, even before they were built. A special testing committee was formed and thousands of people were recruited to be mock passengers, culminating in a test with 2000 volunteer passengers a few weeks before the terminal opened. On the other hand, the planned testing regime was curtailed because the terminal construction was not completed as early as planned; in fact, hard-hats were required in the baggage handling area until shortly before opening day. In addition, a decision was made to move 70% of the flights targeted for Terminal 5 on the very first day of operations, because it was difficult to imagine how to move in smaller increments.[1]

Those of us in the software industry have heard this story before: the time runs out for system testing, but a big-bang cut-over to a mission critical new system proceeds anyway, because the planned date just can’t be delayed. The result is predictable: wishful thinking gives way to various degrees of disaster.

That kind of disaster wasn’t going to happen in Beijing. I was in Beijing a month before the Olympics, and every single person I met – from tour guide to restaurant worker – seemed to feel personally responsible for projecting a favorable image as the eyes of the world focused on their city. I imagine that for every worker in Terminal 3, a smooth startup was a matter of national pride. But the terminal didn’t open smoothly just because everyone wanted things to go well. It opened smoothly because the airport authorities understood how to orchestrate such a large, complex undertaking that involved hundreds of people. After all, they had just finished building the airport at amazing speed.[2]

The opening ceremony of the Beijing Olympics was also a large, complex undertaking that involved hundreds of people. It’s easy to imagine the many rehearsals that took place to make sure that everyone knew their part. When it comes to opening a new terminal, the idea of rehearsals doesn’t usually occur to the authorities, but at Beijing Capital Airport, rehearsals started in early February. First a couple of thousand mock passengers took part in a rehearsal, then five thousand, and finally, on February 23rd, 8000 mock passengers checked in luggage for 146 flights. This was the average daily load expected a week later, when six minor airlines moved into Terminal 3. During the month of March, Terminal 3 operated on a trial basis, ironing out any problems that arose. Meanwhile, staff from the large airlines about to move to the terminal rehearsed their jobs in the new terminal day after day, so that when the big moving day arrived, everyone knew what to do. On March 26, all the practice paid off when the Terminal was opened with very few problems.

This certainly wasn’t the approach taken at Heathrow Terminal 5. It’s pretty clear that the opening chaos was caused because people did not know what to do: they didn’t know where to park, couldn’t get through security, didn’t know how to sign on to the new PDA’s to get their work assignments, didn’t know where to get help, and didn’t know how to stop all the luggage from coming at them until their problems got sorted out. Even the worst of the technical problems was actually a people problem: the baggage handling software had been put in a ‘safe’ mode for testing, and apparently no one was responsible for removing the patch which cut off communication to other terminals in the airport. It took three days to realize that this very human error was the main cause of the software problems![3]

In testimony to the British House of Commons, union Shop Steward Iggy Vaid testified:[4]

We raised [worker concerns] with our senior management team especially in British Airways. … [Their response was to] involve what we call process engineers who came in and decided what type of process needed to be installed. They only wanted the union to implement that process and it was decided by somebody else, not the people who really worked it. The fact is that they paid lip service to, ignored or did not implement any suggestion we made.

… as early as January there was a meeting with the senior management team at which we highlighted our concerns about how the baggage system and everything else would fail, that the process introduced would not work and so on. We highlighted all these concerns, but there was no time to change the whole plan.

... [Workers] had two days of familiarization in a van or were shown slides; they were shown where their lockers were and so on, but there was no training for hands-on work.….

The opening of a new airport terminal is an exercise in dealing with complexity. At Heathrow Terminal 5, new technical systems and new work arrangements had to come together virtually overnight – and changing the date once it has been set would have been difficult and expensive. Hundreds of people were involved, and every glitch in the work system had a tendency to cascade into ever larger problems.

If this sounds familiar, it’s because this scenario has been played out several times in the lives of many of us in software development. Over time, we have learned a lot about handling unforgiving, complex systems, particularly systems that include people interacting with new technology. But every time we encounter messy transition like the one at Heathrow Terminal 5, we wonder if our hard-learned lessons for dealing with complexity couldn’t be spread a bit wider.

Socio-technical Systems
Not very far from Heathrow, the Tavistock Institute of London has spent some decades researching work designs that deal effectively with turbulence and complexity. In the 1950’s and 60’s, renowned scientists such as Eric Trist and Fred Emery documented novel working arrangements that were particularly effective in the coal mines and factories of Great Britain. They found that especially effective work systems were designed (and continually improved) by semi-autonomous work teams of between 10 and 100 people that accepted responsibility for meaningful (end-to-end) tasks. The teams used their knowledge of the work and of high-level objectives to design a system to accomplish the job in a manner that optimized the overall results. Moreover, these teams were much better at managing uncertainty and rapidly adapting to various problems as they were encountered. The researchers found that the most effective work design occurs when the social aspects of the work are balanced with its technical aspects, so they called these balanced work systems Socio-technical systems.

In 1981, Eric Trist published “The Evolution of Socio-technical Systems,” an engaging history of his work. He attributes the “old paradigm” of work design to Max Weber (bureaucracy) and Frederick Taylor (work fragmentation). He proposed that a “new paradigm” would be far more effective for organizations in turbulent, competitive, or rapidly changing situations:[5]

Old Paradigm	New Paradigm
The technological imperative	Joint optimization [of social & technical systems]
Man as an extension of the machine	Man as complementary to the machine
Man as an expendable spare part	Man as a resource to be developed
Maximum task breakdown, simple narrow skills	Optimum task grouping, multiple broad skills
External controls (supervisors, specialist staffs, procedures)	Internal controls (self-regulating subsystems)
Tall organization chart, autocratic style	Flat organization chart, participative style
Competition, gamesmanship	Collaboration, collegiality
Organization’s purposes only	Members’ and society’s purposes also
Alienation	Commitment
Low risk taking	Innovation

In the 1980’s the socio-technical paradigm gained increased popularity when team work practices from Japan were widely copied in Europe and America. In the 1990’s socio-technical ideas merged with general systems theory, and the term “socio-technical systems” fell into disuse. But the ideas lived on. These days, it is generally accepted that the most effective way to deal with complex or fast-changing situations is by structuring work around semi-autonomous teams that have the leadership and training to respond effectively to any situation the groups are likely to encounter.

The clearest example we have of semi-autonomous work teams are emergency response teams – firefighters, paramedics, emergency room staff. Their job is to respond to challenging, complex, rapidly changing situations, frequently in dangerous surroundings and often with lives at stake. Emergency response teams prepare for these difficult situations by rehearsing their roles, so everyone knows what to do. During a real emergency, that training coupled with the experience of internal leaders enables the teams to respond dynamically and creatively to the emergency as events unfold.

Design Social Systems Along with Technical Systems
Developing a software system that automates a work system is fraught with just about as much danger as moving to a new airport terminal. There are many things we can do to mitigate that risk:

1. Cutover to any new system should be in small increments. Impossible? Don’t give up on increments too quickly – and don’t leave this to “customers” to decide! The technical risk of a big-bang cut-over is immense. And it’s almost always easier to divide the system in some way to facilitate incremental deployment than it is to deal with the virtually guaranteed chaos of a big-bang cutover.

2. Simplify before you automate. Never automate a work process until the work teams have devised as simple a work process as they possibly can. Automating the right thing is at least as important as automating it right.

3. Do not freeze work design into code! Leave as much work design as possible for work teams to determine and modify. If that is not possible, make sure that the people who will live with the new system are involved in the design of their work.

4. Rehearse! Don’t just test the technical part, include the people who will use the new system in end-to-end rehearsals. Be prepared to adapt the technical system to the social system and to refine the social system as well. Be sure everyone knows what to do; be sure that the new work design makes sense. Leave time to adjust and adapt. Don’t cut this part short.

5. Organize to manage complexity. Structure work around work teams that can adapt to changing situations, especially if the environment is complex, could change rapidly, or is mission critical. At minimum, have emergency response teams on hand when the new system goes live.

Much of the software we write ends up having an impact on the lives of other people; in other words, our work creates changes in social systems. We would do well to consider those social systems as we develop the technical systems. If we want to create systems that are truly successful, the technical and social aspects of our systems must be designed together and kept in balance.
___________________
Footnotes:
[1] “The opening of Heathrow Terminal 5” report to the House of Commons Transportation Committee pages 13-14.

[2] Contrast this with the absence of the BAA management team that oversaw the on-time, on-budget construction of Heathrow Terminal 5; they were replaced after a 2006 takeover of BAA by the Spanish company Ferrovial.

[3] See “The opening of Heathrow Terminal 5” report to the House of Commons Transportation Committee.

[4] “The opening of Heathrow Terminal 5” report to the House of Commons Transportation Committee pages 22-25.

[5] From "The Evolution of Socio-technical Systems" by Eric Trist, 1981. p 42.

Train-Wreck Management

“On October 5, 1841, two Western Railroad passenger trains collided somewhere between Worchester, Massachusetts and Albany, New York, killing a conductor and a passenger and injuring seventeen passengers. That disaster marked the beginning of a new management era." [1] These words open Peter Scholtes classic book on leadership. He goes on to explain how the term "management" was unknown in the days of cottage industries. As business grew and became geographically disperse in the 1800's, a way to run these businesses had to be found. But there were no models outside the church and the military, so investigators into the train-wreck disaster looked to the Prussian army for a model. And there they found the classic organization chart - the one we know so well today. Scholtes calls it the "train-wreck" chart. It was revolutionary at the time.

The purpose of what became today's organization chart was clear: The assignment of responsibility would enable "prompt detection of derelictions of duty... and point out the delinquent." Scholtes says: "A fundamental premise of the 'train-wreck' approach to management is that the primary cause of problems is 'dereliction of duty'. The purpose of the organizational chart is to sufficiently specify those duties so that management can quickly assign blame, should another accident occur."[1].

Blame
Note the thinking here: Problems are caused by people who don't do their job well, so finding someone to blame is the first step to correcting problems. Scholtes notes: "The era of management that began in the mid-1800's can be characterized as "management by results.".... Since managers could no longer do the work themselves or direct others in the doing of the work, managers exercised their authority by holding people accountable for results.... In the 1950's, management by results reached its epitome in MBO (Management By Objectives) and performance appraisals, the Harvardization of train-wreck management."[1] He goes on to say that at the time, this theory of management was the best available, and it succeeded in creating order out of chaos. "People like Whistler, McCallum, Frederick Taylor or Henry Ford in the United States or Darby, the Stephensons, or Brunel in England were pioneers.... they did their best and, by and large, what they did was very good."

"Meanwhile, in Japan...." is the title of the next section Scholtes' book. He chronicles how a better approach to management emerged in Japan in the 1950's assisted by W. Edwards Deming. Deming taught that most of the problems we encounter (perhaps 90%) are the result of multiple influences, they generally cannot be attributed to a single cause. Assigning blame for a problem to the last person involved is worse than counterproductive, it will probably make the bad situation worse. Exhorting people to "be careful," "try harder," and "work smarter" is not useful if individuals have little effect on results. Rewarding or punishing people for outcomes that are not under their control can only result in discouragement - or in gaming the system. Instead, chronic problems must be fixed by finding their underlying causes and addressing these effectively. As Deming points out, this usually involves changing the system - the way things are done. And according to Deming, it is management's job to change the system.

Process or People?
Agile software development places a strong emphasis on putting change into the hands of front-line people on self-directed teams - isn't this contrary to Deming's philosophy? Writing in 1995, Scholtes lists what he calls "fads" for addressing systemic problems: "empower people, put them into self-directed teams, motivate them, offer incentives, reengineer and reinvent them." And then he says: "All of the empowered, motivated, teamed-up, self-directed, incentivized, accountable, reengineered, and reinvented people you can muster cannot compensate for a dysfunctional system.... A well-run organization with well-functioning systems allows people from top to bottom do work of which they can be proud."[1] So where does this leave us? Which is more important - process or people?

It helps if we trade in the overloaded word "process" and use "system."

In the article "Managing a Living System, not a Ledger"[2] H. Thomas Johnson says "Managers at Toyota believe that improving the system is the surest way to improve long term financial results." He points out that Toyota takes lots and lots of measurements, but they do not use these as performance measurements. Johnson writes: "...Toyota makes virtually no use of management accounting targets (or 'levers') to control or motivate operations... Toyota focuses its operations on continuous system improvement through endless rapid problem solving. And they emphasize genchi genbutsu, or 'going to the place,' to see where a problem occurs, firsthand. They don't rely on second-hand reports or tables and charts of data to achieve a true understanding of root cause. Instead they go to the place (gemba) where you can watch, observe, and 'ask why five times.' This attitude shows a deep appreciation that results (and problems) ultimately emanate from, and are explained by, complex processes and concrete relationships, not by abstract, quantitative relationships that describe results in simple, linear, additive terms." Winding up the article, Johnson says: "Financial quantities cannot reveal if a system is improving or not... No company that talks about improving performance can know what it is doing if its primary window on results is financial information and not system principles.... Companies that intend to perform like Toyota should recognize that... they will never get there by trying to motivate and direct 'lean' initiatives with 'lean accounting' and management accounting 'levers of control.'"

Taiichi Ohno on Standard Work
Let's go back to the source of the Toyota Production System, Taiichi Ohno, and see what he had to say about process - how it is established and how it is changed.[3]

There is something called standard work, but standards should be changed constantly. Instead, if you think of the standard as the best you can do, it's all over. The standard work is only a baseline for doing further kaizen. It is kai-aku [change for the worse] if things get worse than now, and it is kaizen [change for the better] if things get better than now. Standards are set arbitrarily by humans, so how can they not change?

When creating Standard Work, it will be difficult to establish a standard if you are trying to achieve 'the best way.' This is a big mistake. Document exactly what you are doing now. If you make it better than it is now, it is kaizen. If not, and you establish the best possible way, the motivation for kaizen will be gone. That is why one way of motivating people to do kaizen is to create a poor standard. But don't make it too bad. Without some standard, you can't say 'We made it better' because there is nothing to compare it to, so you must create a standard for comparison.

Take that standard, and if the work is not easy to perform, give many suggestions and do kaizen.

We need to use the words 'you made' as in 'follow the decisions you made.' When we say 'they were made' people feel like it was forced upon them. When a decision is made, we need to ask who made the decision. Since you also have the authority to decide, if you decide, you must at least follow your decision, and then this will not be forced upon you at all.

But in the beginning, you must perform the Standard Work, and as you do, you should find things you don't like, and you will think of one kaizen idea after another. Then you should implement these ideas right away, and make this the new standard.

Years ago, I made them hang the standard work documents on the shop floor. After a year I said to a team leader, 'The color of the paper has changed, which means you have been doing it the same way, so you have been a salary thief for the last year.' I said 'What do you come to work to do each day? If you are observing every day you ought to be finding things you don't like, and rewriting the standard immediately. Even if the document hanging there is from last month, this is wrong.' At Toyota in the beginning we had the team leaders write down the dates on the standard work sheets when they hung them. This gave me a good reason to scold the team leaders, saying 'Have you been goofing off all month?'

If it takes one or two months to create these documents, this is nonsense. You should not create these away from the job. See what is happening on the gemba and write it down.

Process AND People
Ohno believed that the primary job of team leaders (first line supervisors) is the constant improvement of the way work gets done. Work standards should be written and posted, but this had better not take very long because the standards should change all the time - at least once a month. Standards are not about how work should be done, but how work is being done. You don't want the standard to be too perfect, because that leaves no incentive for workers to improve their standards. If workers are annoyed by a standard, they are expected to change it. They do not drop a suggestion in a suggestion box, they do kaizen. That is, workers - led by their team leader - do many rapid experiments, find a better way, agree on the improvement, quickly document the new way, and use it. When a standard is improved, the decision for the change must be made by the people doing the work, so they won't feel it is being forced upon them.

People like to use effective processes, and they also like to have control over their own environment. The Toyota Production System provides for both. Ohno made it clear that people must be at the center of improving their own processes. Process improvement may be done only "at the gemba" and it is up to the workers to decide whether or not a proposed improvement should be implemented. Workers are expected to keep changing the way they do their job; in fact, it is bad leadership to have a process so perfect that workers have little incentive to improve it!

Assessment and Certification
Scholtes takes process improvement assessment programs such as ISO 9000 to task because even though they seem good on the surface, they have some problems:[1]

1. The pursuit of quality must be guided by a larger context than certification - it requires a holistic, integrated, long term commitment.
2. Certification is not equal to satisfied customers - you can do the wrong thing as long as you do it consistently.
3. Assessment has a tone of paternalism and mistrust - it replaces internal motivation with external motivation.
4. Assessment assumes that inspectors are all the same - but inspections are not standardized.
5. A certified process is difficult to change - Ohno would be appalled.

Conclusion
When Deming said "change the system", he was talking about changing the complex, interrelated processes used to get work done. Deming believed that changing the system is management's primary job, and in order to do this, managers need competency in four areas:

1. Appreciation for the overall system in which work is done
2. An understanding of variation - and the true relationship between cause and effect
3. Constant pursuit of learning (improvement) through designed experiments
4. An understanding of the psychology of people

When all of these areas are balanced and working together, great things can happen.

References
[1] The Leader's Handbook, by Peter R. Scholtes, McGraw-Hill, 1998.

[2] "Managing a Living System, Not a Ledger,", by H. Thomas Johnson, Lean Manufacturing 2007, Supplement to Manufacturing Engineering, SME, August 2007. Johnson also coauthored Profit Beyond Measure: Extraordinary Results through Attention to Work and People, Free Press, November, 2000.

[3] Workplace Management, by Taiichi Ohno, originally published in 1982, from translation by Jon Miller, Gemba Press, 2007.

Screen Beans Art, © A Bit Better Corporation

Cause and Effect

Is low cost achieved by focusing on cutting costs?
Is high utilization achieved by trying to utilize resources full time?
Does standardized work mean that work processes are followed without challenge?
Does everyone in your organization agree on the answers to these questions?

Cost
When Jane Beseda took over Toyota’s North American Parts Operation (NAPO), she knew that really dramatic results required breaking down the barriers between departments [1].  So she set three year Stretch Goals that were all but impossible – a 50% reduction in inventory, 25% increase in throughput, 25% reduction in freight costs, 50% reduction in packaging expense, 25% increase in space utilization, 50% decrease in backorders.  After a year of effort, department managers began to realize that that they were not going to achieve the Stretch Goal targets unless they changed their focus to cross-organizational projects.  Project teams had been struggling to coordinate their work through the traditional functional department planning approach. This was changed; the departments started to consider potential impact of their plans on other functions and areas of the broader organization. Only then were truly significant gains in eliminating waste and reducing cost realized. After three years, the results at NAPO were truly amazing; almost all of the Stretch Goals were achieved. But this would never have happened if everyone in the organization had not focused on overall system waste rather than individual department costs.

Low overall costs rarely come from lowering costs in individual departments; they come from lowering system costs. Lean companies have learned that this requires a keen understanding of underlying cost drivers and a system-wide effort to eliminate waste.  Consider Zara, a huge women’s fashion clothing chain headquartered in western Spain. It fills retail store orders twice a week, shipping garments around the world in two days – not folded compactly in boxes – but ironed and hanging on hangers ready to display. Zara realizes that it lowers overall costs by getting the clothing that women are asking for on shelves very rapidly.  Zara has fewer markdowns and unsold clothes, and it drives more business to stores with less advertising than its competitors. These tremendous savings would disappear if Zara focused on reducing shipping costs. This is systems thinking at work.

Over time, individual departments will find ways to drive out costs, but rarely do organizations attack the larger costs that occur between departments, divisions, functions, and companies.  A quick and easy cost reduction approach for a department might be to outsource work to obtain lower labor costs.  However, in most cases, outsourcing simply adds more boundaries to cross, and boundaries typically add 20 to 30% to overall costs [2]. One company I know outsourced the process of engaging contractors.  Since training is included in this outsourcing arrangement, all trainers must work through the outsourced contractor management company.  The amount of time and paperwork necessary to set up a training engagement at this company is about an order of magnitude greater than at any other company I have dealt with. However, the arrangement appears to be a cost reduction for the company, because contractors pay the contractor management firm through a fee imposed on their payment. This financial arrangement hides the tremendous waste in the engagement process, which includes a great deal of problem resolution by employees in the company receiving the training. But because of organizational barriers, these same employees have no way to change or improve a very burdensome process.

Standardized Work
In studying Toyota, many companies notice that standardized work processes are a key element of the company’s success.  What they generally miss is that at Toyota, standards exist to be challenged and changed by the front line employees doing the work.  Toyota actively encourages all employees to question any part of their job that is annoying or gets in the way of doing a good job. Employees are expected to look for a better way, prove with experiments that the new way is better, and then implement a new standard.  Taiichi Ohno, father of the Toyota Production System, wrote: “Something is wrong if workers do not look around each day, find things that are tedious or boring, and then rewrite the procedures. Even last month’s manual should be out of date.”

There are those who believe that standardized processes should be imposed from the outside and followed without question. However, companies with a Lean perspective believe that the essence of eliminating waste is to encourage everyone to attack and change the things that annoy them about their jobs. Only by engaging people in doing their job better on a daily basis can you get the sustainable gains that companies like Toyota enjoy. Work standards in a lean company spell out the current best known way of doing things, and they are a baseline for change. New standards are developed through constant experimentation by work teams, using the current work standards as the baseline against which improvement is measured.

At Toyota’s Georgetown, Kentucky plant the paint department has always been the biggest bottleneck preventing one-piece-flow of vehicles through the plant.  Over the past few years, however, things began to change [3].  First the people in the department devised a way to paint any color in any order – instead of piping paint to a robot through a tube that needs cleaning between colors, they now pipe each paint color to a canister just the right size to paint a car.  The robot picks up the right canister and Viola! the cost and time needed to clean out paint lines is eliminated.  The bottom line:  30% less paint used, a huge drop in the use of cleaning solvents, and cars can now be painted any color in any order. The throughput of the shop has increased from 33 to 50 an hour while the space required for painting was reduced by a third and employees were freed up for other work.

At most auto companies, this achievement would be cause for celebration – but at Toyota, constant improvement is the day-in-day-out job of the paint manager. There are no black belts, no outside process experts – just the people doing the work led by their first line managers. Everyone’s job is to keep on improving their work processes, every day, every week, every month. All workers are engaged in the relentless pursuit of perfection – the elusive goal which keeps everyone looking forward rather than patting themselves on the back.

In software development, we have been led to believe that “maturity” involves documenting best practices and making sure that they are followed. On the contrary, a truly mature organization expects development teams to constantly challenge and improve their processes.  The day an organization settles into believing that it is perfect is the day it invites its people to stop thinking. Real, sustainable improvement on all fronts comes only when all workers are expected to challenge and fix anything about their job that annoys them or keeps them from taking pride in their work.

Utilization
Managers in manufacturing plants, supervisors of computer operations, and highway engineers agree on one cause and effect relationship:  if your resources approach full utilization, response time slows to a crawl. Queuing theory is widely applied to servers and highways and manufacturing equipment.  However, in a development environment, managers seem to be unaware that attempting to achieve full utilization slows response time to a crawl and decreases actual utilization. Development organizations are no more immune from the laws of queuing theory than airlines. We see how constantly full airlines with no slack cause cascading problems whenever a slight perturbation is introduced into the airline system. And weather being what it is, perturbations happen all the time.

Similarly, when a slight perturbation is introduced into a fully scheduled development organization, cascading problems are inevitable. Knowledge work being what it is, there will always be perturbations to the most carefully laid out schedules. The only way to deal effectively with these perturbations is to follow the advice of queuing theory:  work in small batches, minimize the length of queues of work to be done, and never, ever schedule an organization beyond its capacity to deliver.  Do this and utilization will increase.  Focus directly on utilization and it is guaranteed to be sub-optimal.

Many organizations queue up requests for software development into large batches of work to be done, presumably because better decisions can be made if the whole picture is visible before work begins.  Challenge this assumption.  Why is it better to respond very slowly to a pile of accumulated requests rather than respond very quickly to the current most important outstanding request? What good, really, does it do to have a long list of work to do?  Looked at closely, it's hard to make a case for piling up lists of work-to-do that are far longer than you have a hope of accomplishing. You're pretty much wasting your time keeping track of stuff you can never get around to, and you're probably setting incorrect expectations on the part of the requestors. Just say no.

Unfortunately, once software has been around for a while, we often see that it takes longer and longer to run the regression tests as each new feature is added, since every feature increases the regression test load.  We call this increasing regression load the regression deficit, and unless it is systematically tackled and reduced, the code base will become increasingly difficult to change.  You will be tempted to release software in larger and larger batches, because regression testing takes so long. Similar to the paint shop manager, a software development manager’s job should be to help development teams chip away at the regression deficit every day, every week, every month, until throughput is increased and one-piece-flow of small feature sets becomes practical.

Cause & Effect
In order for organizations to perform brilliantly, there are two prerequisites:  First, everyone has to agree on what they want, and second everyone has to agree on cause and effect [4].  Let’s assume that everyone in your organization agrees on the results they want, their values and priorities, and the trade-offs they are willing to make in order to achieve those results.  The question to ask is – does everyone agree on cause and effect?

Is there general agreement on what actions will result in system-wide cost reduction?  Are individual departments expected to reduce costs independently, or is it clear that departments must work together to reduce overall system costs, even at the expense of costs in individual departments? Are the measurements in place to reduce the costs of crossing boundaries?

Is there general agreement on the best way to achieve consistent results through standardization?  Are standards always followed?  If not, do you really understand the root cause – are the standards irrelevant or inaccurate, too complex, or too far from the actual work being done? Are standards maintained by a central process group, or are they continually improved by the people doing the work?  Are front line people expected and encouraged to continually challenge and change standards?

Do you have a project scheduling system aimed at optimizing utilization?  Does the management team believe that the system is in fact optimizing utilization? Do people believe that full utilization is the right measurement to emphasize? Is time-to-market an important factor in your business? Does everyone agree on the relationship between time-to-market and utilization?

Lean thinking is counterintuitive; it flies in the face of conventional wisdom concerning cause and effect. No matter how well proven the results, no matter how intellectually sound the arguments of systems thinking are, it is almost impossible for those who have been successful with conventional wisdom to change their habits. It is very difficult for leaders to address the system costs that flourish between department boundaries if they believe that lowering costs in each department will add up to lower overall costs. When leaders believe that standardized processes are an end rather than a beginning, they have a difficult time leveraging the wisdom of front line employees. For those leaders who believe that focusing on full utilization will increase productivity, the outstanding productivity gains that come from focusing instead on throughput will not be available.  It all comes down to measures – small measures cause small results; global measures promote globally optimized results.

Those who have seen the dramatic results of Lean Thinking have come to believe that concentrating on throughput, making front line employees the center of focus of the company, and eliminating waste across the system are the tools of choice in a competitive environment.  Most management teams that have been threatened by fierce competition and survived have changed their habits and adopted the counterintuitive tenets of Lean Thinking. Virtually all companies facing fierce competition from Lean companies that have failed to adopt similar thinking have failed to thrive in the long run.
_________________
Footnotes:

[1] This story is from The Elegant Solution: Toyota’s Formula for Mastering Innovation, by Matthew E. May, Free Press, 2007, p. 139.

[2] Management Challenges for the 21st Century, by Peter Drucker, HarperBusiness, 2001, p. 33.

[3] From “No Satisfaction” by Charles Fishman, Fast Company, Dec 2006 / Jan 2007

[4] See “The Tools of Cooperation and Change,” by Clayton Christensen and co authors, Harvard Business Review, October 2006.


Screen Beans Art, © A Bit Better Corporation

Toward A New Definition of Maturity

In the mid 90's, a company named Zeos assembled PC's in my home state of Minnesota. I was impressed when Zeos was named as a finalist for the Malcolm Baldrige Award, because competing for this quality award is more or less the equivalent of a software company trying to reach CMM Level 5. At the same time that Zeos was focusing on the Malcolm Baldrige Award, a similar company in Austin, Texas, called Dell Computer, was focusing all of its energy on two rather different objectives: 1) keep inventory as low as possible, because it is the biggest risk in the PC assembly business, and 2) deliver computers just as fast as possible after customers decide what they want. Dell could not possibly meet these goals unless it had the key Malcolm Baldrige capabilities in place, but that was not its focus. On the other hand, in order to be a finalist, Zeos had to spend huge amounts of executive and management attention on Malcolm Baldrige activities. So which of these two companies would you call the most mature?

A company does not rise to the top of its industry by perfecting its normative procedures. While General Motors was busy refining it’s four phase process for product development, Toyota and Honda were developing cars in something approaching half the time, for half the cost. The resulting cars were less expensive to produce and captured a much larger market share. Yet when looked at through Detroit’s eyes, the development approach of the Japanese companies was decidedly immature – why, they even let die makers start cutting dies long before the final drawings were released!

The problem with maturity models is that they foster a mental model of ‘good practice’ that can block out paradigm shifts that are destined to reshape the industry. This has happened in manufacturing. It has happened in product development. Can it be happening in software development today?

Why Do We Do This To Ourselves?
We human beings a tendency to decompose complex problems into smaller pieces, and then focus on the individual pieces, often, unwittingly, at the expense of the whole. One reason for this is that people can only hold a few chunks of information in short term memory at once; Miller’s law claims that this number is seven plus or minus two. Thus we have a strong tendency to take concepts which we cannot get our minds around and decompose them into parts, so we can deal with one part at a time. There is nothing wrong with our very natural tendency to decompose problems, except that we often forget to reconfigure the parts into a whole and take their interactions into account.

The problem is, decomposition only works if the whole is indeed equal to the sum of its parts, and interactions between the parts can be ignored. In practice, this is rarely the case; in fact, optimization of the parts has a tendency to sub-optimize the whole. For example, a manager might think that the best way to run a testing department is to make sure that every single person is working all of the time. In order to guarantee maximum productivity of each individual, he makes sure there is always a pile of testing waiting to be done. Down the hall, the operations manager knows better. She knows that if she runs the servers at high utilization, the entire system bogs down, just like rush hour traffic. If only the testing manager would realize that his policy of full utilization of testing resources creates the same kind of traffic jam in software development!

Decomposition of a problem into pieces is a standard problem solving heuristic, but it needs to be accompanied by regular aggregation of the parts into a whole. It is here that iterative development shines, because it forces us to develop, test, integrate, and release complete threads of the system early and often. Thus we reset our view of the big picture every iteration. However, iterative development requires that we decompose the problem space along dimensions that are orthogonal to those traditionally recommended. Instead of decomposing the problem into requirements, analysis, design, programming, testing, integration, deployment, we decompose the problem into features along the natural fault lines of the domain.

Decomposition vs. Abstraction
There is an alternative to the decomposition of a problem into components, and that is to abstract the problem to a higher level, rather than decompose it to a lower level. The result is the same in both cases – you have reduced the number of things you need to think about to 7 +/- 2. But when you move the problem up to a higher level of abstraction, the interaction of the parts is maintained, so abstractions would seem to be the better approach to solving complex problems.

There is a catch: people can’t create abstractions without understanding the domain, because correct abstractions require that the abstractor knows what is important, what fits together naturally, and where to find the natural joints or fault lines of the domain. For this reason, experts in the domain have a greater tendency to abstract to a higher level, while those who are not familiar with the domain tend to decompose the problem, and that decomposition will probably not be along the natural fault lines of the domain.

It is when we decompose a problem in a manner which does not fit the domain, and then drill down to the details too fast, that important things get missed. We would like to think that a lot of early detail will help us find all the hidden gremlins that might bite us later. This is only true if those gremlins are actually inside the areas we investigate – but the tough problems usually lurk between the cracks in our thinking. Unfortunately we only discover those problems when we integrate the pieces together at the end, and by that time we have invested so much in the details that change is very difficult.

It is safer to work from a higher level or abstraction, but this means we don’t drill down to detail right away. We take a breadth-first approach, keep options open and gradually fill in the details. With this approach we are more likely to find those gremlins while we still can do something about them, but some people might think that we don’t have a plan, aren’t tracking the project, and can’t manage requirements.

Assessment Rather than Certification
Quite frequently CMM is implemented as a certification process, decomposing ‘mature’ into separately verifiable capabilities. The danger is that we may lose sight of forest for the trees. For example, focusing on requirements management can be detrimental to giving users what they really want. Focusing on quality assurance separate from development can destroy the integrating effect of developers and testers working side-by-side every day. Focusing on planning as a predictive process can keep us from using planning as an organizing process.

A better approach to discovering the underlying competence of an organization is to use an assessment process rather than a certification process. Assessments present challenging situations that cannot be dealt with unless a host of capabilities are in place; if the challenge is properly handled, the presence of these capabilities can be inferred. For example, a pilot’s ability to fly a plane can be assessed by observing how the pilot lands a plane in a stiff cross wind.

Consider your hiring process. When a candidate lists Microsoft or PMI certifications, you take that into account, but you are really looking for a track record of success which demonstrates that the certifications were put to good use. One software company I know administers a one hour logic tests to job applicants, a test without a line of code in it. Yet the test does an admirable job of assessing the capability of the candidate to think like a good developer.

Measure UP
If we accept that we get what we measure then, as Rob Austin points out in “Measuring and Managing Performance in Organizations,” we do not get what we don’t measure. If we are not measuring everything, if something important just didn’t make it into our measurement system, then we are not going to get it. To counter this problem, we have a tendency to pile one measurement on top of another, every time we discover that we forgot to measure something. In CMM, for example, each KPA addresses something that caused a failure of some software project somewhere. As people discovered new ways for software projects to fail, a KPA’s was added to address the new failure mode. Nevertheless, all of those KPA’s still don’t cover everything that could go wrong, although they certainly try.

Once we admit that we just can’t measure everything, then we are ready to move to assessment-style measurements rather than a certification-style measurements. For example, in project management, we decompose the measurement system into cost, schedule, scope and defects, and we try hard to make these measurements work because we think they are the measurements we should use. But no matter how hard we try, we are often unsuccessful in measuring true business value by totaling up the yardsticks of cost, schedule, scope and defects.

What if we just measured business value instead of cost, schedule, scope, and defects? When I was developing new products at 3M, we did not pay much attention to cost, schedule and scope. Instead we developed a P&L which we would use to check out the impact of a late introduction date or a lower unit cost. The development team tried to optimize the overall P&L, not any one dimension. It may seem strange that a project team would concern itself with the financial model of the business, but I assure you it is far better decision-support tool than cost, schedule and scope.

The Measure of Maturity
I believe that our industry could use a simpler measurement of software development maturity, one that is not subject to the dangers of decomposition and does not attempt to defy Miller’s Law with a long lists of things to think about. I propose that we use a measurement that has been successfully used in countless organizations, and has proven to be a true indicator of the presence of the kind of capabilities measured by CMM. To introduce this measurement, let’s go back to Dell and see what it measured:

1. The level of inventory throughout the entire system, and
2. The speed with which the organization can repeatedly and reliably respond to a customer request.

Okay, you say, fine for manufacturing, but software development is different.

True, I counter, but these measurements still work.

The level of inventory in your system is the amount of stuff you have under development. The more inventory of unfinished development work you have sitting around, the more you are at risk of it growing obsolete, getting lost, and hiding defects. If you capitalized it, you also bear the risk of having to write it off if it doesn’t work. The less of that kind of stuff you have on hand, the better off you will be.

The speed with which you can responds to a customer is directly proportional to the amount of unfinished development work you have clogging up your system. In truth, the two measurements above are one and the same – you can deliver faster if your system is not clogged with unfinished work. But more to the point, you cannot reliably and repeatedly deliver fast if you do not have a mature organization. You need version control, built-in quality, ways to gather requirements fast and routinely translate them correctly into code. You need everything you measure in CMM, and you need all of those capabilities working together as an integrated whole. In short, the speed with which you can repeatedly and reliably deliver on customers requests is a better measure of the maturity of your software development organization.

Published as “Measuring Maturity” in Software Development Magazine, February 2004.

Screen Beans Art, © A Bit Better Corporation

Lean Software Development

I often hear developers say: Life would be so much easier if customers would just stop changing their minds. In fact, there is a very easy way to keep customers from changing their minds – give them what they ask for so fast that they don’t have the time to change their minds.

When you order something on-line, it’s usually shipped before you have time for second thoughts. Because it ships so fast, you can wait until you’re sure of what you want before you order. But once you place the order, you have very little time to change your mind.

The idea behind lean thinking is exactly this: Let customers delay their decisions about exactly what they want as long as possible, and when they ask for something, give it to them so fast they don’t have time to change their minds.

Sure, you say, this is fine for Amazon.com, but how does it relate to software development? The way to deliver things rapidly is to deliver them in small packages. The bigger the increment of functionality you try to deliver, the longer it takes to decide what is needed and then get it developed, tested, and deployed. Maintenance programmers have known this for years. When a piece of production code breaks, they find the cause, create a patch, test it rigorously, and release it to production – usually in the space of a few hours or days.

But, you say, development is different; we need to develop a large system, we need a design, and we can’t deploy the system piecemeal. Even large systems that have to be deployed all-at-once should be developed in small increments. True, design is needed, but there is scant proof that great designs are achieved by meticulously gathering detailed requirements and analyzing them to death. Great designs come from great designers, and great designers understand that designs emerge as they develop a growing understanding of the problem.

As Harold Thimbleby said in Delaying Commitment [1], “In many disciplines, the difference between amateurs and experts seems to be that experts know how to delay their commitments…. Amateurs, on the other hand, try to get things completely right the first time and often fail… in their anxious desire to avoid error, they make early commitments – often the wrong ones… In fact, the expert’s strategy of postponing firm decisions, discovering constraints, and then filling in the details is a standard heuristic to solve problems.”

Lean Thinking
Over the last two decades, Lean Thinking has created fundamental transformations in the way we organize manufacturing, logistics, and product development. For example, concurrent development has replaced sequential development in industries from airlines to automobiles, cutting product development time by perhaps a third and development cost in half, while improving the quality and timeliness of the product. There are those who think that rapid development is equivalent to shoddy work, but lean organizations have demonstrated quite the opposite. The measure of the maturity of an organization is the speed with which it can reliably and repeatedly respond to a customer request.

Yes, you heard that right. Maturity is not measured by the comprehensiveness of process documentation or the ability to make detailed plans and follow them. It is measured by operational excellence, and the key indicator of operational excellence is the speed with which the organization can reliably and repeatedly serve its customers. Thirty years ago, Frederick W. Smith, the founder of FedEx, envisioned an overnight delivery system the seemed ridiculous to most people; today even the post office offers overnight delivery. A decade ago, Toyota could develop a new car twice as fast as GM; today all automotive companies measure and brag about the speed with which they can bring a new car to market.

Let’s revisit those customers you wish would make up their minds and stick to their decisions. Ask yourself this question: Once they do make up their minds, how fast can you reliably and repeatedly deliver what they want? Perhaps the problem does not lie in customers who can’t make up their minds and keep changing their decisions. Perhaps the problem lies in asking them to decide well before they have the necessary information to make a decision, and in taking so long to deliver that their circumstances change.

Principles of Lean Software Development
There are seven principles of Lean Software Development, drawn from the seven principles of Lean Thinking. These principles are not cook-book recipes for software development, but guideposts for devising appropriate practices for your environment.[2] These lean principles have led to dramatic improvements in areas as diverse as military logistics, health care delivery, building construction, and product development. When wisely translated to your environment, they can change your basis of competition.

Eliminate Waste
All lean thinking starts with a re-examination of what waste is and an aggressive campaign to eliminate it. Quite simply, anything you do that does not add value from the customer perspective is waste. The seven wastes of software development are: 

• Partially Done Work (the “inventory” of a development process)
• Extra Processes (easy to find in documentation-centric development)
• Extra Features (develop only what customers want right now)
• Task Switching (everyone should do one thing at a time)
• Waiting (for instructions, for information)
• Handoffs (tons of tacit knowledge gets lost)
• Defects (at least defects that are not quickly caught by a test)

All lean approaches focus on eliminating waste by looking at the flow of value from request to delivery. So if a customer wants something, what steps does that customer request have to go thorough to get delivered to the customer? How fast does that process flow? If a customer request waits in a queue for approval, a queue for design, a queue for development, a queue for testing, and a queue for deployment, work does not flow very fast. The idea is to create cells (or teams) of people chartered to take each request from cradle to grave, rapidly and without interruption. Then value flows.

Amplify Learning
The game of development is a learning game: hypothesize what might work, experiment to see if it works, learn from the results, do it again. People who design experiments know that greatest learning occurs when half of the experiments fail, because this exposes the boundary conditions. A development environment is not a place for slogans such as:

• Plan The Work And Work The Plan
• Do It Right The First Time
• Eliminate Variability

The learning that comes from short feedback loops is critical to any process with inherent variation. The idea is not to eliminate variation, it is to adapt to variation through feedback. Your car’s cruise control adapts to hilly terrain by frequently measuring the difference desired speed and actual speed and adjusting the accelerator. Similarly, software development uses frequent iterations to measures the difference between what the software can do and what the customer wants, and makes adjustments accordingly.

A lean development environment focuses on increasing feedback, and thus learning. The primary way to do this in software development is with short, full-cycle iterations. Short means a week to a month. Full cycle means the iteration results in working software: tested, integrated, deployable code. There should be a bias to deploy each iteration into production, but when that is not possible, end users should simulate use of the software in a production-equivalent environment.

We have known for a long time that iterative (evolutionary) development is the best approach for software development. In 1987 the report of the Defense Science Board Task Force on Military Software noted: “Document-driven, specify-then-build approach lies at the heart of so many DoD software problems…. Evolutionary development is best technically, and it saves time and money.”[3]


Delay Commitment
Delaying commitment means keeping your options open as long as possible. The fundamental lean concept is to delay irreversible decisions until they can be made based on known events, rather than forecasts. Economic markets develop options as a way to deal with uncertainty. Farmers, for example, can buy an option on the future price of grain. If the price drops, they have protected their profits. If the price raises, they can ignore the option and sell at the higher price. Options let people delay decisions until they have more information.

There are a lot of ways to keep options open in software development. Here are a few:

• Share partially complete design information.
• Organize for direct, worker-to-worker collaboration.
• Develop a sense of when decisions must be made.
• Develop a sense of how to absorb changes.
• Avoid Repetition
• Separate Concerns
• Encapsulate Variation
• Defer Implementation of Future Capabilities
• Commit to Refactoring
• Use Automated Test Suites

Deliver Fast
To those who equate rapid software development with hacking, there seems to be no reason to deliver results fast, and every reason to be slow and careful. Similarly, when Just-in-Time concepts surfaced in Japan in the early 1980’s, most western manufacturers could not fathom why they made sense. Everybody knew that the way to make customers happy was to have plenty of product on the shelf, and the way to maximize profits was to build massive machines and keep them busy around the clock. It took a long time for people to realize that this conventional wisdom was wrong.

The goal is to let your customer take an options-based approach to making decisions, letting them delay their decisions a long as possible so they can make decisions based on the best possible information. Once your customers decide what they want, your goal should be to create that value just as fast as possible. This means no delay in deciding what requests to approve, no delay in staffing, immediate clarification of requirements, no time-consuming handoffs, no delay in testing, no delay for integration, no delay in deployment. In a mature software development organization, all of this happens in one smooth, rapid flow in response to a customer need.

Empower the Team
In a lean organization, things moves fast, so decisions about what to do have to be made by the people doing the work. Flow in a lean organization is based on local signaling and commitment amongst team members, not on management directives. The work team designs its own processes, makes its own commitments, gathers the information needed to reach its goals, and polices itself to meet its milestones.

Wait a minute, you say. Why would I want to be empowered? If I make the decisions, then I’ll get blamed when things go wrong. A team is not empowered unless it has the training, expertise, and leadership necessary to do the job at hand. But once those elements are in place, a working team is far better equipped to make decisions than those who are not involved in the day-to-day activities of developing software. It is true that decision-making carries greater responsibility, but it also brings significantly greater influence on the course of events and the success of the development effort.

Consider an emergency response unit such as firefighters. Members receive extensive training, both in the classroom and on the job. Exercises and experience imprint patterns of how to respond to difficult situations. When an emergency occurs, the team responds to the situation as it unfolds; there is little time to ask remote commanders what to do, nor would it make sense. The important thing is that the responding teams have the training, organization and support to assess the situation as they encounter it and make basically correct decisions on their own.

Similarly in software, the development team is in the best position to know how to respond to difficult problems and urgent requests. The best way to be sure that you get things right is to work directly with customers to understand their needs, collaborate with colleagues to figure out how to meet those needs, and frequently present the results to customers to be sure you are on the right track. Management’s job to supply the organization, training, expertise, leadership, and information so that you generally make the right decisions, make rapid course corrections as you learn, and end up with a successful outcome.

Build Integrity In
There are two kinds of integrity – perceived integrity and conceptual integrity. Software with perceived integrity delights the customer – it’s exactly what they want even though they didn’t know how to ask for it. Google comes to mind – when I use Google I imagine that the designers must have gotten inside my head when they added the spelling checker. I would never have known to ask for this feature, but these days I type most URLs into the Google toolbar because if I mistype, Google will set me straight.

The way to achieve perceived integrity is to have continuous, detailed information flow from the users, or user proxies, to the developers. This is often done by an architect or master designer who understands the user domain in detail and makes sure that the developers always have the real user needs in front of them as they make day-to-day design decisions. Note that the technical leader facilitates information flow and domain understanding, she or he is intimately involved in the day-to-day work of the developers, keeping the customer needs always in mind. However, it is the developers, not the leader, who make the detailed, day-to-day decisions and tradeoffs that shape the system.

Conceptual integrity means that all of the parts of a software system work together to achieve a smooth, well functioning whole. Software with conceptual integrity presents the user with a single metaphor of how a task is to be done. You don’t have one way of buying airline tickets if you are paying with cash and a totally different way if you are using frequent flier miles. You don’t use kilometers in one module and miles in another.

Conceptual integrity is achieved through continuous, detailed information flow between various technical people working on a system. There are no two ways about it, people have to talk to each other, early and often. There can be no throwing things over the wall, no lines between supplier, development team, support team, customer. Everyone should be involved in detailed discussions of the design as it develops, from the earliest days of the program.

For example, Boeing credits its rapid and successful development of the 777 to its ‘Working Together’ program, where customers, designers, suppliers, manufacturers, and support teams all met from the start to design the plane. Early on it was discovered that the fuel tank was beyond the reach of all existing refueling trucks, a mistake that was easily fixed. In a normal development process this expensive mistake would not have been discovered until someone tried to fuel the first plane.

There are those who believe that software integrity comes from a documentation-centric approach to development: define the requirements in detail and trace every bit of code back to those requirements. In fact, such an approach tends to interfere with the technical communication that is essential to integrity. Instead of a documentation-centric approach, use a test-centric approach to integrity. Test early, test often, test exhaustively, and make sure an automated test suite is delivered as part of the product.

See the Whole
When you look at them closely, most theories of how to manage software projects are based on a theory of disaggregation: break the whole into individual parts and optimize each one. Lean thinking suggests that optimizing individual parts almost always leads to sub-optimized overall system.

Optimizing the use of testing resources, for example, decreases the ability of the overall system to rapidly produce tested, working code. Measuring an individual’s ability to produce code without defects ignores the well-known fact that about 80% of defects are caused by the way the system works, and hence are management problems.

The best way to avoid sub-optimization and encourage collaboration is to make people accountable for what they can influence, not just what they can control. This means measuring performance one level higher than you would expect. Measure the team’s defect count, not that of individuals. Make testing as accountable for defect-free code as developers. To some it seems unfair to hold a team accountable for individual performance, but lean organizations have found that individuals are rarely able to change the system which influences their performance. However a team, working together and responsible for its own processes, can and will make dramatic improvements.

Conclusion
To keep customers from changing their minds, raise the maturity of your organization to the level where it can reliably deliver what customers want so fast that they have no time to change their minds. Focus on value, flow, and people, the rest will take care of itself
_________________
References
[1]Harold Thimbleby, "Delaying Commitment" IEEE Software, May 1988.

[2]The book Lean Software Development: An Agile Toolkit by Mary Poppendieck and Tom Poppendieck, Addison-Wesley Professional, 2003, provides twenty two tools for converting lean principles into agile software development practices.

[3]Craig Larman, “A History of Iterative and Incremental Development”, IEEE Computer, June 2003

Published in C++ Magazine, Methodology Issue, Fall 2003

Screen Beans Art, © A Bit Better Corporation