Friction

One third of the fuel that goes into a car is spent overcoming friction. By comparison, an electric car loses half as much energy - one sixth - to friction. Who knew electric cars had such an advantage?

Friction is the force that resists motion when the surface of one object comes into contact with the surface of another. You can imagine parts moving against one another in cars, but do you ever wonder what happens when your products and services come in contact with customers? Might this create some friction? Could there be competing offerings that create considerably less friction? If so, you can be sure your customers will find the low friction offering more attractive than yours.

Friction in the Customer Journey

Think of friction as the cognitive overhead that a system places on those who use it. Let’s consider the friction involved in taking a taxi from an airport to a hotel. When I arrive at most airports, I get in a taxi queue, heeding the conspicuous warnings not to ride with unauthorized drivers. When I reach the front of the queue, I take the next taxi in line, and I assume that the cost is the same no matter which taxi I take. But this is not true in Stockholm, where taxis can charge any rate they wish simply by posting it in the window. Nor is it true in many other locations, so I have learned to research the taxi systems in every city I visit. That’s cognitive load. I also bring enough local currency to pay for a taxi ride to my hotel and check on whether a tip is expected. More cognitive load.

Uber set out to remove the friction from taking a taxi by reimagining the entire experience, from hailing to routing to paying; from drivers and cars to insurance and regulations. By removing as many points of friction as possible for riders, Uber has become wildly popular in a very short time. In January 2015, four years after launch, Uber reported that its revenue its home city of San Francisco had grown to three times the size of the entire taxi market in that city. Uber has recently opened a robotics center in Pittsburgh and joins Google in working to create a practical driverless car. Its intent is to bring the cost and convenience of ride services to a point where owning a car becomes the expensive option.

Full Stack Startups

Uber is among the largest of a new crop of startups – investor Chris Dixon calls them full stack startups – that bypass incumbents and reinvent the entire customer experience from start to finish with the aim of making it as frictionless as possible. Full stack startups focus on creating a world that works the way it should work, given today’s technology, rather than optimizing the way it does work, given yesterday’s mental models. Because these companies are creating a new end-to-end experience, they rarely leverage existing capabilities aimed at serving their market; they develop a “full stack” of new capabilities.

"The challenge with the full stack approach is you need to get good at many different things: software, hardware, design, consumer marketing, supply chain management, sales, partnerships, regulation, etc. The good news is that if you can pull this off, it is very hard for competitors to replicate so many interlocking pieces." Chris Dixon

Large companies have the same full stack of capabilities as startups, but these capabilities lie in different departments and there is friction at every department boundary. Moreover, incumbents are deeply invested in the way things work today, so large incumbent companies are usually incapable of truly reinventing a customer journey. As hard as they try to be innovative, incumbents tend to be blind to the friction embedded in the customer journey that they provide today.

Consider banks. They have huge, complex back end systems-of-record that are expensive to maintain and keep secure.  But customers expect mobile access to their bank accounts, so banks have added “front end teams” to build portals (mobile apps) to access the back end systems. Typically banks end up with what Gartner calls “Bimodal IT.” One part of IT handles the backend systems using traditional processes, while a different group uses different processes to deliver web and mobile apps. As a result, the front end teams are not able to reimagine the customer journey; they are locked into the practices and revenue models embedded in the back end systems. So in the end, banks have done little to change the customer journey, the fee structure, or anything else fundamental to banking.

For example, in the US it is nearly impossible for me to transfer money to my granddaughter’s bank account without physically mailing a check or paying an exorbitant wire transfer fee. Not only that, but I cannot use my chip-and-pin card at many places in Europe because US banks don’t let me enter a pin with my card (they still depend on signatures!), while unstaffed European kiosks always require a pin. Anyone who banks in Europe would find US banking practices archaic. I find that they generate a lot of friction and I expect that they cost a lot of money.

Creative Friction

Why do banks adopt Bimodal IT? According to Gartner, “There is an inherent tension between doing IT right and doing IT fast.” I respectfully disagree; there is nothing inherently wrong about being fast. In fact, when software development is done right, speed, quality and low cost are fully compatible. Hundreds of enterprises, including Amazon and Google (whose systems manage billions of dollars of revenue every month), have demonstrated that the safest approach to software development is automated, it is adaptive, and it is fast.

It is true that there is tension between different disciplines: front end and back end; dev and ops; product and technology. But the best way to leverage these tensions is not to separate the parties, but to put them together on the same team with a common goal. You will never have a great product, or a great process, without making tradeoffs – that is the nature of difficult engineering problems. If your teams lack multiple perspectives on a problem, they will be unable to make consistently good tradeoff decisions, and their results will be mediocre.

Friction in the Code

The Prussian general and military theorist Carl von Clausewitz (1780-1831) thought of friction as the thing which tempers the good intentions of generals with the reality of the battlefield. He was thinking of the friction caused by boggy terrain that horses cannot cross, soldiers exhausted by heat and heavy burdens, fog that obscures enemy positions, supplies that don’t keep pace with military movements. He noted that battalions are made up of many individuals moving at a different rates with different amounts of confusion and fear, each one affecting the others around him in unpredictable ways. It is impossible for the thousands of individual agents on the battlefield to behave exactly according to a theoretical plan, Clausewitz wrote. Unless generals have actually experienced war, he said, they will not be able to account for the accumulated friction created by all of these agents interacting with each other and their environment.

Anyone who has ever looked closely at a large code base would be forgiven for thinking that Clausewitz was writing about software systems. Over time, any code base acquires lots of moving parts and increasing amounts of friction develops between these parts, until eventually the situation becomes hopeless and the system is either replaced or abandoned. Unless, of course, the messy parts are systematically cleaned up and friction is kept in check. But who is allowed to take time for this sort of refactoring if the decision-makers have never written any code, never been surprised by hidden dependencies, never been bitten by the unintended consequences of seemingly innocuous changes?

Failure

Not long ago the New York Stock Exchange was shut down for half a day due to “computer problems.” It’s not uncommon for an airline reservation systems suffer from “computer problems” so severe that planes are grounded. But we don’t expect to hear about “computer problems” at Twitter or Dropbox or Netflix or similar systems – maybe they had problems a few years ago, but they seem to be reasonably reliable these days. The truth is, cloud-based systems fail all the time, because they are built on unreliable hardware running over unreliable communication links. So they are designed to fail, to detect failure, and to recover quickly, without interrupting or corrupting the services they provide. They appear to be reliable because their robust failure detection and recovery mechanisms isolate users from the unreliable infrastructure.

The first hint of this approach was Google’s early strategy for building a server farm. They used cheap off-the-shelf components that would fail at a known rate, and then they automated failure detection and recovery. They replicated server contents so nothing was lost during a failure, and they automated the monitoring, detection, and recovery process. Amazon built its cloud with the same philosophy – they knew that at the scale they intended to pursue, everything would fail sooner rather than later, so automated failure detection and recovery had to be designed into the system.

Designing failure recovery into a system requires a special kind of software architecture and approach to development. To compensate for unreliable communication channels, messaging is usually asynchronous and on a best-efforts basis. Because servers are expected to fail, interfaces are idempotent so you get the same results on a retry as you get the first time. Since distributed data may not always match, software is written to deal with the ambiguities and produce eventual consistency.

Fault tolerance is not a new concept. Back in the days before solid state components, computer hardware was expected to fail, so vast amounts of time and energy were dedicated to failure detection and recovery. My first job was programming the Number 2 ESS (Electronic Switching System) being built in Naperville, IL by Bell Labs about the time I got out of college. This system, built out of discrete components prior to the days of integrated circuits, had a design goal of a maximum downtime of two hours in forty years. The hardware was completely duplicated and easily half of the software was dedicated to detecting faults, switching out the bad hardware, and identifying the defective discrete component so it could be replaced. This allowed a system built on unreliable electronic components to match the reliability of the electro-mechanical switching systems that were commonly in use at the time.

Situational Awareness

Successful cloud-based systems have a LOT of moving parts – that pretty much comes as a byproduct of success. With all of these parts moving around, designing for failure hardly seems like an adequate explanation for the robustness of these systems. And it isn’t. At the heart of a reliable cloud-based system are small teams (you might call them “full stack” teams) of people who are fully responsible for their piece of the system. They pay attention to how their service is performing, they fix it when it fails, and they continuously improve it to better serve its consumers.

Full stack teams that maintain end-to-end responsibility for a software service do not fit the model we used to have of the “right” way to develop software. These are not project teams that write code according to spec, turn it over to testing, and disband once it’s tossed over the wall to operations. They are engineering teams that solve problems and make frequent changes to the code they are responsible for. Code bases created and maintained by full stack teams are much more resilient than the large and calcified code bases created by the project model precisely because people pay attention to (and change!) the internal workings of “their” code on an on-going basis.

Limited Surface Area

Clearly, many small teams making independent changes to a large code base can generate a lot of Clausewitzian friction. But since friction occurs when the surfaces of two objects come in contact with each other, strictly limiting the surface area of the code exposed by each team can dramatically reduce friction. In cloud-based systems, services are designed to be as self-contained as possible and interactions with other services are strictly limited to hardened interfaces. Teams are expected to limit changes to the surface area (interfaces) of their code and proactively test any changes that might make it through that surface to other services.

Modern software development includes automated testing strategies and automated deployment pipelines that take the friction out of the deployment process, making it practical and safe to independently deploy small services. Containers are used to standardize the surface area that services expose to their environment, reducing the friction that comes from unpredictable surroundings. Finally, when small changes are made to a live system, the impact of each change is monitored and measured. Changes are typically deployed to a small percentage of users (limiting the deployment surface area), and if any problems are detected small changes can be rolled back quickly. We know that the best way to change a complex system is to probe and adapt, and we know that software systems are inherently complex. This explains why the small rapid deployments common in cloud-based systems turn out to be much safer and more robust than the large releases that we used to think were the “right” way to deliver software.

Shared Learning

Do you ever wonder how the sophisticated testing and deployment tools used at companies like Netflix actually work? Would you like to know how Netflix stores and analyzes data or how it monitors the performance of its platform? Just head over to the Netflix Open Source Center on GitHub; it’s all there for you to see – and use if you’d like. Want to analyze a lot of data? You will undoubtedly consider Hadoop, originally developed at Yahoo! based on Google research papers, open sourced through Apache, and now at the core of many open source tools that abstract its interface and extend its capability.

The world-wide software engineering community has developed a culture of sharing intellectual property, in stark contrast to the more common practice of keeping innovative ideas and novel tools proprietary. The rapid growth of large, reliable, secure software systems can be directly linked to the fact that software engineers routinely contribute to and build upon the work of their world-wide colleagues. Because of this, methods and tools for building highly reliable complex software systems have advanced extraordinarily quickly and are widely available.

Friction in the Process

Between 2004 and 2010, the FBI tried twice to develop an electronic case management system, and it failed both times, squandering hundreds of millions of dollars. UK’s National Health system lost similar amounts of money on a patient booking system that was eventually abandoned, and multiple billions of pounds on a patient record system that never worked. In 2012 Sweden decided to scrap and rewrite PUST, a police automation system that actually worked quite well, but not well enough for those who chose to have it rewritten the “right” way. The rewrite never worked and was eventually abandoned, an expensive fiasco that left the police without any system at all.

I could go on and on – just about every country has its story about an expensive government-funded computer system that cost extraordinary amounts of money and never actually worked. The reason? Broadly speaking, these fiascoes are caused by the process most governments use to procure software systems – a high friction process with a very high rate of failure.

One country that does not have an IT fiasco story is Estonia, probably the most automated country in the world. A few years ago British MP Francis Maude visited Estonia to find out how they managed to implement such sophisticated automation on a small budget. He discovered that Estonia automated its government because it had such a small budget, and properly automated government services are much less expensive than their manual counterparts.

Estonia’s process is simple: small internal teams work directly with consumers, understand their journey, and remove the friction. Working software is delivered in small increments to a small number of consumers, adjustments are made to make it work better, and once things work well the new capability is rolled out more broadly. Then another capability is added in the same manner, and thus the system grows steadily in small steps over time. (Incidentally, when this process is used, it is almost impossible to spend a lot of money only to find out the system doesn’t work.)

The UK government formed a consortium with Estonia and three other countries (called the Digital 5) to “provide a focused forum to share best practice [and] identify how to improve the participants’ digital services.” Maude started up the UK’s Government Digital Services, where small internal teams focus on making the process of obtaining government information and services as frictionless as possible. If you want to see how the UK Government Digital Services actually works, check out its Design Principles which summarize a new mental model for creating digital services, and the Governance approach, which outlines an effective, low friction software development process.

The HealthCare.gov fiasco in the US in 2013 led to the creation of US Digital Services, which is working in partnership with UK Digital Services to rethink government software development and delivery strategies. The US Digital Services Playbook is a great place for any organization to find advice on implementing a low friction development process.

DIGITAL SERVICE PLAYS:

1. Understand what people need
2. Address the whole experience, from start to finish
3. Make it simple and intuitive
4. Build the service using agile and iterative practices
5. Structure budgets and contracts to support delivery
6. Assign one leader and hold that person accountable
7. Bring in experienced teams
8. Choose a modern technology stack
9. Deploy in a flexible hosting environment
10. Automate testing and deployments
11. Manage security and privacy through reusable processes
12. Use data to drive decisions
13. Default to open

US Digital Services Playbook

The New Mental Model

The UK government changed – seemingly overnight – from high friction processes orchestrated by procurement departments to small internal teams governed by simple metrics. Instead of delivering “requirements” that someone else thinks up, teams are required to track four key performance indicators and figure out how to move these metrics in the right direction over time.

UK Digital Service’s four core KPIs:

1. Cost per transaction
2. User satisfaction
3. Completion rate
4. Digital take-up

See Gov.UK’s Performance Dashboard.

This is an entirely new mental model about how to develop effective software – one that removes all of the intermediaries between an engineering team and its consumers. It is a model that makes no attempt to define requirements, make estimates, or limit changes; instead it assumes that digital services are best developed through experimentation and require on-going improvement.

This is the mental model used by those who developed the first PUST system in Sweden, the one that was successful and appreciated by the police officers who used it. But unfortunately, conventional wisdom said it was not developed the “right” way, so the working system was shut down and rebuilt using the old mental model. And thus Sweden snatched failure from the jaws of success, proving once again that when it comes to developing interactive services, the old mental model simply Does. Not. Work.

Unexpected Points of Friction

It turns out that when governments move from the old mental model to the new mental model, many of the things that were considered “good” or “essential” in the past turn out to be “questionable” or “to be avoided” going forward. It’s a bit jarring to look at the list of good ideas that should be abandoned, but when you consider the friction that these ideas generate, it’s easier to see why forward-looking governments have eliminated them.

1. Requirements generate friction. The concept that requirements are specified by [someone] and implemented by “the team” has to be abandoned. Rather a team of engineers should explore hypotheses, testing and modifying ideas until they are proven or abandoned. Engineering teams should be expected to figure out how to make a positive impact on business metrics within valid constraints.

2. Handovers generate friction. The engineering team should have direct contact with at least a representative sample of the people whose journey they are automating. Just about any intermediary is problematic, whether the go-between is a procurement officer, business analyst, or product owner.

3.  Organizational boundaries generate friction. There is a reason why the UK and US use internal teams to develop Digital Services. Going through a procurement office creates insurmountable friction – especially when procurement is governed by laws passed in the days of the old mental model. The IT departments of enterprises often generate similar friction, especially when they are cost centers.

4. Estimates generate friction. Very little useful purpose is served by estimates at the task level. Teams should have a good idea of their capacity by measuring the rate at which they complete their current work or the time it takes work to move through their workflow. Teams should be asked "What can be completed in this time-frame?" rather than "How long will this take?" The UK Digital Services funds service development incrementally, with a general time limit for each phase. If a service does not fall within the general time boundaries, it is usually broken down into smaller services. 

5.  Multitasking generates friction. Teams should do one thing at a time and get it done, because task switching burns up a lot of cognitive overhead. Moreover, partially done work that has been put aside gums up the workflow and slows things down. 

6. Backlogs generate friction. A long "to do" list takes time to compile and time to prioritize, while everything on the list grows old and whoever put it there grows impatient. Don't prioritize - decide! Either the capacity exists to do the work, or it doesn't. Teams need only three lists: Now, Next, and Never. There is no try.

If Governments can do it, so can Enterprises

If governments can figure out how to design award-winning services [GOV.UK won the Design Museum Design of the Year Award in 2013] while moving quickly and saving money, surely enterprises can do the same. But first there is a lot of inertia to overcome. Once upon a time, governments assumed that obtaining software systems through a procurement process was essential, because it would be impossible to hire the people needed to design and develop these systems internally.  They were wrong. They assumed that having teams scattered about in various government agencies would lead to a bunch of unconnected one-of systems. They were wrong. They were afraid that without detailed requirements and people accountable for estimates, there would be no governance. They were wrong. Once they abandoned these assumptions and switched to the low friction approach pioneered by Estonia, governments got better designs, more satisfied consumers, lower cost, and far more predictable results.

Your organization can reap the same benefits, but first you will have to check your assumptions at the door and question some comforting things like requirements, estimates, IT departments, contracts, backlogs – you get the idea. Read the US Digital Services Playbook.  Could you run those 13 plays in your organization? If not, you need to uncover the assumptions that are keeping you in the grasp of the old mental model.

The Scaling Dilemma

“One of the most scalable organizations in human history was the Roman army. Its defining unit: The squad – eight guys. The number of guys that could fit in a tent,” says Chris Fry, who knows a bit about scaling. He led software development at Salesforce.com during its years of hyper growth, and is now SVP of Engineering at Twitter. Fry found that the way to build a scalable organization is to focus on the basic building blocks – small, stable, multidisciplinary teams that are expected to independently tackle problems, make decisions, and get things done. Fry advises, “When it comes to building a deeply efficient engineering organization, there are several things you can do to move the needle:

• Build strong teams first. Assign them problems later.
• Keep teams together.
• Go modular. Remove dependencies.
• Establish a short, regular ship cycle.”[1]

Amazon.com works on the same principle. Its basic unit is the two-pizza team – a team small enough to be fed with two pizzas. When Amazon needed to scale, two-pizza teams were chartered to build Amazon Web Services, one service per team. A team includes everyone needed to design, deliver, and support the service – from specifications to operations. If a service is too large for a two-pizza team, Amazon prefers split the service into smaller pieces rather than to combine teams to deal with the larger service, because this preserves the dynamic interactions of small teams.

What Could Go Wrong?

If it works for Salesforce, Amazon and Twitter, surely it will work for you… So you form a lot of small, independent, multidisciplinary teams and you are careful to give them all clear goals. What could go wrong with that?

August 27, Scene 1

“Hi Owen, how’s it going?”

“Just great! We’re going to make our target. We got the last piece done overnight. We’re working on integration testing right now. We’ll have it ready to release tomorrow. Lucky for me, because this month I really need that bonus.”

August 27, Scene 2

“Hey George, I hear you had a shutdown last night.”

“Yeah, we got things up fast, but it still counts against our shutdown limit. It’s the last one we can afford this month, if we want our bonuses. So there won’t be any more.”

“Do you know what caused it?”

“The usual. A new release from development. A naive piece of code, the kind of thing you can’t test for. Anyway, it’s fixed.”

“And you’ve made sure they won’t make that mistake again?”

“Nah, they don’t want to listen to us. We’re just not going to put up any more releases until September. We’re not going to miss our target.”

“But I hear they have another release almost ready to go.”

“Over my dead body.”

The Dilemma

It is a beautiful thing when the building block squads of an organization gel into high performance teams and can be counted on to meet challenging goals. But tricky part is, once you create these strong independent teams, how do you get them to work together? How do teams maintain their autonomous character while working in concert with an increasingly large network of other teams? How do you make sure that each team has a clear goal, but none of the goals are in conflict?

In a lean environment, the leader’s role is to set up strong teams, to be sure, but it is also to devise a system – let’s call it a goal system – which assigns goals to teams. This is no easy task. Teams need clear, meaningful goals; they have to be the right goals; teams must have the capacity, capability and autonomy to achieve their goals; and most important, the goals of various teams cannot conflict with each other.

One thing we know for certain is that local goals create local optimization. So it’s clear that the start of a goal system is a system-level, unifying goal. Something that conveys the purpose of the work, the why. Some way to confirm that progress is being made – at the team level – toward achieving the overall purpose. Of course, this is a lot easier said than done.

The Goal System

Many companies use projects to set up a goal system. A project manager lets the team know what the project goal is and what everyone needs to do to reach the goal. If there are several competing projects, a Project Management Office (PMO) is added to manage the project portfolio and distribute corporate goals among projects. However, there are problems with the project approach. You don’t often see stable teams in a project company, because people are usually assigned at project start and reassigned at the end. Worse, people are often assigned to multiple projects with competing demands on team members’ time. Finally, since most projects are conceived of as a relatively large batch of work, project teams tend to be quite a bit larger than a squad of eight to fourteen people. So the basic building blocks of scale – small, intact, multidisciplinary teams – are rarely found in a project environment.

One of the things Scrum has contributed to the practice of software development is the idea that small autonomous teams perform much better than large project teams or single-discipline teams that work in sequence. So Scrum provides the building blocks of scale, but unfortunately, it does not contain a scalable system for choosing team goals, making sure they contribute to organizational goals and are in sync with the goals of other teams. So we need to look elsewhere for ways to set up a goal system.

The Theory of Constraints

People with a lean mindset might look to the Theory of Constraints (TOC) for guidance on choosing and communicating team goals because it has a good track record for directing the efforts of multiple teams toward a single goal, at least in manufacturing.[2] TOC starts with the assumption that in any system there will always be a constraint that gets in the way of achieving the system goal, and the way to keep teams working toward the overall system goal is to be sure that everyone is focused on getting more work through the system constraint. Let’s see how TOC might be applied to developing a software system.

When the Constraint is Technical

The first step – after clarifying the overall system purpose and goal – is to find the biggest constraint to achieving that goal. For purpose of discussion, let’s choose one of the most typical technical constraints encountered in delivering a software system: the integration of various components of the system without the introduction of defects or unintended consequences. In fact, project organizations typically allocate a third or more of the project time to release overhead – including integration, testing, fixing defects, and deployment – with the largest portion going to finding and fixing problems discovered during integration. When integration is the system constraint, TOC tells us that the most important focus for development teams should be removing this constraint.

Agile approaches to software development recommend the frequent delivery of working software to customers. When this recommendation is followed literally (software is released to end customers frequently), the integration constraint is regularly exposed and has to be confronted. One of the earliest agile approaches, Extreme Programming (XP), includes technical practices such as Test Driven Development and Continuous Integration that help make frequent releases practical. Continuous Delivery[3], which expands on these practices, has gained widespread favor as the agile approach which explicitly focuses on the integration constraint. In Continuous Delivery we find actionable advice on how to tackle the integration problem with techniques that scale across large networks of teams.

The objective of Continuous Delivery is to dramatically increase the number of times integration occurs while decreasing the amount of time it takes to negligible levels. This has the same effect that just-in-time flow does in manufacturing: the impact of defects is reduced to near zero because they are discovered immediately before they can propagate or hide. The problem is, a much wider swath of an organization needs to get involved in Continuous Delivery than is typically found on a development team. The system architecture has to be devisable, the marketing department has to figure out how to deal with frequent deliveries, the development and operations departments have to work closely together.

The Theory of Constraints can help here. If the constraint is integration, TOC recommends that we measure the rate at which work moves through the constraint – in this case the rate at which completely integrated and tested software is released to production – and make improvement of this rate the goal of every team involved in the system. It turns out that a throughput measurement on the system constraint is a great metric for team goals because it is easy to measure, provides immediate feedback, and is structured to result in improved system-wide performance. If everyone working on a system is trying to stabilize and improve the rate at which tested, integrated software is successfully released to end customers, then teams across the system will naturally have to work together and will find that their goals are compatible.

When the Constraint is Knowledge

More often than not, however, technical issues are not the biggest constraint in system development and the fundamental problem is not an integration problem, it is a design problem or a fitness-for-use problem. Far too often we end up with a system that doesn’t work well or is difficult to use. In this case, the biggest constraint in developing software systems is the way in which we decide what to build and parse that decision amongst the teams doing the work.

Project organizations spend a good deal of time deciding what to do and turning these decisions into goals or requirements. However, this activity is front-loaded into the beginning of the project, while verification that the project requirements are correct waits until the project is complete – too late to make changes. Most project organizations consider deciding what to build to be an execution problem rather than a constraint. They would say: It’s unfortunate that project goals and requirements are sometimes wrong, but the way to fix this is to work harder on getting them right before the next project.

Organizations with a lean mindset would frame the problem differently. They are likely to identify the biggest barrier to making good decisions as incomplete knowledge, and thus the biggest constraint of the system would be the rate at which knowledge is generated. So a lean organization would focus on the feedback loop between customers and development teams. They would decrease the length of the feedback loop, increase the speed of the feedback loop, and remove barriers to the free flow of information inside the feedback loop. These organizations would say: If we can test our assumptions and designs more quickly we will learn faster and make better decisions.[4]

No More Politics

One of the signs that teams have conflicting goals is negative politics. A good way to eliminate political wrangling is to get teams to work together toward a unifying goal and to show team members the impact of their actions on that goal – in real time.

August 27, Alternate Scene

“Hi George, I noticed a little glitch in the last night’s graph so I thought I’d check to see if anything went wrong.”

“Oh hi, Owen. Glad you stopped by. As a matter of fact, we did have a shutdown last night, but we were lucky and caught it right away and got things back up fast enough we didn’t lose any customers.”

“So what caused it, do you think?”

“It was a database lockout. It seems to happen a lot about six to eight hours after a new release. When we find it we do a workaround that lasts until the next release.”

“Whoa! That means it’s something in the code that isn’t getting caught in testing.”

“Well, yeah! There’re a lot of problems in code that only come out during production.”

“Um, well, we worked late last night to get that feature ready that Chris asked for. Do you think we can do another release tomorrow?”

“Well Owen, let’s take a look at the graph. You see last night there was a downward spike here, where we closed down the site and no one could use the app. But we got it up so fast that almost everyone was still around and we could just reconnect them. It was the middle of the night here, so we mostly had browsers in Europe and Asia. We don’t have many customers there – yet. But what if we hadn’t caught it so fast? In five minutes we would have lost maybe half of the people using the app. What if it had happened during the daytime here? That spike would have been an order of magnitude bigger, and we would have lost a lot more people. It’s one of our busiest seasons, right before school starts. Do you really think we should take such a risk?”

“But George, we have to do it sometime. Sooner is better than later.”

“Not the way I see it, Owen. The fewer releases we put out, the fewer customers we’re likely to disrupt.”

“Okay, I see your point, but we’ve got to fix that problem so we can release frequently, because new features are what drives that graph up higher.”

“Not if the system crashes, they aren’t.”

“Whatever. We still have to fix the problem.”

“Yeah, so how do you propose we do that?”

“How about a side-by-side release with a trigger that knocks out the new system if it gets flaky?”

“Easy for you to say, Owen. You don’t have to make it happen.”

“But if I could make it happen, would you let me?”

“Sure, why not? But it won’t be easy.”

“How about I get with the team and tell them that in order to have a release, we have to write some failure detection and recovery code, and babysit the next release around the clock to be sure it works.”

“Can’t hurt to give it a try.”

The Unifying Goal

A network of strong teams is the first step to scale. The second step is to set up a system that distributes goals to teams in a way that avoids goal conflict. One good way to do this is to find a goal that is the final arbitrator of ‘good,’ make it visible to all teams in real time, and hold teams responsible for it.

The purpose of throughput accounting in the Theory of Constraints is to create just such a unifying goal. Simply stated, throughput accounting provides a measurement of the rate at which an organization achieves its purpose. This rate is effectively the same as the rate of throughput at the system constraint, so teams working to improve throughput at the constraint or to improve overall system impact are working toward the same goal. Either way, a single unifying goal allows individual teams to act autonomously, confident that they are not working at cross purposes with other teams.

What if you cannot find a unifying goal that represents the system constraint, or if a team’s work has no apparent impact on that goal? Over time it would be better to move to a decoupled architecture so that individual teams can have an impact on the overall system goal. But in the meantime, each team should monitor the impact of its work on its immediate customers. The question is not: Did the team complete its work? The proper question is: Did the team give its immediate customers what was needed, when it was needed, in a state that allowed the downstream teams to perform their work well?

Finally, remember that scaling is a two way street. If you think that scaling gives you problems as a leader, imagine the problems it brings to other people in the organization. It’s fun to work at a small company where everyone knows what’s important and works together to make the company successful. But as the company grows, people at all levels are in danger of losing their line of sight from what they are doing to the company’s success. This limits their ability to act with initiative to bring about that success, and thus undermines their engagement. You can’t scale unless you keep all of the bright minds in the company engaged and working together to help the company grow.
______________________________

Footnotes

[1] From First Round Review: Unlocking the Power of Stable Teams with Twitter's SVP of Engineering.

[2] I was reminded of this as I read Tame the Flow by Steve Tendon and Wolfram Müller, which provides a nice summary of how the Theory of Constraints in general, and throughput accounting in particular, can generate much better team goals than either the work-bin WIP limits of Kanban or the cost/profit focus of traditional cost accounting.

[3] See Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble and Dave Farley (2010), and Lean Enterprise: Adopting Continuous Delivery, DevOps, and Lean Startup at Scale by Jez Humble , Barry O’Reilly, and Joanne Molesky (2014).

[4] It’s interesting to note that Continuous Delivery is an excellent way to address the knowledge constraint as well as the technical constraint of software development.

Team Compensation

The New Venture team had done an incredible job, and they knew it. Increment by increment they had built a new software product, and when the deadline came, everything that had to be operational was working flawlessly. The division vice president thanked everyone who had contributed to the effort at an afternoon celebration, and the team members congratulated each other as they relived some of the more harrowing moments of the last six months.

The next day the team’s Scrum Master was catching up on long-ignored e-mail when Dave, the development manager, called. “Say Sue,” he said, “Great job your team did! I’ve been waiting for the product launch before I bothered you with this, but the appraisal deadline is next week. I need your evaluation of each team member. And if you could, I’d like you to rank the team, you know, who contributed the most down to who contributed the least.”

Sue almost heard the air escaping as her world deflated. “I can’t do that,” she said. “Everyone pitched in 100%. We could not have done it otherwise.” “But Sue,” Dave said. “certainly there must have been a MVP. And a runner-up. And so on.” “No, not really,” Sue replied. “But what I can do is evaluate everyone’s contribution to the effort.”

Sue filled out an appraisal input form for each team member. She rated everyone’s performance, but found that she had to check the ‘far exceeded expectations’ box for each team member. After all, getting out the product on time was a spectacular feat, one that far exceeded everyone’s expectations.

Two days later Sue got a call from Janice in human resources. “Sue,” she said, “Great job your team did! And thanks for filling out all of those appraisal input forms. But really, you can’t give everyone a top rating. Your average rating should be ‘meets expectations’. You can only have one or two people who ‘far exceeded expectations’. Oh and by the way, since you didn’t rank the team members, would you please plan on coming to our ranking meeting next week? We are going to need your input on that. After all, at this company we pay for performance, and we need to evaluate everyone carefully so that our fairness cannot be questioned.”

Sue felt like a flat tire. In the past, when she had a particularly difficult problem she always consulted the team, and she decided to consult them once again. At 10:00 the next morning, the entire team listened as Sue explained her problem. The had always come up with a creative solutions to problems she presented before, and she could only hope they would be able to do it again. She thought she might convince them to elect an MVP or two, to help her put some variation into the evaluations.

Sue suspected that Dave and Janice might not approve of her approach, but she didn’t realize that when the team members heard her dilemma, they would deflate just as quickly as she had. The best they could do was insist that everyone had given 200% effort, they had all helped each other, and they had thought that every single person had done a truly outstanding job. They were not interested in electing a MVP, but they were willing to choose a LVP. It would the unnamed manager who was asking Sue to decide amongst them.

Now Sue really had a problem. She had no idea how to respond to Dave and Janice, and the New Venture team had turned angry and suspicious. Tomorrow they would have to start working together on the next release. How could something that was supposed to boost performance do such a through job of crushing the team’s spirit?

Deming’s View
Sue is not the only one who had trouble with merit pay evaluation and ranking systems. One of the greatest thought leaders of the 20th century, W Edwards Deming, wrote that un-measurable damage is created by ranking people, merit systems, and incentive pay. Deming believed that every business is a system and the performance of individuals is largely the result of the way the system operates. In his view, the system causes 80% of the problems in a business, and the system is management’s responsibility. He wrote that using exhortations and incentives to get individuals to solve management problems simply doesn’t work. Deming opposed ranking because it destroys pride in workmanship, and merit raises because they address the symptoms, rather than the causes, of problems.

It’s a bit difficult to take Deming at face value on this; after all, companies have been using merit pay systems for decades, and their use is increasing. Moreover Deming was mainly involved in manufacturing, so possibly his thinking does not apply directly to knowledge work like software development. Still, someone as wise as Deming is not to be ignored, so let’s take a deeper look into employee evaluation and reward systems, and explore what causes them to become dysfunctional.

Dysfunction #1: Competition
As the New Venture team instinctively realized, evaluation systems which rank people for purposes of merit raises pit individual employees against each other and strongly discourage collaboration. Even when the rankings are not made public, the fact that they happen does not remain a secret. Sometimes ranking systems are used as a basis for dismissing the lowest performers, making them even more threatening. When team members are in competition with each other for their livelihood, teamwork quickly evaporates.

Competition between teams, rather than individuals, may seem like a good idea, but it can be equally damaging. Once I worked in a division in which there were two separate teams developing software products that were targeting similar markets. The members of the team which attracted the largest market share were likely have more secure jobs and enhanced career opportunities. So each team expanded the capability of its product to attract a broader market. The teams ended up competing fiercely with each other for the same customer base as well as for division resources. In the end, both products failed. A single product would have had a much better chance at success.

Dysfunction #2: The Perception of Unfairness
There is no greater de-motivator than a reward system which is perceived to be unfair. It doesn’t matter if the system is fair or not, if there is a perception of unfairness, then those who think that they have been treated unfairly will rapidly loose their motivation.

People perceive unfairness when they miss out on rewards they think they should have shared. What if the vice president had given Sue a big reward? Even if Sue had had acknowledged the hard work of the team, they would probably have felt that she was profiting at their expense. You can be sure that Sue would have had a difficult time generating enthusiasm for work on the next release, even if the evaluation issues had not surfaced.

Here’s another scenario: What would have happened if the New Venture team was asked out to dinner with the VP and each member given a good sized bonus? The next day the operations people who worked late nights and weekends to help get the product out on time would have found out and felt cheated. The developers who took over maintenance tasks so their colleagues could work full time on the product would have also felt slighted. Other teams might have felt that they could have been equally successful, except that they got assigned to the wrong product.

Dysfunction #3: The Perception of Impossibility
The New Venture Team met their deadline by following the Scrum practice of releasing a high quality product containing only the highest priority functionality. But let’s try a different scenario: Let’s assume that the team was given a non-negotiable list of features that had to be done by a non-negotiable deadline, and let’s further speculate that the team was 100% positive that the deadline was impossible. (Remember this is hypothetical; surely this would never happen in real life.) Finally, let’s pretend that the team was promised a big bonus if they met the deadline.

There are two things that could happen in this scenario. Financial incentives are powerful motivators, so there is a chance that the team might have found a way to do the impossible. However, the more likely case is that the promise of a bonus that was impossible to achieve would make the team cynical, and the team would be even less motivated to meet the deadline than before the incentive was offered. When people find management exhorting them to do what is clearly impossible rather than helping to make the task possible, they are likely to be insulted by the offer of a reward and give up without half trying.

Dysfunction #4: Sub-Optimization
I recently heard of a business owner who offered testers $5 for every defect they could find in a product about to go into beta release. She thought this would encourage the testers to work harder, but the result was quite different. The good working relationship between developers and testers deteriorated as testers lost their incentive to help developers quickly find and fix defects before they propagated into multiple problems. After all, the more problems the testers found, the more money they made.

When we optimize a part of a chain, we invariably sub-optimize overall performance. One of the most obvious examples of sub-optimization is the separation of software development from support and maintenance. If developers are rewarded for meeting a schedule even if they deliver brittle code without automated test suites or an installation process, then support and maintenance of the system will cost far more than was saved during development.

Dysfunction #5: Destroying Intrinsic Motivation
There are two approaches to giving children allowances. Theory A says that children should earn their allowances, so money is exchanged for work. Theory B says that children should contribute to the household without being paid, so allowances are not considered exchange for work. I know one father who was raised with Theory B, but switched to Theory A for his children. He put a price on each job and paid the children weekly for the jobs they had done. This worked for a while, but then the kids discovered that they could choose amongst the jobs and avoid doing the ones they disliked. When the children were old enough to earn a paycheck, they stopped doing household chores altogether, and the father found himself mowing the lawn along side the teenage children of his neighbors. Were he to do it again, this father says he would not tie allowance to work.

In the same way, once employees get used to receiving financial rewards for meeting goals, they begin to work for the rewards, not the intrinsic motivation that comes from doing a good job and helping their company be successful. Many studies have shown that extrinsic rewards like grades and pay will, over time, destroy the intrinsic reward that comes from the work itself.

One Week Later
Sue was nervous as she entered the room for the ranking meeting. She had talked over her problem with Wayne, her boss, and although he didn’t have any easy solutions, he suggested that she present her problem to the management team. Shortly after the meeting started, Janice asked Sue how she would rank her team members. Sue took a deep breath, got a smile of encouragement from Wayne, and explained how the whole idea of ranking made no sense for a team effort. She explained how she had asked for advice from the team and ended up with an angry and suspicious team.

“You should never have talked to the team about this,” said Janice. “Hold on a minute,” Wayne jumped in. “I thought our goal in this company is to be fair. How can we keep our evaluation policies secret and expect people to consider them fair? It doesn’t matter if we think they are fair, it matters if employees think they are fair. If we think we can keep what we are doing a secret, we’re kidding ourselves. We need to be transparent about how we operate; we can’t make decisions behind closed doors and then try to tell people ‘don’t worry, we’re being fair’”

Sue was amazed at how fast the nature of the discussion changed after Wayne jumped to her defense. Apparently she wasn’t the only one who thought this ranking business was a bad idea. Everyone agreed that the New Venture team had done an excellent job, and the new product was key to their business. No one had thought that it could be done, and indeed the team as a whole had far exceeded everyone’s expectations. It became apparent that there wasn’t a person in the room who was willing to sort out who had contributed more or less to the effort, so Sue’s top evaluation for every team member was accepted. More importantly, the group was concerned that a de-motivated New Venture team was a serious problem. Eventually the vice president agreed to go to the next meeting of the New Venture team and discuss the company’s evaluation policies. Sue was sure that this would go a long way to revitalize the team spirit.

Now the management team had a problem of its own. The knew that they had to live within a merit pay system, but they suspected they needed to rethink the way it was implemented. Since changes like that don’t happen overnight, they formed a committee to look into various evaluation and pay systems.
The committee started by agreeing that evaluation systems should not be used to surprise employees with unexpected feedback about their performance. Performance feedback loops must be far shorter than an annual, or even a quarterly, evaluation cycles. Appraisals are good times to review and update development plans for an employee, but if this is the only time an employee finds out how they are doing, a lot more needs fixing than the appraisal system.

With this disclaimer in mind, the committee developed some guidelines for dealing with various forms of differential pay systems.

Guideline #1: Make Sure The Promotion System Is Unassailable
In most organizations, significant salary gains come from promotions which move people to a higher salary grade, not merit increases. Where promotions are not available, as is the case for many teachers, merit pay systems have a tendency to become contentious, because merit increases are the only way to make more money. When promotions are available, employees tend to ignore the merit pay system and focus on the promotion system. Of course this system of promotions tends to encourage people to move into management as they run out of promotional opportunities in technical areas. Companies address this problem with ‘dual ladders’ that offer management-level pay scales to technical gurus.

The foundation of any promotion system is a series of job grades, each with a salary range in line with industry standards and regional averages. People must be placed correctly in a grade so that their skills and responsibilities match the job requirements of their level. Initial placements and promotion decisions should be carefully made and reviewed by a management team.

Usually job grades are embedded in titles, and promotions make the new job grade public through a new title. Thus a person’s job grade is generally considered public information. If employees are fairly placed in their job grade, and promoted only when they are clearly performing at the new job grade, then salary differences based on job grade are generally perceived to be fair. Thus a team can have both senior and junior people, generalists and highly skilled specialists, all making different amounts of money. As long the system of determining job grades and promotions is transparent and perceived to be fair, this kind of differential pay is rarely a problem.

The management team at Sue’s company decided to focus on a promotion process that did not use either a ranking or a quota system. Instead, clear promotion criteria would be established for each level, and when someone had met the criteria, they would be eligible for promotion. A management committee would review each promotion proposal and gain a consensus that the promotion criteria were met. This would be similar to existing committees that reviewed promotions to fill open supervisor or management positions.

Guideline #2: De-emphasize The Merit Pay System
When the primary tool for significant salary increases is promotion, then it’s important to focus as much attention as possible on making sure the promotion system is fair. When it comes to the evaluation system that drives merit pay, it’s best not try too hard to sort people out. Studies show that when information sharing and coordination are necessary, organizations that reduce pay differences between the highest and the lowest paid employees tend to perform better over time.

Use evaluations mainly to keep everyone at an appropriate level in their salary grade. Evaluations might flag those who are ready for promotion and those who need attention, but that should trigger a separate promotion or corrective action process. About four evaluation grades are sufficient, and a competent supervisor with good evaluation criteria and input from appropriate sources can make fair evaluations that accomplish these purposes.

Even when annual raises are loosely coupled to merit, evaluations will always be a big deal for employees, so attention should be paid to making them fair and balanced. Over the last decade, balanced scorecards have become popular for management evaluations; at least in theory, balanced scorecards ensure that the multiple aspects of a manager’s job all receive attention. A simple version of a balanced scorecard might also be used for a merit pay evaluations, to emphasize the fact that people must perform well on many dimensions to be effective. A supervisor might develop a scorecard with each employee that takes into account team results, new competencies, leadership, and so on. It is important that employees perceive that the input to a scorecard is valid and fairly covers the multiple aspects of their job. It is important to keep things simple, because too much complexity will unduly inflate the attention paid to a pay system which works better when it is understated. Finally, scorecards should not be used to feed a ranking system.

Guideline #3: Tie Profit Sharing To Economic Drivers
Nucor Steel decided to get into the steel business in 1968, and thirty years later it was the biggest steel company in the US. When Nucor started up, Bethlehem Steel considered it a mere gnat, but 35 years later Bethlehem Steel was not only bankrupt, but sold off for assets. So Nucor Steel is one very successful company that has done a lot of things right in a tough industry. Quite surprisingly, Nucor has a decades-old tradition of paying for performance. How does the company avoid the dysfunctions of rewards?
Nucor Steel started with the realization that profit per ton of finished steel as its key economic driver, and based its profit sharing plan on the contribution a team makes to improving this number. So for example, a team that successfully develops a new steel-making process or starts up a new plant on schedule will not get see an increase in pay until the process or plant has improved the company’s profit per ton of steel. Thus Nucor avoids sub-optimization by tying its differential pay system as close to the economic driver of its business as possible.

Guideline # 4: Reward Based on Span of Influence, Not Span of Control
Conventional wisdom says that people should be evaluated based on results that are under their control. However, this kind of evaluation creates competition rather than collaboration. Nucor makes sure that its profit sharing formula rewards relatively large teams, not just the individuals or small groups who have direct responsibility for an area. Following this principle, if a software program creates a significant profit increase, everyone from those who brought the idea into the company to developers and testers to operations and support people to the end users should share in any reward.

Nucor Steel works hard to create a learning environment, where experts move from one plant to another, machine operators play a significant role in selecting and deploying new technology, and tacit knowledge spreads rapidly throughout the company. It’s reward system encourages knowledge sharing by rewarding people for influencing the success of areas they do not control.

How, exactly, can rewards be based on span of influence rather than span of control? I recommend a technique called ‘Measure UP’. No matter how hard you try to evaluate knowledge work or how good a scorecard you create, something will go unmeasured. Over time, the unmeasured area will be de-emphasized and problems will arise. We have a tendency to add more measurements to the scorecard to draw attention to the neglected areas.

However, it is a lot easier to catch everything that falls between the cracks by reducing the number of measurements and raising them to a higher level. For instance, instead of measuring software development with cost and schedule and earned value, try creating a P&L or ROI for the project, and help the team use these tools to drive tradeoff decisions.

Guideline #5: Find Better Motivators Than Money
While monetary rewards can be a powerful driver of behavior, the motivation they provide is not sustainable. Once people have an adequate income, motivation comes from things such as achievement, growth, control over one’s work, recognition, advancement, and a friendly working environment. No matter how good your evaluation and reward system may be, don’t expect it to do much to drive stellar performance over the long term.

In the book “Hidden Value” , Charles O’Reilly and Jeffrey Pfeffer present several case studies of companies that obtain superb performance from ordinary people. These companies have people-centered values which are aligned with actions at all levels. They invest in people, share information broadly, and rely on teams, and emphasize leadership rather than management. Finally, they do not use money as a primary motivator; they emphasize the intrinsic rewards of fun, growth, teamwork, challenge and accomplishment.

Treat monetary rewards like explosives because they have will a powerful impact whether you intend it or not. So use them lightly and with caution. They can get you in trouble much faster than they can solve your problems. Once you go down the path of monetary rewards, you may never be able to go back, even when they cease to be effective, as they inevitably will. Make sure that people are fairly and adequately compensated, and then move on to more effective ways to improve performance.

Six Months Later
The New Venture Team is having another celebration. They had all been surprised when the VP came to their team meeting six months earlier. But they quickly recovered and told her that they each wanted to be the best, they wanted to work with the best, and they did not appreciate the implication that some of them were better than others. When the VP left, the team cheered Sue for sticking up for them, and then got down to work with renewed enthusiasm. Now, two releases later, the customers were showing their appreciation with their pocketbooks.

There haven’t been any dramatic pay increases and only the occasional, well-deserved promotion. But the company has expanded its training budget and New Venture team members have found themselves mentoring other teams. Sue is rather proud of them all as she fills out the newly revised appraisal input forms that have more team-friendly evaluation criteria. This time Sue is confident that her judgment will not be questioned.

Published in Better Software Magazine July, 2004 

Screen Beans Art, © A Bit Better Corporation

Toward A New Definition of Maturity

In the mid 90's, a company named Zeos assembled PC's in my home state of Minnesota. I was impressed when Zeos was named as a finalist for the Malcolm Baldrige Award, because competing for this quality award is more or less the equivalent of a software company trying to reach CMM Level 5. At the same time that Zeos was focusing on the Malcolm Baldrige Award, a similar company in Austin, Texas, called Dell Computer, was focusing all of its energy on two rather different objectives: 1) keep inventory as low as possible, because it is the biggest risk in the PC assembly business, and 2) deliver computers just as fast as possible after customers decide what they want. Dell could not possibly meet these goals unless it had the key Malcolm Baldrige capabilities in place, but that was not its focus. On the other hand, in order to be a finalist, Zeos had to spend huge amounts of executive and management attention on Malcolm Baldrige activities. So which of these two companies would you call the most mature?

A company does not rise to the top of its industry by perfecting its normative procedures. While General Motors was busy refining it’s four phase process for product development, Toyota and Honda were developing cars in something approaching half the time, for half the cost. The resulting cars were less expensive to produce and captured a much larger market share. Yet when looked at through Detroit’s eyes, the development approach of the Japanese companies was decidedly immature – why, they even let die makers start cutting dies long before the final drawings were released!

The problem with maturity models is that they foster a mental model of ‘good practice’ that can block out paradigm shifts that are destined to reshape the industry. This has happened in manufacturing. It has happened in product development. Can it be happening in software development today?

Why Do We Do This To Ourselves?
We human beings a tendency to decompose complex problems into smaller pieces, and then focus on the individual pieces, often, unwittingly, at the expense of the whole. One reason for this is that people can only hold a few chunks of information in short term memory at once; Miller’s law claims that this number is seven plus or minus two. Thus we have a strong tendency to take concepts which we cannot get our minds around and decompose them into parts, so we can deal with one part at a time. There is nothing wrong with our very natural tendency to decompose problems, except that we often forget to reconfigure the parts into a whole and take their interactions into account.

The problem is, decomposition only works if the whole is indeed equal to the sum of its parts, and interactions between the parts can be ignored. In practice, this is rarely the case; in fact, optimization of the parts has a tendency to sub-optimize the whole. For example, a manager might think that the best way to run a testing department is to make sure that every single person is working all of the time. In order to guarantee maximum productivity of each individual, he makes sure there is always a pile of testing waiting to be done. Down the hall, the operations manager knows better. She knows that if she runs the servers at high utilization, the entire system bogs down, just like rush hour traffic. If only the testing manager would realize that his policy of full utilization of testing resources creates the same kind of traffic jam in software development!

Decomposition of a problem into pieces is a standard problem solving heuristic, but it needs to be accompanied by regular aggregation of the parts into a whole. It is here that iterative development shines, because it forces us to develop, test, integrate, and release complete threads of the system early and often. Thus we reset our view of the big picture every iteration. However, iterative development requires that we decompose the problem space along dimensions that are orthogonal to those traditionally recommended. Instead of decomposing the problem into requirements, analysis, design, programming, testing, integration, deployment, we decompose the problem into features along the natural fault lines of the domain.

Decomposition vs. Abstraction
There is an alternative to the decomposition of a problem into components, and that is to abstract the problem to a higher level, rather than decompose it to a lower level. The result is the same in both cases – you have reduced the number of things you need to think about to 7 +/- 2. But when you move the problem up to a higher level of abstraction, the interaction of the parts is maintained, so abstractions would seem to be the better approach to solving complex problems.

There is a catch: people can’t create abstractions without understanding the domain, because correct abstractions require that the abstractor knows what is important, what fits together naturally, and where to find the natural joints or fault lines of the domain. For this reason, experts in the domain have a greater tendency to abstract to a higher level, while those who are not familiar with the domain tend to decompose the problem, and that decomposition will probably not be along the natural fault lines of the domain.

It is when we decompose a problem in a manner which does not fit the domain, and then drill down to the details too fast, that important things get missed. We would like to think that a lot of early detail will help us find all the hidden gremlins that might bite us later. This is only true if those gremlins are actually inside the areas we investigate – but the tough problems usually lurk between the cracks in our thinking. Unfortunately we only discover those problems when we integrate the pieces together at the end, and by that time we have invested so much in the details that change is very difficult.

It is safer to work from a higher level or abstraction, but this means we don’t drill down to detail right away. We take a breadth-first approach, keep options open and gradually fill in the details. With this approach we are more likely to find those gremlins while we still can do something about them, but some people might think that we don’t have a plan, aren’t tracking the project, and can’t manage requirements.

Assessment Rather than Certification
Quite frequently CMM is implemented as a certification process, decomposing ‘mature’ into separately verifiable capabilities. The danger is that we may lose sight of forest for the trees. For example, focusing on requirements management can be detrimental to giving users what they really want. Focusing on quality assurance separate from development can destroy the integrating effect of developers and testers working side-by-side every day. Focusing on planning as a predictive process can keep us from using planning as an organizing process.

A better approach to discovering the underlying competence of an organization is to use an assessment process rather than a certification process. Assessments present challenging situations that cannot be dealt with unless a host of capabilities are in place; if the challenge is properly handled, the presence of these capabilities can be inferred. For example, a pilot’s ability to fly a plane can be assessed by observing how the pilot lands a plane in a stiff cross wind.

Consider your hiring process. When a candidate lists Microsoft or PMI certifications, you take that into account, but you are really looking for a track record of success which demonstrates that the certifications were put to good use. One software company I know administers a one hour logic tests to job applicants, a test without a line of code in it. Yet the test does an admirable job of assessing the capability of the candidate to think like a good developer.

Measure UP
If we accept that we get what we measure then, as Rob Austin points out in “Measuring and Managing Performance in Organizations,” we do not get what we don’t measure. If we are not measuring everything, if something important just didn’t make it into our measurement system, then we are not going to get it. To counter this problem, we have a tendency to pile one measurement on top of another, every time we discover that we forgot to measure something. In CMM, for example, each KPA addresses something that caused a failure of some software project somewhere. As people discovered new ways for software projects to fail, a KPA’s was added to address the new failure mode. Nevertheless, all of those KPA’s still don’t cover everything that could go wrong, although they certainly try.

Once we admit that we just can’t measure everything, then we are ready to move to assessment-style measurements rather than a certification-style measurements. For example, in project management, we decompose the measurement system into cost, schedule, scope and defects, and we try hard to make these measurements work because we think they are the measurements we should use. But no matter how hard we try, we are often unsuccessful in measuring true business value by totaling up the yardsticks of cost, schedule, scope and defects.

What if we just measured business value instead of cost, schedule, scope, and defects? When I was developing new products at 3M, we did not pay much attention to cost, schedule and scope. Instead we developed a P&L which we would use to check out the impact of a late introduction date or a lower unit cost. The development team tried to optimize the overall P&L, not any one dimension. It may seem strange that a project team would concern itself with the financial model of the business, but I assure you it is far better decision-support tool than cost, schedule and scope.

The Measure of Maturity
I believe that our industry could use a simpler measurement of software development maturity, one that is not subject to the dangers of decomposition and does not attempt to defy Miller’s Law with a long lists of things to think about. I propose that we use a measurement that has been successfully used in countless organizations, and has proven to be a true indicator of the presence of the kind of capabilities measured by CMM. To introduce this measurement, let’s go back to Dell and see what it measured:

1. The level of inventory throughout the entire system, and
2. The speed with which the organization can repeatedly and reliably respond to a customer request.

Okay, you say, fine for manufacturing, but software development is different.

True, I counter, but these measurements still work.

The level of inventory in your system is the amount of stuff you have under development. The more inventory of unfinished development work you have sitting around, the more you are at risk of it growing obsolete, getting lost, and hiding defects. If you capitalized it, you also bear the risk of having to write it off if it doesn’t work. The less of that kind of stuff you have on hand, the better off you will be.

The speed with which you can responds to a customer is directly proportional to the amount of unfinished development work you have clogging up your system. In truth, the two measurements above are one and the same – you can deliver faster if your system is not clogged with unfinished work. But more to the point, you cannot reliably and repeatedly deliver fast if you do not have a mature organization. You need version control, built-in quality, ways to gather requirements fast and routinely translate them correctly into code. You need everything you measure in CMM, and you need all of those capabilities working together as an integrated whole. In short, the speed with which you can repeatedly and reliably deliver on customers requests is a better measure of the maturity of your software development organization.

Published as “Measuring Maturity” in Software Development Magazine, February 2004.

Screen Beans Art, © A Bit Better Corporation

Measure Up

Getting measurements right can be devilishly difficult, but getting them wrong can be downright dangerous. If you look underneath most self-defeating behavior in organizations, you will often find a well-intentioned measurement which has gone wrong. Consider the rather innocent-sounding measurement of productivity, and it’s close cousin, utilization. One of the biggest impediments to adopting Just-in-Time manufacturing was the time-honored practice of trying to extract maximum productivity out of every machine. The inevitable result was that mounds of inventory collected to feed machines and additional piles of inventory stacked up at the output side of the machines. The long queues of material slowed everything down, as always queues do. Quality problems often took days to surface, and customer orders often took weeks to fill. Eventually manufacturing people learned that running machines for maximum productivity was a sub-optimizing practice, but it was a difficult lesson.

As software development organizations search for productivity on today’s tight economy, we see the same lesson being learned again. Consider the testing department which is expected to run at 100% utilization. Mounds of code tend to accumulate at the input side of the testing department, and piles completed tests stack up at the output side of the testing department. Many defects lurk in the mountain of code, and more are being created by developers who do not have immediate feedback on their work. When a testing department is expected to run at full utilization, the likely result will be an increased defect level, resulting in more work for the testing department.

Nucor Steel grew from a startup in 1968 into a $4 billion giant, attributing much of its success to an incentive pay system based on productivity. Productivity? How did Nucor keep their productivity measurement robust and honest throughout all of that growth? How did they avoid the sub-optimization so common most productivity measurements?

The secret is that Nucor measures productivity at a team level, not at an individual level. For example, a plant manager is not rewarded on the productivity of his or her plant, but on the productivity of all plants. The genius of Nucor’s productivity measurement is that it avoids sub-optimization by measuring results at one level higher than one would expect, thus encouraging knowledge sharing and system-wide optimization.

How can this be fair? How can plant managers be rewarded based on productivity of plants over which they have no control? The problem is, if we measure people solely on results over which they have full control, they have little incentive to collaborate beyond their own sphere of influence to optimize the overall business. While local measurements may seem fair to individuals, they are hardly fair to the organization as a whole.

Measure-UP, the practice of measuring results at the team rather than the individual level, keeps measurements honest and robust. The simple act of raising a measurement one level up from the level over which an individual has control changes its dynamic from a personal performance measurement to a system effectiveness indicator.

In the book “Measuring and Managing Performance in Organizations”, Dorset House 1996, Robert Austin discusses the dangers of performance measurements. The beauty of performance measurements is that “You get what you measure.” The problem with performance measurements is that “You get only what you measure, nothing else.” You tend to loose the things that you can’t measure: insight, collaboration, creativity, dedication to customer satisfaction.

Austin recommends aggregating individual performance measurements into higher level informational measures that hide individual results in favor of group results. As radical as this may sound, it is not unfamiliar. Edward Demming, the noted quality expert, insisted that most quality defects are not caused by individuals, but by management systems that make error-free performance all but impossible. Attributing defects to individuals does little to address the systemic causes of defects, and placing blame on individuals when the problem is systematic perpetuates the problem.

Software defect measurements are frequently attributed to individual developers, but the development environment often conspires against individual developers and makes it impossible to write defect-free code. Instead of charting errors by developer, a systematic effort to provide developers with immediate testing feedback, along with a root cause analysis of remaining defects, is much more effective at reducing the overall software defect rate.

By aggregating defect counts into an informational measurement, and hiding individual performance measurements, it becomes easier to address the root causes of defects. If an entire development team, testers and developers alike, feel responsible for the defect count, then testers will tend to become involved earlier and provide more timely and useful feedback to developers. Defects caused by code integration will become everyone’s problem, not just the unlucky person who wrote the last bit of code.

It flies in the face of conventional wisdom to suggest that the most effective way to avoid the pitfalls of measurements is to use measurements that are outside the personal control of the individual being measured. But conventional wisdom is misleading. Instead of making sure that people are measured within their span of control, it is more effective to measure people one level above their span of control. This is the best way to encourage teamwork, collaboration, and global, rather than local, optimization.

Screen Beans Art, © A Bit Better Corporation