The Cost Center Trap

In the 1960’s, IT was largely an in-house back-office function focused on process automation and cost reduction. Today, IT plays a significant strategic and revenue role in most companies, and is deeply integrated with business functions. By 2010, over 50% of firms’ capital spending was going to IT, up from 10-15% in the 1960’s.[1] But one thing hasn't changed since the 1960’s: IT has always been considered a cost center. You are probably thinking "Why does this matter?" Trust me, cost center accounting can be a big trap.

Back in the mid 1980’s Just-in-Time (JIT) was gaining traction in manufacturing companies. JIT always drove inventories down sharply, giving companies a much faster response time when demand changed. However, accounting systems count inventory as an asset, so and any significant reduction in inventory had a negative impact on the balance sheet. Balance sheet metrics made their way into senior management metrics, so successful JIT efforts tended to make senior managers look bad. Often senior management metrics made their way down into the metrics of manufacturing organizations, and when they did, efforts to reduce inventory were half-hearted at best. A generation of accountants had to retire before serious inventory reduction was widely accepted as a good thing.[2] 

Returning to the present, being a cost center means that IT performance is judged – from an accounting perspective – solely on cost management. Frequently these accounting metrics make their way into the performance metrics of senior managers, while contributions to business performance tend to be deemphasized or absent. As the metrics of senior managers make their way down through the organization, a culture of cost control develops, with scant attention paid to improving overall business performance. Help in delivering business results is appreciated, of course, but rarely is it rewarded, and rarer still is the cost center that voluntarily accepts responsibility for business results.

Now let’s add an Agile transformation to this cost center culture. Let’s assume that the transformation is supposed to bring benefits such as faster time to market, more relevant products, better customer experiences. And let’s assume that the cost center metrics do not change, or if they do change, process metrics such as number of agile teams and speed of deployment are added. I’ll wager that very few of those agile teams are likely to focus on improving overall business performance. The incentives send a clear message: business performance is not the responsibility of a cost center.

Being in a cost center can be demoralizing. You aren’t on the A team that brings in revenue, you’re on the B team that consumes resources. No matter how well the business performs, you’ll never get credit. Your budget is unlikely to increase when times are good, but when times are tight, it will be the first to be cut. Should you have a good idea, it had better not cost anything, because you can’t spend money to make money. If you think that a bigger monitor would make you more efficient, good luck making your case. Yet if your colleagues in trading suggest larger monitors will help them generate more revenue, the big screens will show up in a flash.[3]

Let’s face it, unless there are mitigating circumstances, IT departments that started out as cost centers are going to remain cost centers even when the company attempts a digital transformation. What kind of mitigating circumstances might help IT escape the cost center trap?

1. There is serious competition from startups.
   Startups develop their software in profit centers; they haven’t learned about cost centers yet. And in a competitive battle, a profit center will beat a cost center every time.
2. IT is recognized as a strategic business driver.
   You would think that a digital transformation would be undertaken only after a company has come to realize the strategic value of digital technology, but this is not the case. IT has been treated as if it were an outside contractor for so long that it is difficult for company leaders to think of IT as a strategic business driver, integral to the company's success going forward.
   
3. A serious IT failure has had a huge impact on business results.
   When it becomes clear exactly how dependent a profit center is on a so-called cost center, people in the profit center are often motivated to share their pain with IT. Smart IT departments will use this opportunity to share the gain also.

Many people in the Agile movement preach that teams should have responsibility for the outcomes they produce and the impact of those outcomes. But responsibility starts at the top and is passed down to teams. When IT is managed as a cost center with cost objectives passed down through the hierarchy, it is almost impossible for team members from IT to assume responsibility for the business outcomes of their work. When IT metrics focus on cost control, digital transformations tend to stall.

Every ‘full stack team’ working on a digital problem should have ‘full stack responsibility’ for results, and that responsibility should percolate up to the highest managers of every person on the team.  Business results, not cost, should receive the focused attention of every member of the team, and every incentive that matters should be aimed at reinforcing this focus.

The Capitalization Dilemma

Let’s return to the surprising assertion that in 2010, over 50% of firms’ capital spending was going to IT.[1] One has to wonder what was being capitalized. Yes, there were plenty of big data centers that were no doubt capitalized, since the movement to the cloud was just beginning. But in addition to that, a whole lot of spending on software development was also being capitalized. And herein lies the seeds of another undue influence of accounting policies over IT practices.

Software development projects are normally capitalized until they are “done” – that is they reach "final operating capability" and are turned over to production and maintenance.[1] But when an organization adopts continuous delivery practices, the concept of final operating capability – not to mention maintenance – disappears. This creates a big dilemma because it's no longer clear when, or even if, software development should be capitalized. Moving expenditures from capitalized to expensed not only changes whose budget the money comes from, it can have tax consequences as well. And what happens when all that capitalized software (which, by the way, is an asset) vanishes? Just as in the days when JIT was young, continuous delivery has introduced a paradigm shift that messes up the balance sheet.

But the balance sheet problem is not the only issue; depreciation of capitalized software can wreck havoc as well. In manufacturing, the depreciation of a piece of process equipment is charged against the unit cost of products made on that equipment. The more products that are made on the equipment, the less cost each product has to bear. So there is strong incentive to keep machines running, flooding the plant with inventory that is not currently needed. In a similar manner, the depreciation of software makes it almost impossible to ignore its sunk cost, which often drives sub-optimal usage, maintenance and replacement decisions.

Capitalization of development creates a hidden bias toward large projects over incremental delivery, making it difficult to look favorably upon agile practices. Hopefully we don't have to wait for another generation of accountants to retire before delivering software rapidly, in small increments, is considered a good thing.

To summarize, the cost center trap and the capitalization dilemma both create a chain reaction:

1. Accounting drives metrics.
   ⇩
2. Metrics drive culture.
   ⇩
3. Culture eats process for lunch.

The best way to avoid this is to break the chain at the top – in step 1. Stop letting accounting drive metrics. Alternatively, if accounting metrics persist at the senior management level, then break the chain at step 2 – do not pass accounting metrics down the reporting chain; do not let them drive culture. When teams focus on improving the performance of the overall business, accounting metrics should move in the right direction on their own; if they don't then clearly something is wrong with the accounting metrics.

Beware of Proxies

This year Jeff Bezos's annual letter to Amazon shareholders[4] listed four essentials that help big companies preserve the vitality of a startup: customer obsession, a skeptical view of proxies, the eager adoption of external trends, and high-velocity decision making. These seem pretty clear, except maybe the second one: a skeptical view of proxies. Just what are proxies? Bezos explains:

“A common example is process as proxy. Good process serves you so you can serve customers. But if you’re not watchful, the process can become the thing. This can happen very easily in large organizations. The process becomes the proxy for the result you want. You stop looking at outcomes and just make sure you’re doing the process right. Gulp.”

“Another example: market research and customer surveys can become proxies for customers – something that’s especially dangerous when you’re inventing and designing products.”

Here are some common proxies we find in software development:

Accounting metrics are proxies, and not very good ones at that, because they encourage local sub-optimization. 

Project metrics – cost, schedule, and scope – are proxies. Worse, these proxies are rarely validated against actual outcomes.  

“The Business” is a proxy for customers. Generally speaking, so is the product owner.

Proxies should be resisted, Bezos argues, if you want a vibrant startup culture in your company. But without proxies, how do you manage the dynamic and increasingly important IT organization? You make a habit of measuring what really matters - skip the proxies and focus on outcomes and impact.

In his excellent book, “a Seat at the Table,”[5]  Mark Schwartz proposes that IT governance and oversight should begin with strategic business objectives and produce investment themes that accomplish these objectives. IT leaders fund teams to produce desirable outcomes that will have impact on the strategic objectives. Note that these outcomes are not proxies, they are real, measurable progress toward the strategic objective. Regular reviews of teams’ progress -- quantified by these measurable outcomes -- provides leaders with insight, flexibility and an appropriate level of control. At the same time, detailed decisions are made by the people closest to customers after careful investigation, experimentation and learning.

Schwartz concludes: "this approach can focus IT planning, reduce risk, eliminate waste, and provide a supportive environment for teams engaged in creating value."[5] What's not to like?

______________________
Footnotes:

[1] From “What is Digital Intelligence” by Sunil Mithas and F. Warren McFarlan, IEEE Computing Edge, November 2017. Pg.9.

[2] The 1962 book “The Structure of Scientific Revolutions” by Thomas Kuhn discussed how significant paradigm shifts in science do not take hold until a generation of scientists brought up with the old paradigm finally retire.

[3] Thanks to Nick Larsen. Does Your Employer See Software Development as a Cost Center or a Profit Center?

[4] Jeff Bezos - Letter to Shareholders - April 12, 2017

[5] "A Seat at the Table" by Mark Schwartz

Lean Software Development: The Backstory

We were in a conference room near the Waterfront in Cape Town. “I just lost a crown from one of my teeth.” my husband Tom declared just before I was scheduled to open the conference. Someone at our table responded, “You’re lucky, Cape Town has some of the best dentists in the world.” It didn’t feel very lucky; Cape Town was the first stop on a ten week trip to Africa, Europe, and Australia.

The situation was eerily familiar. A year earlier a chip had cracked off of my tooth as I ate a pizza in Lima, the first stop of a ten week trip to South America. I ate gingerly during the rest of the trip, worried that the tooth would crack further. Luckily I made it back home with no pain and little additional damage. Once there, it took three days to get a dentist appointment. The dentist made an impression of the gap in my tooth and fashioned a temporary crown. “This will have to last for a week or two,” she said. “If it falls out, just stick it back in and be more careful what you eat.” Luckily the temporary crown held, and ten days later a permanent crown arrived from the lab. Two weeks after we arrived home, my tooth was fixed.

We were scheduled to be in Cape Town for only two days. How was Tom going to get a crown replaced in two days? A small committee formed. Someone did a phone search; apparently the Waterfront was a good place to find dentists. A call was made. “You can go right now – the dental office is nearby. Do you want someone to walk you over?” As Tom headed out the door with an escort, I got ready for my presentation. Half way through the talk, I saw Tom return and signal that all was well.

“I lost a part of my tooth, not just the crown,” Tom told me after the talk. “I’m supposed to return at 3:30 this afternoon; I should have a new crown by the end of the day.” The dentist had a mini-lab in his office. Instead of making a temporary crown, he used a camera to take images of the broken tooth and adjacent teeth. The results were combined into a 3D model of the crown to which the dentist made a few adjustments. Then he selected a ceramic blank that matched the color of Tom’s teeth and put it in a milling machine. With the push of a button, instructions to make the crown were loaded into the machine. Cutters whirled and water squirted to keep the ceramic cool. Ten minutes later the crown was ready to cement in place. Ninety minutes after he arrived that afternoon and eight hours after the incident, Tom walked out of the dental office with a new permanent crown. It cost approximately the same amount as my crown had cost a year earlier.

Lean is about Flow Efficiency

The book This is Lean (Modig and Ahlström, 2013) describes “lean” as a relentless focus on efficiency – but not the kind of efficiency that cuts staff and money, nor the kind of efficiency that strives to keep every resource busy all of the time. In fact, a focus on resource efficiency will almost always destroy overall efficiency, the authors contend, because fully utilized machines (and people) create huge traffic jams, which end up creating a lot of extra work. Instead, Modig and Ahlström demonstrate that lean is about flow efficiency – that is, the efficiency with which a unit of work (a flow unit) moves through the system.

Consider our dental experience. It took two weeks for me get a new crown, but in truth, only an hour and a half of that time was needed to actually fix the tooth; the rest of the time was mostly spent waiting. My flow efficiency was 1.5÷336 (two weeks) = 0.45%. On the other hand, Tom’s tooth was replaced in eight hours – 42 times faster – giving him a flow efficiency of 1.5÷8 = 18.75%.

In my case, the dental system was focused on the efficiency of the lab’s milling machine – no doubt an expensive piece of equipment. But add up all of the extra costs: a cast of the crown for the lab, a temporary crown for me, two separate hour-long sessions with the dentist, plus all of the associated logistics – scheduling, shipping, tracking, etc. In Tom’s case, the dental system was focused on the speed with which it could fix his tooth – which was good for us, because a long wait for a crown was not an option. True, the milling machine in the dentist’s office sits idle much of each day. (The dentist said he has to replace two crowns a day to make it economically feasible.) But when you add up the waste of temporary crowns, the piles of casts waiting for a milling machine, and the significant cost of recovering from a mistake – an idle milling machine makes a lot of sense.

What does flow efficiency really mean? Assume you have a camera and efficiency means keeping the camera busy – always taking a picture of some value-adding action. Where do you aim your camera? In the case of resource efficiency, the camera is aimed at the resource – the milling machine – and keeping it busy is of the utmost importance. In the case of flow efficiency, the camera is on the flow unit – Tom – and work on replacing his crown is what counts.  The fundamental mental shift that lean requires is this: flow efficiency trumps resource efficiency almost all of the time.

Lean Product Development: The Predecessor

During the 1980’s Japanese cars were capturing market share at a rate that alarmed US automakers. In Boston, both MIT and Harvard Business School responded by launching extensive studies of the automotive industry. In 1990 the MIT research effort resulted in the now classic book The Machine that Changed the World: the Story of Lean Production (Womack et al., 1990), which gave us the term “lean.” A year later, Harvard Business School published Product Development Performance. (Clark and Fujimoto, 1991) and the popular book Developing Products in Half the Time (Smith and Reinertsen, 1991) was released. These two 1991 books are foundational references on what came to be called “lean product development,” although the term “lean” would not be associated with product development for another decade.

Clark and Fujimoto documented the fact that US and European volume automotive producers took three times as many engineering hours and 50% more time to develop a car compared to Japanese automakers, yet the Japanese cars had substantially higher quality and cost less to manufacture. Clearly the Japanese product development process produced better cars faster and at lower cost that typical western development practices of the time. Clark and Fujimoto noted that the distinguishing features of Japanese product development paralleled features found in Japanese automotive production. For example, Japanese product development focused on flow efficiency, reducing information inventory, and learning based on early and frequent feedback from downstream processes. By contrast, product development in western countries focused on resource efficiency, completing each phase of development before starting the next, and following the original plan with as little variation as possible.

In 1991 the University of Michigan began its Japan Technology Management Program. Over the next several years, faculty and associate members included Jeffrey Liker, Allen Ward, Durward Sobek, John Shook, and Mike Rother. This group has published numerous books and articles on lean thinking, lean manufacturing, and lean product development, including The Toyota Product Development System (Morgan and Liker, 2006), and Lean Product and Process Development (Ward, 2007). The second book summarizes the essence of lean product development this way:

1. Understand that knowledge creation is the essential work of product development.
2. Charter a team of responsible experts led by an entrepreneurial system designer.
3. Manage product development using the principles of cadence, flow, and pull.

It is important to recognize that even though lean product development is based on the same principles as lean production, the practices surrounding development are, quite frankly, not the same as those considered useful in production. In fact, transferring lean practices from manufacturing to development has led to some disastrous results. For example, lean production emphasizes reducing variation – exactly the wrong thing to do in product development. The western practice of following a plan and measuring variance from a plan is often justified by the slogan “Do it right the first time.” Unfortunately, this approach does not allow for learning; it confines designs to those conceived when the least amount of knowledge is available. A fundamental practice in lean product development is to create variation (not avoid it) in order to explore the impact of multiple approaches. (This is called set-based engineering.)

The critical thing to keep in mind is that knowledge creation is the essential work of product development. While lean production practices support learning about and improving the manufacturing process, their goal is to minimize variation in the product. This is not appropriate for product development, where variation is an essential element of the learning cycles that are the foundation of good product engineering. Thus instead of copying lean manufacturing practices, lean product development practices must evolve from a deep understanding of fundamental lean principles adapted to a development environment.

Lean Software Development: A Subset of Lean Product Development

In 1975, computers were large, expensive, and rare. Software for these large machines was developed in the IT departments of large companies and dealt largely with the logistics of running the company – payroll, order processing, inventory management, etc. But as mainframes morphed into minicomputers, personal computers, and microprocessors, it became practical to enhance products and services with software. Then the internet began to invade the world, and it eventually became the delivery mechanism for a large fraction of the software being developed today. As software moved from supporting business process to enabling smart products and becoming the essence services, software engineers moved from IT departments to line organizations where they joined product teams.

Today, most software development is not a stand-alone process, but rather a part of developing products or services. Thus lean software development might be considered a subset of lean product development; certainly the principles that underpin lean product development are the same principles that form the basis of lean software development.

Agile and Lean Software Development: 2000 - 2010

It’s hard to believe these days, but in the mid 1990’s, developing software was a slow and painful process found in the IT departments of large corporations. As the role of software expanded and software engineers moved into line organizations, reaction against the old methods grew. In 1999, Kent Beck proposed a radically new approach to software development in the book “Extreme Programming Explained” (Beck, 1999). In 2001 the Agile Manifesto (Beck et al., 2001) gave this new approach a name – “Agile.”

In 2003, the book Lean Software Development (Poppendieck, 2003) merged lean manufacturing principles with agile practices and the latest product development thinking, particularly from the book Managing the Design Factory (Reinertsen, 1997). Lean software development was presented as a set of principles that form a theoretical framework for developing and evolving agile practices:

1. Eliminate waste
2. Amplify learning
3. Decide as late as possible
4. Deliver as fast as possible
5. Empower the team
6. Build quality in
7. See the whole

Although the principles of lean software development are consistent with lean manufacturing and (especially) lean product development, the specific practices that emerged were tailored to a software environment and aimed at the flaws in the prevailing software development methodologies. One of the biggest flaws at the time was the practice of moving software sequentially through the typical stages of design, development, test, and deployment – with handovers of large inventories of information accumulating at each stage. This practice left testing and integration at the end of the development chain, so defects went undetected for weeks or months before they were discovered. Typical sequential processes reserved a third of a release cycle for testing, integration, and defect removal. The idea that it was possible to “build quality in” was not considered a practical concept for software.

To counter sequential processes and the long integration and defect removal phase, agile software development practices focused on fast feedback cycles in these areas:

1. Test-driven development: Start by writing tests (think of them as executable specifications) and then write the code to pass the tests. Put the tests into a test harness for ongoing code verification.
   
2. Continuous integration: Integrate small increments of code changes into the code base frequently – multiple times a day – and run the test harness to verify that the changes have not introduced errors.
   
3. Iterations: Develop working software in iterations of two-to four weeks; review the software at the end of each iteration and make appropriate adjustments.
   
4. Cross-functional teams: Development teams should include customer proxies and testers as well as developers to minimize handovers.

During its first decade, agile development moved from a radical idea to a mainstream practice. This was aided by the widespread adoption of Scrum, an agile methodology which institutionalized the third and fourth practices listed above, but unfortunately omitted the first two practices.

The Difference between Lean and Agile Software Development

When it replaced sequential development practices typical at the time, agile software development improved the software development process most of the time – in IT departments as well as product development organizations. However, the expected organizational benefits of agile often failed to materialize because agile focused on optimizing software development, which frequently was not the system constraint. Lean software development differed from agile in that it worked to optimize flow efficiency across the entire value stream “from concept to cash.” (Note the subtitle of the book Implementing Lean Software Development: From Concept to Cash (Poppendieck, 2006)). This end-to-end view was consistent with the work of Taiichi Ohno, who said:

“All we are doing is looking at the time line, from the moment the customer gives us an order to the point when we collect the cash. And we are reducing that time line by removing the non-value-added wastes.” (Ohno, 1988. p ix)

Lean software development came to focus on these areas:

1. Build the right thing: Understand and deliver real value to real customers. 
2. Build it fast: Dramatically reduce the lead time from customer need to delivered solution. 
3. Build the thing right: Guarantee quality and speed with automated testing, integration and deployment. 
4. Learn through feedback: Evolve the product design based on early and frequent end-to-end feedback.

Let’s take a look at each principle in more detail:

1. Understand and deliver real value to real customers.

A software development team working with a single customer proxy has one view of the customer interest, and often that view is not informed by technical experience or feedback from downstream processes (such as operations). A product team focused on solving real customer problems will continually integrate the knowledge of diverse team members, both upstream and downstream, to make sure the customer perspective is truly understood and effectively addressed. Clark and Fujimoto call this “integrated problem solving” and consider it an essential element of lean product development.

2. Dramatically reduce the lead time from customer need to delivered solution.

A focus on flow efficiency is the secret ingredient of lean software development. How long does it take for a team to deploy into production a single small change that solves a customer problem?  Typically it can take weeks or months – even when the actual work involved consumes only an hour. Why? Because subtle dependencies among various areas of the code make it probable that a small change will break other areas of the code; therefore it is necessary to deploy large batches of code as a package after extensive (usually manual) testing. In many ways the decade of 2000-2010 was dedicated to finding ways to break dependencies, automate the provisioning and testing processes, and thus allow rapid independent deployment of small batches of code.

3. Guarantee quality and speed with automated testing, integration and deployment.

It was exciting to watch the expansion of test-driven development and continuous integration during the decade of 2000-2010. First these two critical practices were applied at the team level – developers wrote unit tests (which were actually technical specifications) and integrated them immediately into their branch of the code. Test-driven development expanded to writing executable product specifications in an incremental manner, which moved testers to the front of the process. This proved more difficult than automated unit testing, and precipitated a shift toward testing modules and their interactions rather than end-to-end testing. Once the product behavior could be tested automatically, code could be integrated into the overall system much more frequently during the development process – preferably daily – so software engineers could get rapid feedback on their work.

Next the operations people got involved and automated the provisioning of environments for development, testing, and deployment. Finally teams (which now included operations) could automate the entire specification, development, test, and deployment processes – creating an automated deployment pipeline. There was initial fear that more rapid deployment would cause more frequent failure, but exactly the opposite happened. Automated testing and frequent deployment of small changes meant that risk was limited. When errors did occur, detection and recovery was much faster and easier, and the team became a lot better at it. Far from increasing risk, it is now known that deploying code frequently in small batches is best way to reduce risk and increase the stability of large complex code bases.

4. Evolve the product design based on early and frequent end-to-end feedback.

To cap these remarkable advancements, once product teams could deploy multiple times per day they began to close the loop with customers. Through canary releases, A/B testing, and other techniques, product teams learned from real customers which product ideas worked and how to fine tune their offerings for better business results.

When these four principles guided software development in product organizations, significant business-wide benefits were achieved. However, IT departments found it difficult to adopt the principles because they required changes that lay beyond span of control of most IT organizations.

Lean Software Development: 2010 - 2015

2010 saw the publication of two significant books about lean software development. David Anderson’s book Kanban (Anderson, 2010) presented a powerful visual method for managing and limiting work-in-process (WIP).  Just at the time when two week iterations began to feel slow, Kanban gave teams a way to increase flow efficiency while providing situational awareness across the value stream. Jez Humble and Dave Farley’s book Continuous Delivery (Humble and Farley, 2010) walked readers through the steps necessary to achieve automated testing, integration and deployment, making daily deployment practical for many organizations. A year later, Erik Reis’s book The Lean Startup (Reis, 2011) showed how to use the rapid feedback loop created by continuous delivery to run experiments with real customers and confirm the validity of product ideas before incurring the expense of implementation.

Over the next few years, the ideas in these books became mainstream and the limitations of agile software development (software-only perspective and iteration-based delivery) were gradually expanded to include a wider part of the value stream and a more rapid flow. A grassroots movement called DevOps worked to make automated provision-code-build-test-deployment pipelines practical. Cloud computing arrived, providing easy and automated provisioning of environments. Cloud elements (virtual machines, containers), services (storage, analysis, etc.) and architectures (microservices) made it possible for small services and applications to be easily and rapidly deployed. Improved testing techniques (simulations, contract assertions) have made error-free deployments the norm.

The State of Lean Software Development in 2015

Today’s successful internet companies have learned how to optimize software development over the entire value stream.  They create full stack teams that are expected to understand the consumer problem, deal effectively with tough engineering issues, try multiple solutions until the data shows which one works best, and maintain responsibility for improving the solution over time. Large companies with legacy systems have begun to take notice, but they struggle with moving from where they are to the world of thriving internet companies.

Lean principles are a big help for organizations that want to move from old development techniques to modern software approaches. For example, (Calçado, 2015) shows how classic lean tools – Value Stream Mapping and problem solving with Five Whys – were used to increase flow efficiency at Soundcloud, leading over time to a microservices architecture. In fact, focusing on flow efficiency is an excellent way for an organization to discover the most effective path to a modern technology stack and development approach.

For traditional software development, flow efficiency is typically lower than 10%; agile practices usually bring it up to 30 or 40%. But in thriving internet companies, flow efficiency approaches 70% and is often quite a bit higher. Low flow efficiencies are caused by friction – in the form of batching, queueing, handovers, delayed discovery of defects, as well as misunderstanding of consumer problems and changes in those problems during long resolution times. Improving flow efficiency involves identifying and removing the biggest sources of friction from the development process.

Modern software development practices – the ones used by successful internet companies – address the friction in software development in a very particular way. The companies start by looking for the root causes of friction, which usually turn out to be 1) misunderstanding of the customer problem, 2) dependencies in the code base and 3) information and time lost during handovers and multitasking. Therefore they focus on three areas: 1) understanding the consumer journey, 2) architecture and automation to expose and reduce dependencies, and 3) team structures and responsibilities. Today (2015), lean development in software usually focuses on these three areas as the primary way to increase efficiency, assure quality, and improve responsiveness in software-intensive systems.

1. Understand the Customer Journey.

Software-intensive products create a two-way path between companies and their consumers. A wealth of data exists about how products are used, how consumers react to a product’s capabilities, opportunities to improve the product, and so on. Gathering this data and analyzing it has become an essential capability for companies far beyond the internet world: car manufactures, mining equipment companies, retail stores and many others gather and analyze “Big Data” to gain insights into consumer behavior. The ability of companies to understand their consumers through data has changed the way products are developed. (Porter, 2015) No longer do product managers (or representatives from “the business”) develop a roadmap and give a prioritized list of desired features to an engineering team. Instead, data scientists work with product teams to identify themes to be explored. Then the product teams identify consumer problems surrounding the theme and experiment with a range of solutions. Using rapid deployment and feedback capabilities, the product team continually enhances the product, measuring its success by business improvements, not feature completion.

2. Architecture and Automation.

Many internet companies, including Amazon, Netflix, eBay, realestate.com.au, Forward, Twitter, PayPal, Gilt, Bluemix, Soundcloud, The Guardian, and even the UK Government Digital Service have evolved from monolithic architectures to microservices. They found that certain areas of their offerings need constant updating to deal with a large influx of customers or rapid changes in the marketplace. To meet this need, relatively small services are assigned to small teams which then split their services off from the main code base in such a way that each service can be deployed independently. A service team is responsible for changing and deploying the service as often as necessary (usually very frequently), while insuring that the changes do not break any upstream or downstream services. This assurance is provided by sophisticated automated testing techniques as well as automated incremental deployment.

Other internet companies, including Google and Facebook, have maintained existing architectures but developed sophisticated deployment pipelines that automatically send each small code change through a series of automated tests with automatic error handling. The deployment pipeline culminates in safe deployments which occur at very frequent intervals; the more frequent the deployment, the easier it is to isolate problems and determine their cause. In addition, these automation tools often contain dependency maps so that feedback on failures can be sent directly to the responsible engineers and offending code can be automatically reverted (taken out of the pipeline in a safe manner).

These architectural structures and automation tools are a key element in a development approach that uses Big Data combined with extremely rapid feedback to improve the consumer journey and solve consumer problems. They are most commonly found in internet companies, but are being used in many others, including organizations that develop embedded software. (See case study, below.)

3. Team Structures and Responsibilities.

When consumer empathy, data analytics and very rapid feedback are combined, there is one more point of friction that can easily reduce flow efficiency. If an organization has not delegated responsibility for product decisions to the team involved in the rapid feedback loop, the benefits of this approach are lost. In order for such feedback loops to work, teams with a full stack of capabilities must be given responsibility to make decisions and implement immediate changes based on the data they collect. Typically such teams include people with product, design, data, technology, quality, and operations backgrounds. They are responsible for a improving set of business metrics rather than delivering a set of features. An example of this would be the UK Government Digital Service (GDS), where teams are responsible for delivering improvements in four key areas: cost per transaction, user satisfaction, transaction completion rate, and digital take-up.

It is interesting to note that UK laws makes it difficult to base contracts on such metrics, so GDS staffs internal teams with designers and software engineers and makes them responsible for the metrics. Following this logic to its conclusion, the typical approach of IT departments – contracting with their business colleagues to deliver a pre-specified set of features – is incompatible with full stack teams responsible for business metrics. In fact, it is rare to find separate IT departments in companies founded after the mid 1990’s (which includes virtually all internet companies). Instead, these newer companies place their software engineers in line organizations, reducing the friction of handovers between organizations.

In older organizations, IT departments often find it difficult to adopt modern software development approaches because they have inherited monolithic code bases intertwined with deep dependencies that introduce devious errors and thwart independent deployment of small changes. One major source of friction is the corporate database, once considered essential as the single source of truth about the business, but now under attack as a massive dependency generator. Another source of friction are outsourced applications, where even small changes are difficult and knowledge of how to make them no longer resides in the company. But perhaps the biggest source of friction in IT departments is the distance between their technical people and the company’s customers. Because most IT departments view their colleagues in line businesses as their customers, the technical people in IT lack a direct line of sight to the real customers of the company. Therefore insightful trade-offs and innovative solutions struggle to emerge.

The Future of Lean Software Development

The world-wide software engineering community has developed a culture of sharing innovative ideas, in stark contrast to the more common practice of keeping intellectual property and internally developed tools proprietary. The rapid growth of large, reliable, secure software systems can be directly linked to the fact that software engineers routinely contribute to and build upon the work of their world-wide colleagues through open source projects and repositories like GitHub. This reflects the longstanding practices of the academic world but is strikingly unique in the commercial world. Because of this intense industry-wide knowledge sharing, methods and tools for building highly reliable complex software systems have advanced extraordinarily quickly and are widely available.

As long as the software community continues to leverage its knowledge-sharing culture it will continue to grow rapidly, because sophisticated solutions to seemingly intractable problems eventually emerge when many minds are focused on the problem. The companies that will benefit the most from these advances are the ones that not only track new techniques as they are being developed, but also contribute their own ideas to the knowledge pool.

As microstructured architectures and automated deployment pipelines become common, more companies will adopt these practices, some earlier and some later, depending on their competitive situation. The most successful software companies will continue to focus like a laser on delighting customers, improving the flow of value, and reducing risks. They will develop (and release as open source) an increasingly sophisticated set of tools that make software development easier, faster, and more robust. Thus a decade from now there will be significant improvements in the way software is developed and deployed. The Lean principles of understanding value, increasing flow efficiency, eliminating errors, and learning through feedback will continue to drive the evolution, but the term “lean” will disappear as it becomes “the way things are done.”

— Case Study —

Hewlett Packard LaserJet Firmware 

The HP LaserJet firmware department had been the bottleneck of the LaserJet product line for a couple of decades, but by 2008 the situation had turned desperate. Software was increasingly important for differentiating the printer line, but the firmware department simply could not keep up with the demand for more features. Department leaders tried to spend their way out of the problem, but more than doubling the number of engineers did little to help. So they decided to engineer a solution to the problem by reengineering the development process.

The starting point was to quantify exactly where all the engineers’ time was going.  Fully half of the time went to updating existing LaserJet printers or porting code between different branches that supported different versions the product. A quarter of the time went to manual builds and manual testing, yet despite this investment, developers had to wait for days or weeks after they made a change to find out if it worked. Another twenty percent of the time went to planning how to use the five percent of time that was left to do any new work. The reengineered process would have to radically reduce the effort needed to maintain existing firmware, while seriously streamlining the build and test process. The planning process could also use some rethinking.

It’s not unusual to see a technical group use the fact that they inherited a messy legacy code base as an excuse to avoid change. Not in this case. As impossible as it seemed, a new architecture was proposed and implemented that allowed all printers – past, present and even future – to operate off of the same code branch, determining printer-specific capabilities dynamically instead of having them embedded in the firmware. Of course this required a massive change, but the department tackled one monthly goal after another and gradually implemented the new architecture. But changing the architecture would not solve the problem if the build and test process remained slow and cumbersome, so the engineers methodically implemented techniques to streamline that process. In the end, a full regression test – which used to take six weeks – was routinely run overnight. Yes, this involved a large amount of hardware, simulation and emulation, and yes it was expensive. But it paid for itself many times over.

During the recession of 2008 the firmware department was required to return to its previous staffing levels. Despite a 50% headcount reduction, there was a 70% reduction in cost per printer program once the new architecture and automated provisioning system were in place in 2011. At that point there was a single code branch and twenty percent of engineering time was spend maintaining the branch and supporting existing products. Thirty percent of engineering time was spent on the continuous delivery infrastructure, including build and test automation. Wasted planning time was reclaimed by delaying speculative decisions and making choices based on short feedback loops. And there was something to plan for, because over forty percent of the engineering time was available for innovation.

This multi-year transition was neither easy nor cheap, but it absolutely was worth the effort. If you would like more detail, see (Gruver et al, 2013).

A more recent case study of how the software company Paddy Power moved to continuous delivery can be found in (Chen, 2015). In this case study the benefits of continuous delivery are listed: improved customer satisfaction, accelerated time to market, building the right product, improved product quality, reliable releases, and improved productivity and efficiency. There is really no downside to continuous delivery. Of course it is a challenging engineering problem that can require significant architectural modifications to existing code bases as well as sophisticated pipeline automation. But technically, continuous delivery is no more difficult than other problems software engineers struggle with every day. The real stumbling block is the change in organizational structure and mindset required to achieve serious improvements in flow efficiency.

— End Case Study —

Credit

This essay is a preprint of the author’s original manuscript of a chapter to be published in Netland and Powell (eds) (2016) "Routledge Companion to Lean Management"

References

Anderson, David. Kanban, Blue Hole Press, 2010

Beck, Kent. Extreme Programming Explained, Addison-Wesley, 2000

Beck, Kent et al. Manifesto for Agile Software Development, http://agilemanifesto.org/, 2001

Calçado, Phil. How we ended up with microservices. http://philcalcado.com/2015/09/08/how_we_ended_up_with_microservices.html

Chen, Lianping. "Continuous Delivery: Huge Benefits but Challenges Too" IEEE Software 32 (2). 50-54. 2015

Clark, Kim B. and Takahiro Fujimoto. Product Development Performance, Harvard Business School Press, 1991

Gruver, Gary, Mike Young, and Pat Fulghum. A Practical Approach to Large-Scale Agile Development, Pearson Education, 2013

Humble, Jez and David Farley. Continuous Delivery, Addison-Wesley Professional, 2010

Modig, Niklas, and Par Ahlstrom. This is Lean, Stockholm: Rheologica Publishing, 2012

Morgan, James M and and Jeffrey K Liker. The Toyota Product Development System, Productivity Press, 2006

Ohno, Taiichi. Toyota Production System, English, Productivity, Inc. 1988, published in Japanese in 1978

Poppendieck, Mary and Tom. Lean Software Development, Addison Wesley, 2003

Poppendieck, Mary and Tom. Implementing Lean Software Development, Addison Wesley, 2006

Porter, Michael E. and James E. Heppelmann. How Smart, Connected Products are Transforming Companies, Harvard Business Review 93 (10), 97-112, 2015

Reinertsen, Donald G. Managing the Design Factory, The Free Press, 1997

Ries, Eric. The Lean Startup, Crown Business, 2011

Smith, Preston G. and Donald G. Reinertsen. Developing Products in Half the Time, Van Nostrand Reinhold/co Wiley, 1991

Ward, Allen. Lean Product and Process Development, Lean Enterprise Institute, 2007

Womack, James P., Daniel T. Jones, and Daniel Roos. The Machine That Changed the World; the Story of Lean Production, Rawson & Associates, 1990

How Cadence Predicts Process

If you want to learn a lot about a software development organization very quickly, there are a few simple questions you might ask. You might find out if the organization focuses on projects or products. You might look into what development process it uses. But perhaps most the revealing question is this: How far apart are the software releases?

It is rare that new software is developed from scratch; typically existing software is expanded and modified, usually on a regular basis. As a result, most software development shops that we run into are focused on the next release, and very often releases are spaced out at regular intervals. We have discovered that a significant differentiator between development organizations is the length of that interval – the length of the software release cycle.

Organizations with release cycles of six months to a year (or more) tend to work like this: Before a release cycle begins, time is spent deciding what will be delivered in the next release. Estimates are made. Managers commit. Promises are made to customers. And then the development team is left to make good on all of those promises. As the code complete date gets closer, emergencies arise and changes have to be made, and yet, those initial promises are difficult to ignore. Pressure increases.

If all goes according to plan, about two-thirds of the way through the release cycle, code will be frozen for system integration testing (SIT) and user acceptance testing (UAT). Then the fun begins, because no one really knows what sort of unintended interactions will be exposed or how serious the consequences of those interactions will be. It goes without saying that there will be defects; the real question is, can all of the critical defects be found and fixed before the promised release date?

Releases are so time-consuming and risky that organizations tend to extend the length of their release cycle so as not to have to deal with this pain too often. Extending the release cycle invariably increases the pain, but at least the pain occurs less frequently. Counteracting the tendency to extend release cycles is the rapid pace of change in business environments that depend on software, because the longer release cycles become a constraint on business flexibility. This clash of cadences results in an intense pressure to cram as many features as possible into each release. As lengthy release cycles progress, pressure mounts to add more features, and yet the development organization is expected to meet the release date at all costs.

Into this intensely difficult environment a new idea often emerges – why not shorten the release cycle, rather than lengthen it? This seems like an excellent way to break the death spiral, but it isn’t as simple as it seems. The problem, as Kent Beck points out in his talk “Software G Forces: the Effects of Acceleration,” is that shorter release cycles demand different processes, different sales strategies, different behavior on the part of customers, and different governance systems. These kinds of changes are notoriously difficult to implement.

Quick and Dirty Value Stream Map
I’m standing in front of a large audience. I ask the question: “Who here has a release cycle longer than three months?” Many hands go up. I ask someone whose hand is up, “How long is your release cycle?” She may answer, “Six months.” “Let me guess how much time you reserve for final integration, testing, hardening, and UAT,” I say. “Maybe two months?” If she had said a year, I would have guessed four months. If she had said 18 months, I would have guessed 6 months. And my guess would be very close, every time. It seems quite acceptable to spend two-thirds of a release cycle building buggy software and the last third of the cycle finding and fixing as many of those bugs as possible.

The next question I ask is: When do you decide what features should go into the release? Invariably when the release cycle is six months or longer, the answer is: “Just before we start the cycle.” Think about a six month release cycle: For the half year prior to the start of the cycle, demand for new or changed features has been accumulating – presumably at a steady pace. So the average wait of a feature to even be considered for development is three months – half of the six month cycle time. Thus – on the average – it will take a feature three months of waiting before the cycle begins, plus six months of being in development and test before that feature is released to customers; nine months in all.

Finally I ask, “About how many features might you develop during a six month release cycle?” Answers to this vary widely from one domain to another, but let’s say I am told that that about 25 features are developed in a six month release, which averages out to about one feature per week.

This leaves us with a quick and dirty vision of the value stream: a feature takes a week to develop and best case it takes nine months (38 weeks) to make it through the system. So the process efficiency is 1÷38, or about 2.6%. A lot of this low efficiency can be attributed to batching up 25 features in a single release. A lot more can be attributed to the fact that only 4 months of the 9 total months are actually spent developing software – the rest of the time is spent waiting for a release cycle to start or waiting for integration testing to finish.

Why not Quarterly Releases?
With such dismal process efficiency, let’s revisit to the brilliant idea of shortening release cycles. The first problem we encounter is that at a six month cadence, integration testing generally takes about two months; if releases are annual, integration testing probably takes three or four months. This makes quarterly releases quite a challenge.

For starters, the bulk of the integration testing is going to have to be automated. However, most people rapidly discover that their code base is very difficult to test automatically, because it wasn’t designed or written to be tested automatically. If this sounds like your situation, I recommend that you read Gojko Adzic’s book “Specification by Example.” You will learn to think of automated tests as executable specifications that become living documentation. You will not be surprised to discover that automating integration tests is technically challenging, but the detailed case studies of successful teams will give you guidance on both the benefits and the pitfalls of creating a well architected integration test harness.

Once you have the beginnings of an automated integration test harness in place, you may as well start using it frequently, because its real value is to expose problems as soon as possible. But you will find that code needs to “done” in order to be tested in this harness, otherwise you will get a lot of false negatives. Thus all teams contributing to the release would do well to work in 2-4 week iterations and bring their code to a state that can be checked by the integration test harness at the end of every iteration. Once you can reasonably begin early, frequent integration testing, you will greatly reduce final integration time, making quarterly releases practical.

Be careful, however, not to move to quarterly releases without thinking through all of the implications. As Kent Beck noted in his Software G Forces talk, sales and support models at many companies are based on annual maintenance releases. If you move from an annual to a quarterly release, your support model will have to change for two reasons: 1) customers will not want to purchase a new release every quarter, and 2) you will not be able to support every single release over a long period of time. You might consider quarterly private releases with a single annual public release, or you might want to move to a subscription model for software support. In either case, you would be wise not to guarantee long term support for more than one release per year, or support will rapidly become very expensive.

From Quarterly to Monthly Releases
Organizations that have adjusted their processes and business models to deal with a quarterly release cycle begin to see the advantages of shorter release cycles. They see more stability, more predictability, less pressure, and they can be more responsive to their customers. The question then becomes, why not increase the pace and release monthly? They quickly discover that an additional level of process and business change will be necessary to achieve the faster cycle time because four weeks – twenty days – from design to deployment is not a whole lot of time.

At this cadence, as Kent Beck notes, there isn’t time for a lot of information to move back and forth between different departments; you need a development team that includes analysts and testers, developers and build specialists. This cross-functional team synchronizes via short daily meetings and visualization techniques such as cards and charts on the wall – because there simply isn’t time for paper-based communication. The team adopts processes to ensure that the code base always remains defect-free, because there isn’t time to insert defects and then remove them later. Both TDD (Test Driven Development) and SBE (Specification by Example) become essential disciplines.

From a business standpoint, monthly releases tend to work best with software-as-a-service (SaaS). First of all, pushing monthly releases to users for them to install creates huge support headaches and takes far too much time. Secondly, it is easy to instrument a service to see how useful any new feature might be, giving the development team immediate and valuable feedback.

Weekly / Daily Releases
There are many organizations that consider monthly releases a glacial pace, so they adopt weekly or even daily releases. At a weekly or daily cadence, iterations become largely irrelevant, as does estimating and commitment. Instead, a flow approach is used; features flow from design to done without pause, and at the end of the day or week, everything that is ready to be deployed is pushed to production. This rapid deployment is supported by a great deal of automation and requires a great deal of discipline, and it is usually limited to internal or SaaS environments.

There are a lot of companies doing daily releases; for example, one of our customers with a very large web-based business has been doing daily releases for five years. The developers at this company don’t really relate to the concept of iterations. They work on something, push it to systems test, and if it passes it is deployed at the end of the day. Features that are not complete are hidden from view until a keystone is put in place to expose the feature, but code is deployed daily, as it is written. Occasionally a roll-back is necessary, but this is becoming increasingly rare as the test suites improve. Managers at the company cannot imagine working in at a slower cadence; they believe that daily deployment increases predictability, stability, and responsiveness – all at the same time.

Continuous Delivery
Jez Humble and David Farley wrote “Continuous Delivery” to share with the software development community techniques they have developed to push code to the production environment as soon as it is developed and tested. But continuous delivery is not just a matter of automation. As noted above, sales models, pricing, organizational structure and the governance system all merit thoughtful consideration.

Every step of your software delivery process should operate at the same cadence. For example, with continuous delivery, portfolio management becomes a thing of the past; instead people make frequent decisions about what should be done next. Continuous design is necessary to keep pace with the downstream development, validation and verification flow. And finally, measurements of the “success” of software development have to be based on delivered value and improved business performance, because there is nothing else left to measure.

Breaking The Quality–Speed Compromise

In the hotly contested commodity business of assembling computers, Dell enjoys a 50% cost advantage over its competitors.[1] This commanding advantage comes from Dell’s exceptional responsiveness to customers, flawless operations, and remarkable speed of execution. Conventional wisdom once held that the low cost producer could not provide customized high quality products. But Dell decided that its customers could have it all – low cost computers with the latest technology, built to order and delivered in a week.

In the book Hardball, George Stalk notes that when an industry imposes a compromise on its customers, the company that breaks the compromise stands to gain a significant competitive advantage.[2] For example, the airline industry imposes a big compromise on travelers: if you want low cost tickets, you have to make your plans early and pay a stiff penalty to change them. Southwest Airlines breaks this compromise: its customers can apply the cost of unused tickets to a future flight without a change fee.

In the software development industry, we impose many compromises on our customers. We tell them that high quality software takes a lot of time; we ask them to decide exactly what they want when they don’t really know; we make it clear that changes late in the development process will be very expensive. There’s a significant competitive advantage waiting for companies that can break these compromises. In particular, I’d like to focus on breaking the compromise between quality and speed, because many companies have achieved great leverage by competing on the basis of time.

When I teach classes on Lean Software Development, the first thing we do is draw value stream maps of existing software development processes. Starting with a customer request, the class draws each step that the request goes through as it is turned into deployed software which solves the customer’s problem. The average time for each step is noted, as well as the time between steps, giving a picture of the total time it takes to respond to a customer.

Next the class determines how much of the time between request and deployment is spent actually working on the problem. Typically, less than 20% of the total time is spent doing work on the request; for 80+% of the time the request is waiting in some queue. For starters, driving down this queue-time will let us deliver software much faster without compromising quality.

But reducing wait time is not the only opportunity for faster software development. Typically the value stream maps in my classes show a big delay just before deployment, at a step which is usually called ‘verification’. Now, I don’t have any problem with verification just before deployment, but when I ask, “Do you find any defects during verification?” the answer is always “Yes.” Therein lies the problem. When a computer hits the end of Dell’s assembly line, it is powered on and it is expected to work. The verification step is not the time to find defects; by the time software hits verification, it should work.

The way to get rid of the big delay at verification is to move testing closer to coding – much closer. In fact, testing should happen immediately upon coding; if possible the test should have been written before the code. New code should be integrated into the overall system several times a day, with a suite of automated unit tests run each time. Acceptance tests for a feature should pass as soon as the feature is complete, and regression testing should be run on the integrated code daily or perhaps weekly.

Of course, this testing regime is not feasible with manual testing, automated unit and acceptance tests are required. While this may have been impractical a few years ago, the tools exist today to make automated testing practical. Obviously not all tests can be automated and not all automated test suites are fast enough to run frequently. But there are many ways to make automated testing more effective; for example, each layer is usually tested separately – ie. the business rules are tested below the GUI with most database calls mocked out.

In most of the value stream maps I see in my classes, there is a huge opportunity to move tests far forward in the process and catch defects at their source. Many companies spend a great deal of time tracking, prioritizing, and fixing a long queue of defects. Far better to never let a defect into the queue in the first place.

There is another area of my classes’ value stream maps that raises a flag. Toward the beginning of the map there is usually a step called ‘requirements’ which often interacts with a queue of change requests. Dealing with change requests takes a lot of time and approved changes create significant churn. There has been a feeling that if only we could get the requirements right, this ‘change churn’ would go away. But I generally find that the real problem is that the requirements were specified too early, when it was not really clear what was needed. The way to reduce requirements churn is to delay the detailed clarification of requirements, moving this step much closer to coding. This greatly reduces the change request queue, because you don’t need to change a decision that has not yet been made!

Toward the end of my classes, we draw a future value stream map, and invariably the new value stream maps show a dramatically shortened cycle time, the result of eliminating wait time, moving tests forward, and delaying detailed specification of requirements. We usually end up with a process in which cross-functional teams produce small, integrated, tested, deployment-ready packages of software at a regular cadence.

This kind of software development process exposes another compromise: conventional wisdom says that changes late in the development cycle are costly. If we are developing small bits of code without full knowledge of everything that the system will require, then we are going to have to be able to add new features late in the development process at about the same cost as incorporating them earlier.

The cost of adding or changing features depends on three things: the size of the change, the number of dependencies in the code, and whether or not the change is structural. Since we just agreed to keep development chunks small, let’s also agree to keep changes small. Then let’s agree that we are going to get the structural stuff right – including proper layering, modularization that fits the domain, appropriate scalability, etc.

We are left to conclude that the cost of non-structural change depends on the complexity of the code. There are several measurements of complexity, including the number of repetitions (the target is zero), the use of patterns (which reduce complexity), and McCabe scores (the number of decisions in a module). It has been shown that code with low complexity scores has the fewest defects, so a good measure of complexity is the number of defects.

Which brings us back to our testing regime. The most important thing we can do to break the compromises we impose on customers is to move testing forward and put it in-line with (or prior to) coding. Build suites of automated unit and acceptance tests, integrate code frequently, run the tests as often as possible. In other words, find and fix the defects before they even count as defects.

Companies that respond to customers a lot faster than their industry average can expect to grow three times faster and enjoy twice the profits of their competitors.[3] So there is a lot of competitive advantage available for the software development organization that can break the speed–quality compromise, and compete on the basis of time.

References

[1] Michael L. George and Stephen A. Wilson, Conquering Complexity in Your Business, McGraw-Hill, 2004, p.48.

[2] George Stalk and Rob Lachenauer, Hardball: Are you Playing to Play or Playing to Win, Harvard Business School Press, 2004

[3] George Stalk, Competing against Time: How time-based Competition is Reshaping Global Markets, Free Press, 2003, originally published in 1990. p.4.

Screen Beans Art, © A Bit Better Corporation

Why the Lean in Lean Six Sigma?

A free-wheeling mid-sized company ran smack up against the Sarbanes-Oxley Act a few months ago and found that it had to pause and take stock.[1] The company commissioned a small team to map out every place that financial data moved, and to no one’s surprise, it uncovered a lot of un-coordinated, manual processes. Next the team put into place some simple automation that managed inventory and financial data, an online order entry system, and even some automated human resources capabilities. The company found that with these more streamlined processes, days were cut out of order processing, inventories were reduced, and productivity was significantly improved.

The irony is that the fast-moving company had to slow down to speed up; more discipline led to higher speed. This is not an isolated case, there is a direct relationship between speed and discipline. Adding discipline should streamline a process, and streamlined processes don’t work without discipline.

When I first heard the term Lean Six Sigma, I wondered what Lean added to Six Sigma. I found that the answer is speed. The first principle of Lean Six Sigma is: Delight your customers with speed and quality. The second principle says: Improve process flow and speed. Lean Six Sigma emphasizes that speed is directly tied to excellence.

Speed is not the same thing as schedule. Schedule is about when something is supposed to get done, speed is about how fast it gets done. Speed has a bad reputation; it is often equated with hasty, undisciplined work. But if Lean Six Sigma has anything to teach us, it is that we should be looking for opportunities to streamline our core processes. This does not mean we should be compressing already tight schedules. It means that we first determine what our core processes are, and then focus on making them flow smoothly.

For example, core processes in software development would be naming conventions and coding standards, a configuration management system, an automated build process, a suite of automated unit tests that are built and maintained as part of the code, daily build/integrate/test cycles, acceptance testing integrated into the development process, and usability testing immediately after the features are implemented. Assuring that these disciplines are in place is fundamental to the smooth flow of any software development process.

However, the most important process to streamline in a development project is the knowledge-creating process. Whether we are developing a new product or a new software system, the fundamental thing we are doing is discovering what needs to be in the system in order to delight the customer. Lean thinking supports two basic disciplines for speeding up the knowledge creation process: short, frequent learning cycles and delayed commitment.

Short, frequent learning cycles are the antithesis of thorough front-end planning, but they are the best approach to processes which have the word ‘development’ in their title. In product development, for example, the typical process is to define requirements, choose a solution, refine the solution, and implement the solution. Although this process is common, it is not the best way to generate knowledge.

Toyota has a different approach, one that is much faster and delivers products of superior quality that consistently outsell the competition.[2] Toyota builds sets of possibilities to satisfy customer needs and then, through a series of combining and narrowing, the new product emerges. The combining and narrowing process is paced by timeboxed milestones that define stages of the narrowing process. Milestones are always met, despite the fact that there are no task breakouts or tracking. Decisions are delayed as long as possible, so that they can be based on the maximum amount of information.

When a project involves knowledge creation, rather than just knowledge replication, speed and quality come from improving the flow of creating knowledge. Many of our project management practices have a tendency to impede knowledge creation by forcing early choices and reducing the number of possibilities explored. This often leads us down bind alleys: after a lot of work has gone into a solution, we learn that it has a fatal flaw and we have to retrace our steps and repeat a whole lot of work.

The key to streamlining a development process is to clearly distinguish between true knowledge creating iterations and iterations that lead down blind alleys. Knowledge creating iterations explore multiple options and leave as many possibilities open as possible, delaying decisions until the last responsible moment.

Returning to our software development example, lean practices promote speed and flexibility by implementing core disciplines that promote change tolerance and allow decisions to be delayed as long as possible. Lean software development changes the focus from gathering requirements to encoding all requirements in tests. It introduces the concept of refactoring, that is, creating a simple design at the beginning of development to handle early requirements, and then improving the design later as more requirements are discovered. Finally, lean software development requires full testing and integration of code as soon as it is developed, on a daily basis at minimum. The result is easily maintainable code that has been designed to be flexible and built to be rapidly changed.

In lean software development, scope is not set at the beginning; small features sets are added based on priority determined by their ROI.[3] This tends to lead to a significant increase in both speed and productivity for a simple reason: most of the features we put into software systems are never going to be used. How can this be? When we freeze scope early, we encourage our customers, who don’t really know what they want, to ask for everything they can imagine. When we delay commitment on scope until we are well into the knowledge generating process, we end up reducing scope down to the minimum set that is really going to pay off.

If we’re not careful, Six Sigma might lead us to apply practices aimed at improving replication processes to knowledge generating processes. This often leads to slow, unresponsive, change-intolerant practices that are not appropriate for knowledge creation. Quite often the slow, deliberate nature of these practices is mistaken to be a sign of good discipline. But when we add Lean to Six Sigma, we discover that we need to re-think these slow processes, because we have come to understand that speed, discipline, and excellence go hand-in-hand.

References
[1] “Better Safe Than Sorry,” by Jeffrey Rothfeder, CIO Insight, February 2004

[2] Product Development for the Lean Enterprise, by Michael N. Kennedy, Oaklea Press, 2003.

[3] Software by Numbers, by Mark Denne and Jane Cleland-Huang, Prentice Hall, 2004

Screen Beans Art, © A Bit Better Corporation

Software Development Productivity

Productivity is the prime determinant of our standard of living. If the revenue generated per work hour goes up, income levels rise. If productivity lags, wages also decline. When all else is equal in a market, the more productive company will enjoy greater profits. Thus the key to sustaining and increasing wages in the software development industry is year-to-year improvements in software development productivity.

Over the decade of the 1990’s, productivity increases in the technology sector resulted largely from increased capability of the underlying technology and intense customer demand for new technology. In recent years, however, software development productivity has stagnated as the demand for newer, faster technology has flattened. Thus it should come as no surprise that wages in the software development profession have flattened or declined. In order for the software development industry to see rising incomes, year-to-year increases in software development productivity must be generated.

As the technology sector matures, it can no longer depend on increasing growth in the underlying technology to fuel productivity increases. The time has come for serious efforts to increase productivity through more efficient use of labor and more effective value propositions for customers. This is how more mature economic sectors have been increasing productivity for decades.

But first we need to define what we mean by productivity improvement in the software development industry. Traditionally, we have measured productivity as thousand lines of code (kloc) per labor hour. However, the key process of a development activity is the transformation ideas of into products. To measure the real productivity of software development, we need look at how efficiently and effectively we turn ideas into software. So perhaps we should start with a new definition of software development productivity:

• For companies that develop and sell software as a product, productivity may be defined as the revenue generated per employee.
• For internal IT organizations, productivity may be defined as increased revenue realized by the supported business per dollar spent by the IT organization [1]

There are three basic approaches to productivity improvement:

1. Reduce product costs by eliminating investments in product features that customers do not find valuable.
2. Reduce indirect costs by streamlining processes and eliminating inefficiencies in development, delivery and support.
3. Increase revenue by adding more value to a product so customers will pay more for it.

Let’s explore each of these approaches to improving software development productivity.

1. Reduce Software Development Effort.
About two-thirds of the features of a typical software system are seldom or never used. Only twenty percent of the features are used frequently. [2]  Eliminating extra features that no one really wants represents the single largest opportunity for increasing software development productivity in most organizations.

Extra features are generated by a software development process that attempts to nail down the features in a system at the beginning of the development process. Typically, customers are asked to decide at the start of a project what features they want. Often they have little incentive to keep the features list short, but they are penalized if the forget to include something. Can there be any wonder that a feature list generated with such incentives contains far more features than are really necessary?

The biggest opportunity for reducing software development effort is to limit overproduction of features by developing features on an as-needed basis. For many companies, this may require a paradigm shift in architecture and design. However, it is becoming increasingly apparent that there are many disciplined approaches to software development that provide for an emergent architecture. Key approaches include:

• Automated test harnesses developed at the same time as the underlying code
• Refactoring the code on an on-going basis to keep its design simple and clean

2. Streamline the Development Processes.
The measure of maturity of an organization is the speed at which it can reliably and repeatedly execute it’s key processes. A mature software development organization is one that can rapidly, reliably and reliably translate customer needs into deployed code. Too often, rapid software development has been equated with sloppy work, so attempts to streamline the development process are often looked upon with suspicion. However, in industry after industry, when sequential development processes are replaced with concurrent development processes, costs are slashed, quality is improved, and development time is dramatically reduced.

Organizations which record year-on-year productivity improvements spend a lot of time focusing on streamlining key processes while increasing their reliability. They do this by focusing on the flow of value through the process. There are three main techniques used to do this:

• Value stream mapping is a tried and true approach to streamlining value-creating processes. A value stream map of the current state is invaluable for spotting waste; a value stream map of the desired future state is a roadmap for process improvement. Poppendieck.LLC helps organizations use value stream mapping to improve software development productivity.
• Kaizen events are a typical implementation vehicle for streamlining operational processes. In software development, a modification of Kaizen events is usually required, because software improvement efforts usually take more time than is generally allocated for Kaizen events. We facilitate effective Kaizen events for software development.
• An Integrated Product Team (IPT) is frequently used to facilitate information flow across an entire development team. Software development IPT’s involve not only architects, designers and developers, but also those responsible for deployment, operations, customer support, and maintenance.

3. Increase Customer Value.
Understanding customer value well enough to obtain additional revenue is the third key to increasing software development productivity. In general, customers cannot be relied upon to tell a software development organization how to increase the value of its offerings. In fact, customers generally think of software as a tool that should adapt to their needs over time. This makes understanding customer value elusive not only during a software development project, but even after deployment. And yet, the price that can be charged for software is directly related to understating how to increase its value proposition.

We are not likely to increase software’s value proposition unless we increase our understanding how customers might use our software to create value for themselves. There are three steps to increasing customer value:

• Iterative development with frequent releases creates a short feedback loop between customers and developers. The key to successful iterative development is to prioritize the feature list, implement features in order of business value, and deploy them as soon as possible. The short feedback loop created by iterative development and early deployment not only limits the development of unnecessary code, it brings to light innovative new uses of technology that can significantly improve business results.
• The next step to understanding customer value is to understand how our customers create value for their customers. Increasing the support of key value creating processes for customers is the most likely source of increased revenue for software development. One method of discovering how our customers create value is to focus on their key processes and map their value stream.
• The final step to providing customer value is to link organizations through partnerships that focus on the overall productivity of the combined enterprise. Peter Drucker noted in Management Challenge for the 21st Century [3] that “In every single case…the integration into one management system of enterprises that are linked economically rather than controlled legally, has given a cost advantage of at least 25 percent and more often 30 percent.” The bottom line is that when organizations work together for their mutual benefit rather than optimizing the results of the individual organizations, a large increase in overall productivity can be realized.

Software development organizations have always emphasized process; however, the focus has been on predictable delivery of pre-defined scope rather than increased productivity. Not surprisingly, processes which did not value productivity did not deliver productivity increases; quite often they decreased productivity instead. We need to move from processes which define scope early to processes which allow only the most valuable scope to be addressed. We need to move from slow processes with many wasteful steps to streamlined processes which eliminate non-value-adding activities. We need to focus on increasing revenue potential over cutting costs by discovering how we can help our customers increase their productivity.
____________________
References and Notes
[1] Productivity for internal IT organizations may be also defined as increased revenue realized by the supported business per employee in the IT organization, but such a definition would ignore make-vs-buy decisions, not to mention outsourcing.

[2] Johnson, Jim, Chairman of The Standish Group, ‘ROI, It’s Your Job,’ Keynote: Third International Conference on Extreme Programming, Alghero, Italy, May, 26-29, 2002

[3] Management Challenge for the 21st Century, Peter Drucker, Harper Business, 2001, p 33.

Lean Software Development

I often hear developers say: Life would be so much easier if customers would just stop changing their minds. In fact, there is a very easy way to keep customers from changing their minds – give them what they ask for so fast that they don’t have the time to change their minds.

When you order something on-line, it’s usually shipped before you have time for second thoughts. Because it ships so fast, you can wait until you’re sure of what you want before you order. But once you place the order, you have very little time to change your mind.

The idea behind lean thinking is exactly this: Let customers delay their decisions about exactly what they want as long as possible, and when they ask for something, give it to them so fast they don’t have time to change their minds.

Sure, you say, this is fine for Amazon.com, but how does it relate to software development? The way to deliver things rapidly is to deliver them in small packages. The bigger the increment of functionality you try to deliver, the longer it takes to decide what is needed and then get it developed, tested, and deployed. Maintenance programmers have known this for years. When a piece of production code breaks, they find the cause, create a patch, test it rigorously, and release it to production – usually in the space of a few hours or days.

But, you say, development is different; we need to develop a large system, we need a design, and we can’t deploy the system piecemeal. Even large systems that have to be deployed all-at-once should be developed in small increments. True, design is needed, but there is scant proof that great designs are achieved by meticulously gathering detailed requirements and analyzing them to death. Great designs come from great designers, and great designers understand that designs emerge as they develop a growing understanding of the problem.

As Harold Thimbleby said in Delaying Commitment [1], “In many disciplines, the difference between amateurs and experts seems to be that experts know how to delay their commitments…. Amateurs, on the other hand, try to get things completely right the first time and often fail… in their anxious desire to avoid error, they make early commitments – often the wrong ones… In fact, the expert’s strategy of postponing firm decisions, discovering constraints, and then filling in the details is a standard heuristic to solve problems.”

Lean Thinking
Over the last two decades, Lean Thinking has created fundamental transformations in the way we organize manufacturing, logistics, and product development. For example, concurrent development has replaced sequential development in industries from airlines to automobiles, cutting product development time by perhaps a third and development cost in half, while improving the quality and timeliness of the product. There are those who think that rapid development is equivalent to shoddy work, but lean organizations have demonstrated quite the opposite. The measure of the maturity of an organization is the speed with which it can reliably and repeatedly respond to a customer request.

Yes, you heard that right. Maturity is not measured by the comprehensiveness of process documentation or the ability to make detailed plans and follow them. It is measured by operational excellence, and the key indicator of operational excellence is the speed with which the organization can reliably and repeatedly serve its customers. Thirty years ago, Frederick W. Smith, the founder of FedEx, envisioned an overnight delivery system the seemed ridiculous to most people; today even the post office offers overnight delivery. A decade ago, Toyota could develop a new car twice as fast as GM; today all automotive companies measure and brag about the speed with which they can bring a new car to market.

Let’s revisit those customers you wish would make up their minds and stick to their decisions. Ask yourself this question: Once they do make up their minds, how fast can you reliably and repeatedly deliver what they want? Perhaps the problem does not lie in customers who can’t make up their minds and keep changing their decisions. Perhaps the problem lies in asking them to decide well before they have the necessary information to make a decision, and in taking so long to deliver that their circumstances change.

Principles of Lean Software Development
There are seven principles of Lean Software Development, drawn from the seven principles of Lean Thinking. These principles are not cook-book recipes for software development, but guideposts for devising appropriate practices for your environment.[2] These lean principles have led to dramatic improvements in areas as diverse as military logistics, health care delivery, building construction, and product development. When wisely translated to your environment, they can change your basis of competition.

Eliminate Waste
All lean thinking starts with a re-examination of what waste is and an aggressive campaign to eliminate it. Quite simply, anything you do that does not add value from the customer perspective is waste. The seven wastes of software development are: 

• Partially Done Work (the “inventory” of a development process)
• Extra Processes (easy to find in documentation-centric development)
• Extra Features (develop only what customers want right now)
• Task Switching (everyone should do one thing at a time)
• Waiting (for instructions, for information)
• Handoffs (tons of tacit knowledge gets lost)
• Defects (at least defects that are not quickly caught by a test)

All lean approaches focus on eliminating waste by looking at the flow of value from request to delivery. So if a customer wants something, what steps does that customer request have to go thorough to get delivered to the customer? How fast does that process flow? If a customer request waits in a queue for approval, a queue for design, a queue for development, a queue for testing, and a queue for deployment, work does not flow very fast. The idea is to create cells (or teams) of people chartered to take each request from cradle to grave, rapidly and without interruption. Then value flows.

Amplify Learning
The game of development is a learning game: hypothesize what might work, experiment to see if it works, learn from the results, do it again. People who design experiments know that greatest learning occurs when half of the experiments fail, because this exposes the boundary conditions. A development environment is not a place for slogans such as:

• Plan The Work And Work The Plan
• Do It Right The First Time
• Eliminate Variability

The learning that comes from short feedback loops is critical to any process with inherent variation. The idea is not to eliminate variation, it is to adapt to variation through feedback. Your car’s cruise control adapts to hilly terrain by frequently measuring the difference desired speed and actual speed and adjusting the accelerator. Similarly, software development uses frequent iterations to measures the difference between what the software can do and what the customer wants, and makes adjustments accordingly.

A lean development environment focuses on increasing feedback, and thus learning. The primary way to do this in software development is with short, full-cycle iterations. Short means a week to a month. Full cycle means the iteration results in working software: tested, integrated, deployable code. There should be a bias to deploy each iteration into production, but when that is not possible, end users should simulate use of the software in a production-equivalent environment.

We have known for a long time that iterative (evolutionary) development is the best approach for software development. In 1987 the report of the Defense Science Board Task Force on Military Software noted: “Document-driven, specify-then-build approach lies at the heart of so many DoD software problems…. Evolutionary development is best technically, and it saves time and money.”[3]


Delay Commitment
Delaying commitment means keeping your options open as long as possible. The fundamental lean concept is to delay irreversible decisions until they can be made based on known events, rather than forecasts. Economic markets develop options as a way to deal with uncertainty. Farmers, for example, can buy an option on the future price of grain. If the price drops, they have protected their profits. If the price raises, they can ignore the option and sell at the higher price. Options let people delay decisions until they have more information.

There are a lot of ways to keep options open in software development. Here are a few:

• Share partially complete design information.
• Organize for direct, worker-to-worker collaboration.
• Develop a sense of when decisions must be made.
• Develop a sense of how to absorb changes.
• Avoid Repetition
• Separate Concerns
• Encapsulate Variation
• Defer Implementation of Future Capabilities
• Commit to Refactoring
• Use Automated Test Suites

Deliver Fast
To those who equate rapid software development with hacking, there seems to be no reason to deliver results fast, and every reason to be slow and careful. Similarly, when Just-in-Time concepts surfaced in Japan in the early 1980’s, most western manufacturers could not fathom why they made sense. Everybody knew that the way to make customers happy was to have plenty of product on the shelf, and the way to maximize profits was to build massive machines and keep them busy around the clock. It took a long time for people to realize that this conventional wisdom was wrong.

The goal is to let your customer take an options-based approach to making decisions, letting them delay their decisions a long as possible so they can make decisions based on the best possible information. Once your customers decide what they want, your goal should be to create that value just as fast as possible. This means no delay in deciding what requests to approve, no delay in staffing, immediate clarification of requirements, no time-consuming handoffs, no delay in testing, no delay for integration, no delay in deployment. In a mature software development organization, all of this happens in one smooth, rapid flow in response to a customer need.

Empower the Team
In a lean organization, things moves fast, so decisions about what to do have to be made by the people doing the work. Flow in a lean organization is based on local signaling and commitment amongst team members, not on management directives. The work team designs its own processes, makes its own commitments, gathers the information needed to reach its goals, and polices itself to meet its milestones.

Wait a minute, you say. Why would I want to be empowered? If I make the decisions, then I’ll get blamed when things go wrong. A team is not empowered unless it has the training, expertise, and leadership necessary to do the job at hand. But once those elements are in place, a working team is far better equipped to make decisions than those who are not involved in the day-to-day activities of developing software. It is true that decision-making carries greater responsibility, but it also brings significantly greater influence on the course of events and the success of the development effort.

Consider an emergency response unit such as firefighters. Members receive extensive training, both in the classroom and on the job. Exercises and experience imprint patterns of how to respond to difficult situations. When an emergency occurs, the team responds to the situation as it unfolds; there is little time to ask remote commanders what to do, nor would it make sense. The important thing is that the responding teams have the training, organization and support to assess the situation as they encounter it and make basically correct decisions on their own.

Similarly in software, the development team is in the best position to know how to respond to difficult problems and urgent requests. The best way to be sure that you get things right is to work directly with customers to understand their needs, collaborate with colleagues to figure out how to meet those needs, and frequently present the results to customers to be sure you are on the right track. Management’s job to supply the organization, training, expertise, leadership, and information so that you generally make the right decisions, make rapid course corrections as you learn, and end up with a successful outcome.

Build Integrity In
There are two kinds of integrity – perceived integrity and conceptual integrity. Software with perceived integrity delights the customer – it’s exactly what they want even though they didn’t know how to ask for it. Google comes to mind – when I use Google I imagine that the designers must have gotten inside my head when they added the spelling checker. I would never have known to ask for this feature, but these days I type most URLs into the Google toolbar because if I mistype, Google will set me straight.

The way to achieve perceived integrity is to have continuous, detailed information flow from the users, or user proxies, to the developers. This is often done by an architect or master designer who understands the user domain in detail and makes sure that the developers always have the real user needs in front of them as they make day-to-day design decisions. Note that the technical leader facilitates information flow and domain understanding, she or he is intimately involved in the day-to-day work of the developers, keeping the customer needs always in mind. However, it is the developers, not the leader, who make the detailed, day-to-day decisions and tradeoffs that shape the system.

Conceptual integrity means that all of the parts of a software system work together to achieve a smooth, well functioning whole. Software with conceptual integrity presents the user with a single metaphor of how a task is to be done. You don’t have one way of buying airline tickets if you are paying with cash and a totally different way if you are using frequent flier miles. You don’t use kilometers in one module and miles in another.

Conceptual integrity is achieved through continuous, detailed information flow between various technical people working on a system. There are no two ways about it, people have to talk to each other, early and often. There can be no throwing things over the wall, no lines between supplier, development team, support team, customer. Everyone should be involved in detailed discussions of the design as it develops, from the earliest days of the program.

For example, Boeing credits its rapid and successful development of the 777 to its ‘Working Together’ program, where customers, designers, suppliers, manufacturers, and support teams all met from the start to design the plane. Early on it was discovered that the fuel tank was beyond the reach of all existing refueling trucks, a mistake that was easily fixed. In a normal development process this expensive mistake would not have been discovered until someone tried to fuel the first plane.

There are those who believe that software integrity comes from a documentation-centric approach to development: define the requirements in detail and trace every bit of code back to those requirements. In fact, such an approach tends to interfere with the technical communication that is essential to integrity. Instead of a documentation-centric approach, use a test-centric approach to integrity. Test early, test often, test exhaustively, and make sure an automated test suite is delivered as part of the product.

See the Whole
When you look at them closely, most theories of how to manage software projects are based on a theory of disaggregation: break the whole into individual parts and optimize each one. Lean thinking suggests that optimizing individual parts almost always leads to sub-optimized overall system.

Optimizing the use of testing resources, for example, decreases the ability of the overall system to rapidly produce tested, working code. Measuring an individual’s ability to produce code without defects ignores the well-known fact that about 80% of defects are caused by the way the system works, and hence are management problems.

The best way to avoid sub-optimization and encourage collaboration is to make people accountable for what they can influence, not just what they can control. This means measuring performance one level higher than you would expect. Measure the team’s defect count, not that of individuals. Make testing as accountable for defect-free code as developers. To some it seems unfair to hold a team accountable for individual performance, but lean organizations have found that individuals are rarely able to change the system which influences their performance. However a team, working together and responsible for its own processes, can and will make dramatic improvements.

Conclusion
To keep customers from changing their minds, raise the maturity of your organization to the level where it can reliably deliver what customers want so fast that they have no time to change their minds. Focus on value, flow, and people, the rest will take care of itself
_________________
References
[1]Harold Thimbleby, "Delaying Commitment" IEEE Software, May 1988.

[2]The book Lean Software Development: An Agile Toolkit by Mary Poppendieck and Tom Poppendieck, Addison-Wesley Professional, 2003, provides twenty two tools for converting lean principles into agile software development practices.

[3]Craig Larman, “A History of Iterative and Incremental Development”, IEEE Computer, June 2003

Published in C++ Magazine, Methodology Issue, Fall 2003

Screen Beans Art, © A Bit Better Corporation