Five World-Changing Software Innovations

On the 15th anniversary of the Agile Manifesto, let's look at what else was happening while we were focused on spreading the Manifesto’s ideals. There have been some impressive advances in software technology since Y2K:
          1.   The Cloud
          2.   Big Data
          3.   Antifragile Systems
          4.   Content Platforms
          5.   Mobile Apps 

The Cloud

In 2003 Nicholas Carr’s controversial article “IT Doesn’t Matter” was published in Harvard Business Review. He claimed that “the core functions of IT– data storage, data processing, and data transport” had become commodities, just like electricity, and they no longer provided differentiation. It’s amazing how right – and how wrong – that article turned out to be. At the time, perhaps 70% of an IT budget was allocated to infrastructure, and that infrastructure rarely offered a competitive advantage. On the other hand, since there was nowhere to purchase IT infrastructure as if it were electricity, there was a huge competitive advantage awaiting the company that figured out how package and sell such infrastructure. 

At the time, IT infrastructure was a big problem – especially for rapidly growing companies like Amazon.com. Amazon had started out with the standard enterprise architecture: a big front end coupled to a big back end. But the company was growing much faster than this architecture could support. CEO Jeff Bezos believed that the only way to scale to the level he had in mind was to create small autonomous teams. Thus by 2003, Amazon had restructured its digital organization into small (two-pizza) teams, each with end-to-end responsibility for a service. Individual teams were responsible for their own data, code, infrastructure, reliability, and customer satisfaction.

Amazon’s infrastructure was not set up to deal with the constant demands of multiple small teams, so things got chaotic for the operations department. This led Chris Pinkham, head of Amazon’s global infrastructure, to propose developing a capability that would let teams manage their own infrastructure – a capability that might eventually be sold to outside companies. As the proposal was being considered, Pinkham decided to return to South Africa where he had gone to school, so in 2004 Amazon gave him the funding to hire a team in South Africa and work on his idea. By 2006 the team’s product, Elastic Compute Cloud (EC2), was ready for release. It formed the kernel of what would become Amazon Web Services (AWS), which has since grown into a multi-billion-dollar business.

Amazon has consistently added software services on top of the hardware infrastructure – services like databases, analytics, access control, content delivery, containers, data streaming, and many others. It’s sort of like an IT department in a box, where almost everything you might need is readily available. Of course Amazon isn’t the only cloud company – it has several competitors.

So back to Carr’s article – Does IT matter?  Clearly the portion of a company’s IT that could be provided by AWS or similar cloud services does not provide differentiation, so from a competitive perspective, it doesn’t matter. If a company can’t provide infrastructure that matches the capability, cost, accessibility, reliability, and scalability of the cloud, then it may as well outsource its infrastructure to the cloud.

Outsourcing used to be considered a good cost reduction strategy, but often there was no clear distinction between undifferentiated context (that didn’t matter) and core competencies (that did). So companies frequently outsourced the wrong things – critical capabilities that nurtured innovation and provided competitive advantage. Today it is easier to tell the difference between core and context: if a cloud service provides it then anybody can buy it, so it’s probably context; what’s left is all that's available to provide differentiation. In fact, one reason why “outsourcing” as we once knew it has fallen into disfavor is that today, much of the outsourcing is handled by cloud providers. 

The idea that infrastructure is context and the rest is core helps explain why internet companies do not have IT departments. For the last two decades, technology startups have chosen to divide their businesses along core and infrastructure lines rather than along technology lines. They put differentiating capabilities in the line business units rather than relegating them to cost centers, which generally works a lot better. In fact, many IT organizations might work better if they were split into two sections, one (infrastructure) treated as a commodity and the rest moved into (or changed into) a line organization. 

Big Data

In 2001 Doug Cutting released Lucene, a text indexing and search program, under the Apache software license. Cutting and Mike Cafarella then wrote a web crawler called Nutch to collect interesting data for Lucerne to index. But now they had a problem – the web crawler could index 100 million pages before it filled up the terabyte of data they could easily fit on one machine. At the time, managing large amounts of data across multiple machines was not a solved problem; most large enterprises stored their critical data in a single database running on a very large computer. 

But the web was growing exponentially, and when companies like Google and Yahoo set out to collect all of the information available on the web, currently available computers and databases were not even close to big enough to store and analyze all of that data. So they had to solve the problem of using multiple machines for data storage and analysis. 

One of the bigger problems with using multiple machines is the increased probability that one of machines will fail. Early in its history, Google decided to accept the fact that at its scale, hardware failure was inevitable, so it should be managed rather than avoided. This was accomplished by software which monitored each computer and disk drive in a data center, detected failure, kicked the failed component out of the system, and replaced it with a new component. This process required keeping multiple copies of all data, so when hardware failed the data it held was available in another location. Since recovering from a big failure carried more risk than recovering from a small failure, the data centers were stocked with inexpensive PC components that would experience many small failures. The software needed to detect and quickly recover from these “normal” hardware failures was perfected as the company grew. 

In 2003 Google employees published two seminal papers describing how the company dealt with the massive amounts of data it collected and managed. Web Search for a Planet: The Google Cluster Architecture by Luiz André Barroso, Jeffrey Dean, and Urs Hölzle described how Google managed it’s data centers with their inexpensive components. The Google File System by Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung described how the data was managed by dividing it into small chunks and maintaining multiple copies (typically three) of each chunk across the hardware. I remember that my reaction to these papers was “So that’s how they do it!” And I admired Google for sharing these sophisticated technical insights. 

Cutting and Cafarella had approximately the same reaction. Using the Google File System as a model, they spent 2004 working on a distributed file system for Nutch. The system abstracted a cluster of storage into a single file system running on commodity hardware, used relaxed consistency, and hid the complexity of load balancing and failure recovery from users. 

In fall, 2004, the next piece of the puzzle – analyzing massive amounts of stored data – was addressed by another Google paper: MapReduce: Simplified Data Processing on Large Clusters by Jeffrey Dean and Sanjay Ghemawat. Cutting and Cafarella spent 2005 rewriting Nutch and adding MapReduce, which they released as Apache Hadoop in 2006. At the same time, Yahoo decided it needed to develop something like MapReduce, and settled on hiring Cutting and building Apache Hadoop into software that could handle its massive scale. Over the next couple of years, Yahoo devoted a lot of effort to converting Apache Hadoop – open source software – from a system that could handle a few servers to a system capable of dealing with web-scale databases. In the process, their data scientists and business people discovered that Hadoop was as useful for business analysis as it was for web search. 

By 2008, most web scale companies in Silicon Valley – Twitter, Facebook, LinkedIn, etc. – were using Apache Hadoop and contributing their improvements. Then startups like Cloudera were founded to help enterprises use Hadoop to analyze their data. What made Hadoop so attractive? Until that time, useful data had to be structured in a relational database and stored on one computer. Space was limited, so you only kept the current value of any data element. Hadoop could take unlimited quantities of unstructured data stored on multiple servers and make it available for data scientists and software programs to analyze. It was like moving from a small village to a megalopolis – Hadoop opened up a vast array of possibilities that are just beginning to be explored.

In 2011 Yahoo found that its Hadoop engineers were being courted by the emerging Big Data companies, so it spun off Hortonworks to give the Hadoop engineering team their own Big Data startup to grow. By 2012, Apache Hadoop (still open source) had so many data processing appendages built on top of the core software that MapReduce was split off from the underlying distributed file system. The cluster resource management that used to be in MapReduce was replaced by YARN (Yet Another Resource Negotiator). This gave Apache Hadoop another growth spurt, as MapReduce joined a growing number of analytical capabilities that run on top of YARN. Apache Spark is one of those analytical layers which supports data analysis tools that are more sophisticated and easier to use than MapReduce. Machine learning and analytics on data streams are just two of the many capabilities that Spark offers – and there are certainly more Hadoop tools to come. The potential of Big Data is just beginning to be tapped. 

In the early 1990’s Tim Burners Lee worked to ensure that CERN made his underlying code for HTML, HTTP and URL’s available on a royalty free basis, and because of that we have the world wide web. Ever since, software engineers have understood that the most influential technical advances come from sharing ideas across organizations, allowing the best minds in the industry to come together and solve tough technical problems. Big Data is as capable as it is because Google and Yahoo and many others companies were willing to share their technical breakthroughs rather than keep them proprietary. In the software industry we understand that we do far better as individual companies when the industry as a whole experiences major technical advances. 

Antifragile Systems

It used to be considered unavoidable that as software systems grew in age and complexity, they became increasingly fragile. Every new release was accompanied by fear of unintended consequences, which triggered extensive testing and longer periods between releases. However, the “failure is not an option” approach is not viable at internet scale – because things will go wrong in any very large system. Ignoring the possibility of failure – and focusing on trying to prevent it – simply makes the system fragile. When the inevitable failure occurs, a fragile system is likely to break down catastrophically.[1]  

Rather than prevent failure, it is much more important to identify and contain failure, then recover with a minimum of inconvenience for consumers. Every large internet company has figured this out. Amazon, Google, Esty, Facebook, Netflix and many others have written or spoken about their approach to failure. Each of these companies has devoted a lot of effort to creating robust systems that can deal gracefully with unexpected and unpredictable situations.

Perhaps the most striking among these is Netflix, which has a good number of reliability engineers despite the fact that it has no data centers. Netflix’s approach was described in 2013 by Ariel Tseitlin in the article The Antifragile Organization: Embracing Failure to Improve Resilience and Maximize Availability.  The main way Netflix increases the resilience of its systems is by regularly inducing failure with a “Simian Army” of monkeys: Chaos Monkey does some damage twice an hour, Latency Monkey simulates instances that are sick but still working, Conformity Monkey shuts down instances that don’t adhere to best practices, Security Monkey looks for security holes, Janitor Monkey cleans up clutter, Chaos Gorilla simulates failure of an AWS availability zone and Chaos Kong might take a whole Amazon region off line. I was not surprised to hear that during a recent failure of an Amazon region, Netflix customers experienced very little disruption.

A Simian Army isn’t the only way to induce failure. Facebook’s motto “Move Fast and Break Things” is another approach to stressing a system. In 2015, Ben Maurer of Facebook published Fail at Scale – a good summary of how internet companies keep very large systems reliable despite failure induced by constant change, traffic surges, and hardware failures. 

Maurer notes that the primary goal for very large systems is not to prevent failure – this is both impossible and dangerous. The objective is to find the pathologies that amplify failure and keep them from occurring. Facebook has identified three failure-amplifying pathologies: 

1. Rapidly deployed configuration changes
Human error is amplified by rapid changes, but rather than decrease the number of deployments, companies with antifragile systems move small changes through a release pipeline. Here changes are checked for known errors and run in a limited environment. The system quickly reverts to a known good configuration if (when) problems are found. Because the changes are small and gradually introduced into the overall system under constant surveillance, catastrophic failures are unlikely. In fact, the pipeline increases the robustness of the system over time.

2. Hard dependencies on core services
Core services fail just like anything else, so code has to be written with that in mind. Generally hardened API’s that include best practices are used to invoke these services. Core services and their API’s are gradually improved by intentionally injecting failure into a core service to expose weaknesses that are then corrected as failure modes are identified.

3. Increased latency and resource exhaustion
Best practices for avoiding the well-known problem of resource exhaustion include managing server queues wisely and having clients track outstanding requests. It’s not that these strategies are unknown, it’s that they must become common practice for all software engineers in the organization. 

Well-designed dashboards, effective incident response, and after-action reviews that implement countermeasures to prevent re-occurrence round out Facebook's toolkit for keeping its very large systems reliable.

We now know that fault tolerant systems are not only more robust, but also less risky than systems which we attempt to make failure-free. Therefore, common practice for assuring the reliability of large-scale software systems is moving toward software-managed release pipelines which orchestrate frequent small releases, in conjunction with failure induction and incident analysis to produce hardened infrastructure.

Content Platforms

Video is not new; television has been around for a long time, film for even longer. As revolutionary as film and TV have been, they push content to a mass audience; they do not inspire engagement. An early attempt at visual engagement was the PicturePhone of the 1970’s – a textbook example of a technical success and a commercial disaster. They got the PicturePhone use case wrong – not many people really wanted to be seen during a phone call. Videoconferencing did not fare much better – because few people understood that video is not about improving communication, it’s about sharing experience. 

In 2005, amidst a perfect storm of increasing bandwidth, decreasing cost of storage, and emerging video standards, three entrepreneurs – Chad Hurley, Steve Chen, and Jawed Karim – tried out an interesting use case for video: a dating site. But they couldn’t get anyone to submit “dating videos,” so they accepted any videos clips people wanted to upload. They were surprised at the videos they got: interesting experiences, impressive skills, how-to lessons – not what they expected, but at least it was something. The YouTube founders quickly added a search capability. This time they got the use case right and the rest is history. Video is the printing press of experience, and YouTube became the distributor of experience. Today, if you want to learn the latest unicycle tricks or how to get the back seat out of your car, you can find it on YouTube. 

YouTube was not the first successful content platform. Blogs date back to the late 1990’s where they began as diaries on personal web sites shared with friends and family. Then media companies began posting breaking news on their web sites to get their stories out before their competitors. Blogger, one of the earliest blog platforms, was launched just before Y2K and acquired by Google in 2003 – the same year WordPress was launched. As blogging popularity grew over the next few years, the use case shifted from diaries and news articles to ideas and opinions – and blogs increasingly resembled magazine articles. Those short diary entries meant for friends were more like scrapbooks; they came to be called tumbleblogs or microblogs. And – no surprise – separate platforms for these microblogs emerged: Tumblr in 2006 and Twitter in 2007.

One reason why blogs drifted away from diaries and scrapbooks is that alternative platforms emerged aimed at a very similar use case – which came to be called social networking. MySpace was launched in 2003 and became wildly popular over the next few years, only to be overtaken by Facebook, which was launched in 2004. 

Many other public content platforms have come (and gone) over the last decade; after all, a successful platform can usually be turned into a significant revenue stream. But the lessons learned by the founders of those early content platforms remain best practices for two-sided platforms today:

1. Get the use case right on both sides of the platform. Very few founders got both use cases exactly right to begin with, but the successful ones learned fast and adapted quickly. 
2. Attract a critical mass to both sides of the platform. Attracting enough traffic to generate network effects requires a dead simple contributor experience and an addictive consumer experience, plus a receptive audience for the initial release.
3. Take responsibility for content even if you don’t own it. In 2007 YouTube developed ContentID to identify copyrighted audio clips embedded in videos and make it easy for contributors to comply with attribution and licensing requirements. 
4. Be prepared for and deal effectively with stress. Some of the best antifragile patterns came from platform providers coping with extreme stress such as the massive traffic spikes at Twitter during natural disasters or hectic political events.

In short, successful platforms require insight, flexibility, discipline, and a lot of luck. Of course, this is the formula for most innovation. But don't forget – no matter how good your process is, you still need the luck part. 

Mobile Apps

It’s hard to imagine what life was like without mobile apps, but they did not exist a mere eight years ago. In 2008 both Apple and Google released content platforms that allowed developers to get apps directly into the hands of smart phone owners with very little investment and few intermediaries. By 2014 (give or take a year, depending on whose data you look at) mobile apps had surpassed desktops as the path people take to the internet. It is impossible to ignore the importance of the platforms that make mobile apps possible, or the importance of the paradigm shift those apps have brought about in software engineering. 

Mobile apps tend to be small and focused on doing one thing well – after all, a consumer has to quickly understand what the app does. By and large, mobile apps do not communicate with each other, and when they do it is through a disciplined exchange mediated by the platform. Their relatively small size and isolation make it natural for each individual app to be owned by a single, relatively small team that accepts the responsibility for its success. As we saw earlier, Amazon moved to small autonomous teams a long time ago, but it took a significant architectural shift for those teams to be effective. Mobile apps provide a critical architectural shift that makes small independent teams practical, even in monolithic organizations. And they provide an ecosystem that allows small startups to compete effectively with those organizations.  

The nature of mobile apps changes the software development paradigm in other ways as well. As one bank manager told me, “We did our first mobile app as a project, so we thought that when the app was released, it was done. But every time there was an operating system update, we had to update the app. That was a surprise! There are so many phones to test and new features coming out that our apps are in a constant state of development. There is no such thing as maintenance – or maybe it's all maintenance.”

The small teams, constant updates, and direct access to the deployed app have created a new dynamic in the IT world: software engineers have an immediate connection with the results of their work. App teams can track usage, observe failures and track metrics – then make changes accordingly. More than any other technology, mobile platforms have fostered the growth of small, independent product teams – with end-to-end responsibility – that use short feedback loops to constantly improve their offering. 

Let’s return to luck. If you have a large innovation effort, it probably has a 20% chance of success at best. If you have five small, separate innovation efforts, each with 20% chance of success, you have a much better chance that one of them will succeed – as long as they are truly autonomous and are not tied to an inflexible back end or flawed use case. Mobile apps create an environment where it can be both practical and advisable to break products into small, independent experiments, each owned by its own “full stack” team.[2] The more of these teams you have pursuing interesting ideas, the more likely you are that some of the ideas will become the innovative offerings that propel your company into the future. 

What about “Agile”?

You might notice that “Agile” is not on my list of innovations. And yet, agile values are found in every major software innovation since the Agile Manifesto was articulated in 2001. Agile development does not cause innovation; it is meant to create the conditions necessary for innovation: flexibility and discipline, customer understanding and rapid feedback, small teams with end-to-end responsibility. Agile processes do not manufacture insight and they do not create luck. That is what people do.  
____________________________
Footnotes:
1.    “the problem with artificially suppressed volatility is not just that the system tends to become extremely fragile; it is that, at the same time, it exhibits no visible risks… Such environments eventually experience massive blowups… catching everyone off guard and undoing years of stability or, in almost all cases, ending up far worse than they were in their initial volatile state. Indeed, the longer it takes for the blowup to occur, the worse the resulting harm…”  Antifragile, Nassim Taleb p 106

2.   A full stack team contains all the people necessary to make things happen in not only the full technology stack, but also in the full stack of business capabilities necessary for the team to be successful.

Train-Wreck Management

“On October 5, 1841, two Western Railroad passenger trains collided somewhere between Worchester, Massachusetts and Albany, New York, killing a conductor and a passenger and injuring seventeen passengers. That disaster marked the beginning of a new management era." [1] These words open Peter Scholtes classic book on leadership. He goes on to explain how the term "management" was unknown in the days of cottage industries. As business grew and became geographically disperse in the 1800's, a way to run these businesses had to be found. But there were no models outside the church and the military, so investigators into the train-wreck disaster looked to the Prussian army for a model. And there they found the classic organization chart - the one we know so well today. Scholtes calls it the "train-wreck" chart. It was revolutionary at the time.

The purpose of what became today's organization chart was clear: The assignment of responsibility would enable "prompt detection of derelictions of duty... and point out the delinquent." Scholtes says: "A fundamental premise of the 'train-wreck' approach to management is that the primary cause of problems is 'dereliction of duty'. The purpose of the organizational chart is to sufficiently specify those duties so that management can quickly assign blame, should another accident occur."[1].

Blame
Note the thinking here: Problems are caused by people who don't do their job well, so finding someone to blame is the first step to correcting problems. Scholtes notes: "The era of management that began in the mid-1800's can be characterized as "management by results.".... Since managers could no longer do the work themselves or direct others in the doing of the work, managers exercised their authority by holding people accountable for results.... In the 1950's, management by results reached its epitome in MBO (Management By Objectives) and performance appraisals, the Harvardization of train-wreck management."[1] He goes on to say that at the time, this theory of management was the best available, and it succeeded in creating order out of chaos. "People like Whistler, McCallum, Frederick Taylor or Henry Ford in the United States or Darby, the Stephensons, or Brunel in England were pioneers.... they did their best and, by and large, what they did was very good."

"Meanwhile, in Japan...." is the title of the next section Scholtes' book. He chronicles how a better approach to management emerged in Japan in the 1950's assisted by W. Edwards Deming. Deming taught that most of the problems we encounter (perhaps 90%) are the result of multiple influences, they generally cannot be attributed to a single cause. Assigning blame for a problem to the last person involved is worse than counterproductive, it will probably make the bad situation worse. Exhorting people to "be careful," "try harder," and "work smarter" is not useful if individuals have little effect on results. Rewarding or punishing people for outcomes that are not under their control can only result in discouragement - or in gaming the system. Instead, chronic problems must be fixed by finding their underlying causes and addressing these effectively. As Deming points out, this usually involves changing the system - the way things are done. And according to Deming, it is management's job to change the system.

Process or People?
Agile software development places a strong emphasis on putting change into the hands of front-line people on self-directed teams - isn't this contrary to Deming's philosophy? Writing in 1995, Scholtes lists what he calls "fads" for addressing systemic problems: "empower people, put them into self-directed teams, motivate them, offer incentives, reengineer and reinvent them." And then he says: "All of the empowered, motivated, teamed-up, self-directed, incentivized, accountable, reengineered, and reinvented people you can muster cannot compensate for a dysfunctional system.... A well-run organization with well-functioning systems allows people from top to bottom do work of which they can be proud."[1] So where does this leave us? Which is more important - process or people?

It helps if we trade in the overloaded word "process" and use "system."

In the article "Managing a Living System, not a Ledger"[2] H. Thomas Johnson says "Managers at Toyota believe that improving the system is the surest way to improve long term financial results." He points out that Toyota takes lots and lots of measurements, but they do not use these as performance measurements. Johnson writes: "...Toyota makes virtually no use of management accounting targets (or 'levers') to control or motivate operations... Toyota focuses its operations on continuous system improvement through endless rapid problem solving. And they emphasize genchi genbutsu, or 'going to the place,' to see where a problem occurs, firsthand. They don't rely on second-hand reports or tables and charts of data to achieve a true understanding of root cause. Instead they go to the place (gemba) where you can watch, observe, and 'ask why five times.' This attitude shows a deep appreciation that results (and problems) ultimately emanate from, and are explained by, complex processes and concrete relationships, not by abstract, quantitative relationships that describe results in simple, linear, additive terms." Winding up the article, Johnson says: "Financial quantities cannot reveal if a system is improving or not... No company that talks about improving performance can know what it is doing if its primary window on results is financial information and not system principles.... Companies that intend to perform like Toyota should recognize that... they will never get there by trying to motivate and direct 'lean' initiatives with 'lean accounting' and management accounting 'levers of control.'"

Taiichi Ohno on Standard Work
Let's go back to the source of the Toyota Production System, Taiichi Ohno, and see what he had to say about process - how it is established and how it is changed.[3]

There is something called standard work, but standards should be changed constantly. Instead, if you think of the standard as the best you can do, it's all over. The standard work is only a baseline for doing further kaizen. It is kai-aku [change for the worse] if things get worse than now, and it is kaizen [change for the better] if things get better than now. Standards are set arbitrarily by humans, so how can they not change?

When creating Standard Work, it will be difficult to establish a standard if you are trying to achieve 'the best way.' This is a big mistake. Document exactly what you are doing now. If you make it better than it is now, it is kaizen. If not, and you establish the best possible way, the motivation for kaizen will be gone. That is why one way of motivating people to do kaizen is to create a poor standard. But don't make it too bad. Without some standard, you can't say 'We made it better' because there is nothing to compare it to, so you must create a standard for comparison.

Take that standard, and if the work is not easy to perform, give many suggestions and do kaizen.

We need to use the words 'you made' as in 'follow the decisions you made.' When we say 'they were made' people feel like it was forced upon them. When a decision is made, we need to ask who made the decision. Since you also have the authority to decide, if you decide, you must at least follow your decision, and then this will not be forced upon you at all.

But in the beginning, you must perform the Standard Work, and as you do, you should find things you don't like, and you will think of one kaizen idea after another. Then you should implement these ideas right away, and make this the new standard.

Years ago, I made them hang the standard work documents on the shop floor. After a year I said to a team leader, 'The color of the paper has changed, which means you have been doing it the same way, so you have been a salary thief for the last year.' I said 'What do you come to work to do each day? If you are observing every day you ought to be finding things you don't like, and rewriting the standard immediately. Even if the document hanging there is from last month, this is wrong.' At Toyota in the beginning we had the team leaders write down the dates on the standard work sheets when they hung them. This gave me a good reason to scold the team leaders, saying 'Have you been goofing off all month?'

If it takes one or two months to create these documents, this is nonsense. You should not create these away from the job. See what is happening on the gemba and write it down.

Process AND People
Ohno believed that the primary job of team leaders (first line supervisors) is the constant improvement of the way work gets done. Work standards should be written and posted, but this had better not take very long because the standards should change all the time - at least once a month. Standards are not about how work should be done, but how work is being done. You don't want the standard to be too perfect, because that leaves no incentive for workers to improve their standards. If workers are annoyed by a standard, they are expected to change it. They do not drop a suggestion in a suggestion box, they do kaizen. That is, workers - led by their team leader - do many rapid experiments, find a better way, agree on the improvement, quickly document the new way, and use it. When a standard is improved, the decision for the change must be made by the people doing the work, so they won't feel it is being forced upon them.

People like to use effective processes, and they also like to have control over their own environment. The Toyota Production System provides for both. Ohno made it clear that people must be at the center of improving their own processes. Process improvement may be done only "at the gemba" and it is up to the workers to decide whether or not a proposed improvement should be implemented. Workers are expected to keep changing the way they do their job; in fact, it is bad leadership to have a process so perfect that workers have little incentive to improve it!

Assessment and Certification
Scholtes takes process improvement assessment programs such as ISO 9000 to task because even though they seem good on the surface, they have some problems:[1]

1. The pursuit of quality must be guided by a larger context than certification - it requires a holistic, integrated, long term commitment.
2. Certification is not equal to satisfied customers - you can do the wrong thing as long as you do it consistently.
3. Assessment has a tone of paternalism and mistrust - it replaces internal motivation with external motivation.
4. Assessment assumes that inspectors are all the same - but inspections are not standardized.
5. A certified process is difficult to change - Ohno would be appalled.

Conclusion
When Deming said "change the system", he was talking about changing the complex, interrelated processes used to get work done. Deming believed that changing the system is management's primary job, and in order to do this, managers need competency in four areas:

1. Appreciation for the overall system in which work is done
2. An understanding of variation - and the true relationship between cause and effect
3. Constant pursuit of learning (improvement) through designed experiments
4. An understanding of the psychology of people

When all of these areas are balanced and working together, great things can happen.

References
[1] The Leader's Handbook, by Peter R. Scholtes, McGraw-Hill, 1998.

[2] "Managing a Living System, Not a Ledger,", by H. Thomas Johnson, Lean Manufacturing 2007, Supplement to Manufacturing Engineering, SME, August 2007. Johnson also coauthored Profit Beyond Measure: Extraordinary Results through Attention to Work and People, Free Press, November, 2000.

[3] Workplace Management, by Taiichi Ohno, originally published in 1982, from translation by Jon Miller, Gemba Press, 2007.

Screen Beans Art, © A Bit Better Corporation

Cause and Effect

Is low cost achieved by focusing on cutting costs?
Is high utilization achieved by trying to utilize resources full time?
Does standardized work mean that work processes are followed without challenge?
Does everyone in your organization agree on the answers to these questions?

Cost
When Jane Beseda took over Toyota’s North American Parts Operation (NAPO), she knew that really dramatic results required breaking down the barriers between departments [1].  So she set three year Stretch Goals that were all but impossible – a 50% reduction in inventory, 25% increase in throughput, 25% reduction in freight costs, 50% reduction in packaging expense, 25% increase in space utilization, 50% decrease in backorders.  After a year of effort, department managers began to realize that that they were not going to achieve the Stretch Goal targets unless they changed their focus to cross-organizational projects.  Project teams had been struggling to coordinate their work through the traditional functional department planning approach. This was changed; the departments started to consider potential impact of their plans on other functions and areas of the broader organization. Only then were truly significant gains in eliminating waste and reducing cost realized. After three years, the results at NAPO were truly amazing; almost all of the Stretch Goals were achieved. But this would never have happened if everyone in the organization had not focused on overall system waste rather than individual department costs.

Low overall costs rarely come from lowering costs in individual departments; they come from lowering system costs. Lean companies have learned that this requires a keen understanding of underlying cost drivers and a system-wide effort to eliminate waste.  Consider Zara, a huge women’s fashion clothing chain headquartered in western Spain. It fills retail store orders twice a week, shipping garments around the world in two days – not folded compactly in boxes – but ironed and hanging on hangers ready to display. Zara realizes that it lowers overall costs by getting the clothing that women are asking for on shelves very rapidly.  Zara has fewer markdowns and unsold clothes, and it drives more business to stores with less advertising than its competitors. These tremendous savings would disappear if Zara focused on reducing shipping costs. This is systems thinking at work.

Over time, individual departments will find ways to drive out costs, but rarely do organizations attack the larger costs that occur between departments, divisions, functions, and companies.  A quick and easy cost reduction approach for a department might be to outsource work to obtain lower labor costs.  However, in most cases, outsourcing simply adds more boundaries to cross, and boundaries typically add 20 to 30% to overall costs [2]. One company I know outsourced the process of engaging contractors.  Since training is included in this outsourcing arrangement, all trainers must work through the outsourced contractor management company.  The amount of time and paperwork necessary to set up a training engagement at this company is about an order of magnitude greater than at any other company I have dealt with. However, the arrangement appears to be a cost reduction for the company, because contractors pay the contractor management firm through a fee imposed on their payment. This financial arrangement hides the tremendous waste in the engagement process, which includes a great deal of problem resolution by employees in the company receiving the training. But because of organizational barriers, these same employees have no way to change or improve a very burdensome process.

Standardized Work
In studying Toyota, many companies notice that standardized work processes are a key element of the company’s success.  What they generally miss is that at Toyota, standards exist to be challenged and changed by the front line employees doing the work.  Toyota actively encourages all employees to question any part of their job that is annoying or gets in the way of doing a good job. Employees are expected to look for a better way, prove with experiments that the new way is better, and then implement a new standard.  Taiichi Ohno, father of the Toyota Production System, wrote: “Something is wrong if workers do not look around each day, find things that are tedious or boring, and then rewrite the procedures. Even last month’s manual should be out of date.”

There are those who believe that standardized processes should be imposed from the outside and followed without question. However, companies with a Lean perspective believe that the essence of eliminating waste is to encourage everyone to attack and change the things that annoy them about their jobs. Only by engaging people in doing their job better on a daily basis can you get the sustainable gains that companies like Toyota enjoy. Work standards in a lean company spell out the current best known way of doing things, and they are a baseline for change. New standards are developed through constant experimentation by work teams, using the current work standards as the baseline against which improvement is measured.

At Toyota’s Georgetown, Kentucky plant the paint department has always been the biggest bottleneck preventing one-piece-flow of vehicles through the plant.  Over the past few years, however, things began to change [3].  First the people in the department devised a way to paint any color in any order – instead of piping paint to a robot through a tube that needs cleaning between colors, they now pipe each paint color to a canister just the right size to paint a car.  The robot picks up the right canister and Viola! the cost and time needed to clean out paint lines is eliminated.  The bottom line:  30% less paint used, a huge drop in the use of cleaning solvents, and cars can now be painted any color in any order. The throughput of the shop has increased from 33 to 50 an hour while the space required for painting was reduced by a third and employees were freed up for other work.

At most auto companies, this achievement would be cause for celebration – but at Toyota, constant improvement is the day-in-day-out job of the paint manager. There are no black belts, no outside process experts – just the people doing the work led by their first line managers. Everyone’s job is to keep on improving their work processes, every day, every week, every month. All workers are engaged in the relentless pursuit of perfection – the elusive goal which keeps everyone looking forward rather than patting themselves on the back.

In software development, we have been led to believe that “maturity” involves documenting best practices and making sure that they are followed. On the contrary, a truly mature organization expects development teams to constantly challenge and improve their processes.  The day an organization settles into believing that it is perfect is the day it invites its people to stop thinking. Real, sustainable improvement on all fronts comes only when all workers are expected to challenge and fix anything about their job that annoys them or keeps them from taking pride in their work.

Utilization
Managers in manufacturing plants, supervisors of computer operations, and highway engineers agree on one cause and effect relationship:  if your resources approach full utilization, response time slows to a crawl. Queuing theory is widely applied to servers and highways and manufacturing equipment.  However, in a development environment, managers seem to be unaware that attempting to achieve full utilization slows response time to a crawl and decreases actual utilization. Development organizations are no more immune from the laws of queuing theory than airlines. We see how constantly full airlines with no slack cause cascading problems whenever a slight perturbation is introduced into the airline system. And weather being what it is, perturbations happen all the time.

Similarly, when a slight perturbation is introduced into a fully scheduled development organization, cascading problems are inevitable. Knowledge work being what it is, there will always be perturbations to the most carefully laid out schedules. The only way to deal effectively with these perturbations is to follow the advice of queuing theory:  work in small batches, minimize the length of queues of work to be done, and never, ever schedule an organization beyond its capacity to deliver.  Do this and utilization will increase.  Focus directly on utilization and it is guaranteed to be sub-optimal.

Many organizations queue up requests for software development into large batches of work to be done, presumably because better decisions can be made if the whole picture is visible before work begins.  Challenge this assumption.  Why is it better to respond very slowly to a pile of accumulated requests rather than respond very quickly to the current most important outstanding request? What good, really, does it do to have a long list of work to do?  Looked at closely, it's hard to make a case for piling up lists of work-to-do that are far longer than you have a hope of accomplishing. You're pretty much wasting your time keeping track of stuff you can never get around to, and you're probably setting incorrect expectations on the part of the requestors. Just say no.

Unfortunately, once software has been around for a while, we often see that it takes longer and longer to run the regression tests as each new feature is added, since every feature increases the regression test load.  We call this increasing regression load the regression deficit, and unless it is systematically tackled and reduced, the code base will become increasingly difficult to change.  You will be tempted to release software in larger and larger batches, because regression testing takes so long. Similar to the paint shop manager, a software development manager’s job should be to help development teams chip away at the regression deficit every day, every week, every month, until throughput is increased and one-piece-flow of small feature sets becomes practical.

Cause & Effect
In order for organizations to perform brilliantly, there are two prerequisites:  First, everyone has to agree on what they want, and second everyone has to agree on cause and effect [4].  Let’s assume that everyone in your organization agrees on the results they want, their values and priorities, and the trade-offs they are willing to make in order to achieve those results.  The question to ask is – does everyone agree on cause and effect?

Is there general agreement on what actions will result in system-wide cost reduction?  Are individual departments expected to reduce costs independently, or is it clear that departments must work together to reduce overall system costs, even at the expense of costs in individual departments? Are the measurements in place to reduce the costs of crossing boundaries?

Is there general agreement on the best way to achieve consistent results through standardization?  Are standards always followed?  If not, do you really understand the root cause – are the standards irrelevant or inaccurate, too complex, or too far from the actual work being done? Are standards maintained by a central process group, or are they continually improved by the people doing the work?  Are front line people expected and encouraged to continually challenge and change standards?

Do you have a project scheduling system aimed at optimizing utilization?  Does the management team believe that the system is in fact optimizing utilization? Do people believe that full utilization is the right measurement to emphasize? Is time-to-market an important factor in your business? Does everyone agree on the relationship between time-to-market and utilization?

Lean thinking is counterintuitive; it flies in the face of conventional wisdom concerning cause and effect. No matter how well proven the results, no matter how intellectually sound the arguments of systems thinking are, it is almost impossible for those who have been successful with conventional wisdom to change their habits. It is very difficult for leaders to address the system costs that flourish between department boundaries if they believe that lowering costs in each department will add up to lower overall costs. When leaders believe that standardized processes are an end rather than a beginning, they have a difficult time leveraging the wisdom of front line employees. For those leaders who believe that focusing on full utilization will increase productivity, the outstanding productivity gains that come from focusing instead on throughput will not be available.  It all comes down to measures – small measures cause small results; global measures promote globally optimized results.

Those who have seen the dramatic results of Lean Thinking have come to believe that concentrating on throughput, making front line employees the center of focus of the company, and eliminating waste across the system are the tools of choice in a competitive environment.  Most management teams that have been threatened by fierce competition and survived have changed their habits and adopted the counterintuitive tenets of Lean Thinking. Virtually all companies facing fierce competition from Lean companies that have failed to adopt similar thinking have failed to thrive in the long run.
_________________
Footnotes:

[1] This story is from The Elegant Solution: Toyota’s Formula for Mastering Innovation, by Matthew E. May, Free Press, 2007, p. 139.

[2] Management Challenges for the 21st Century, by Peter Drucker, HarperBusiness, 2001, p. 33.

[3] From “No Satisfaction” by Charles Fishman, Fast Company, Dec 2006 / Jan 2007

[4] See “The Tools of Cooperation and Change,” by Clayton Christensen and co authors, Harvard Business Review, October 2006.


Screen Beans Art, © A Bit Better Corporation

Lazy Workers

Scientific Management
The workers on the first Ford assembly line spoke more than 50 languages, and many of them could barely speak English.[1] It was in this context that Frederick W. Taylor’s book, The Principles of Scientific Management, was published in 1911.[2]


Taylor believed that laborers were uneducated and lazy, reflecting the prevailing thinking of the time. To increase productivity, he proposed the ‘science’ of decomposing tasks into their smallest components, timing and planning each micro-task, and telling the worker exactly how to do each task. Taylor admitted that his methods were inappropriate for educated craftsmen or even intelligent laborers.[3]

Scientific Management was the beginning of the separation of planning from execution. Prior to Ford’s assembly line, automobiles were assembled and maintained by skilled craftsmen. Ford first developed interchangeable parts, then developed a method to have them assembled by interchangeable laborers. After all, Ford’s turnover was 380% in 1913, so it was necessary to give laborers a job they could learn in just a few minutes.

The NUMMI Experiment
In 1982, GM closed it’s Fremont plant, which had the worst productivity and absenteeism record in the company. In 1983, a Toyota and GM re-opened the same plant and hired back the same workers. New United Motor Manufacturing, Inc. (NUMMI) was managed by Toyota-trained management. They borrowed widely from Frederick Taylor in areas of work measurement, but with one major difference. Instead of industrial engineers, small work teams were formed and trained in work measurement and analysis methods. Workers designed their own jobs, and continually worked to improve their own performance. In two years, the same facility with the same workers was operating at twice the productivity and quality, better of any other GM plant. Absenteeism and drug abuse on the job had virtually disappeared, and the plant was being expanded.[4]

The Problem: Separation of Planning from Execution
Lean Production was the end of the separation of planning from execution. The fundamental change at the NUMMI plant was the involvement of the workers in the design and improvement of their own work. ‘Holding back knowledge and effort (has been) repeatedly noted by industrial sociologists as a salient feature of all mass-production systems.’[5] The critical difference in Lean Production was the direct engagement of the workers in improving the process.

Lean Production does not require extraordinary people and is certainly not without discipline. It is build on the principle of a learning environment, where small, educated teams work toward an objective using the basic scientific method: experiment, measure the results, see if it’s an improvement, and if it is, go with it, if not try something else. Don’t guess, gather data.

The fundamental difference between mass production and lean production is the separation of the planning activity from the execution activity. In the NUMMI plant, work planning was done by the workers, which resulted in an extremely short feedback loop that was continually correcting toward the desired set point. In the former GM plant, work planning was done by industrial engineers resulting in open loop control.

The Solution: Closed Loop Control
Similarly, Just-in-Time provides for work planning at the point of execution, with extremely short feedback loops overseen by the workers, who exercise ultimate control. Contrast this to MRP systems, which is divorced from the workers and provides open loop control (if it provides any control at all). These days, MRP systems are used as overall planning systems, while detailed scheduling is done with the work-level pull systems.

The only way to get closed loop control is to have workers plan the process as well as execute it. The separation of planning from execution comes from a paradigm which regards workers as uneducated and lazy. The integration of planning and execution recognizes that given the proper training, leadership and objectives, workers are more capable of designing and improving their processes than any unengaged organization, be it an industrial engineering office, a materials control office or a project office.
______________________
Footnotes:

1. The Machine That Changed the World : The Story of Lean Production, by Womack, James P., Daniel T. Jones, and Daniel Roos, New York: Rawson and Associates; 1990, page 31.

2. The Principles of Scientific Management, Taylor, Frederick W., first published as an essay in 1911, published by Harper & Brothers, New York, 1919, available as a Dover republication printed in 1998.

3. Taylor, Frederick W., Scientific Management - Comprising Shop Management, The principles of Scientific Management and Testimony before the Special House Committee, 1964, Harper and Row


4. “Time-and-Motion Regained”, Alder, Paul, Harvard Business Review (January-February, 1993) pp97-108.

5. The Machine That Changed the World : The Story of Lean Production, by Womack, James P., Daniel T. Jones, and Daniel Roos, New York: Rawson and Associates; 1990, page 53.

Screen Beans Art, © A Bit Better Corporation

Lean Programming

About the time of the 1980 NBC documentary ‘If Japan Can, Why Can’t We?’, I was the System Manager in a video cassette manufacturing plant, and our management team was asking this question every day.  Our Japanese competition was selling superior products at much lower prices, and we couldn’t figure out how they did it.  We knew we needed to make dramatic changes or close up shop, but we didn’t know what to change.

As far as we could tell, we were doing everything right.  We relied on optimized forecasting methods to determine economic lot sizes, and we used the latest MRP (Manufacturing Requirements Planning) software to launch daily schedules into the plant.  We had a sophisticated computer system that analyzed QC results and process parameters, to pinpoint the causes of defects.

We had some quality problems, and it took a month to fill most orders.  In any given week, we were able to pack out about 60% of the planned line items for the week.  But this was okay, because the other 40% of the week’s packout went into finished goods inventory.  Usually we had plenty of on-hand inventory for shipping standard orders.  Special orders were another matter, however.  The division vice president would often call to expedite special orders for important customers.

We moved in-process video cassettes from one workstation to another on carts, and we had a lot of carts.  There was never enough room to store all the carts at the next workstation, so carts full of inventory would get misplaced.   Sometimes cassettes were stacked on top of carts, and occasionally they would spill on the floor.  Video cassettes piled up in front of testing stations, so whenever a process drifted out of control, it took a while to discover that we were producing marginal product.  We had plenty of rework stations, to be absolutely sure that everything we shipped was good product.

All-in-all, we had about a month’s worth of work-in-process inventory. At the time, we blamed our inability to rapidly fill orders on bad forecasts from marketing. Later we were surprised to learn that the real culprit was our in-process inventory.  Today it is well known that the average shipping time of most supply chains is about the same as the average level of inventory in the supply chain.

Lean Manufacturing
At the end of World War II, Sakichi Toyoda, founder of Toyoda Spinning and Weaving company, dreamed of providing cars for the general public, much like Henry Ford’s dream thirty years earlier.  He chartered Taiichi Ohno to put in place an efficient production system to produce high quality automobiles.  Over the next three decades, Ohno developed the Toyota Production System, now known world-wide as Lean Manufacturing[1].  The foundation Ohno’s system was the absolute elimination of waste.

Ohno studied US manufacturing techniques, and learned a lot from Henry Ford’s pioneer work in assembly line flow.  However, the assembly line produced large lots of identical cars.  Ohno didn’t have the customer base to imitate the US practice of manufacturing in ‘economic’ (ie. large) lot sizes.  He was captivated by US supermarkets, however, where a small quantity of every product was placed on shelves, and as shoppers removed  products, the shelves were rapidly replenished.  He decided to place inventory ‘supermarkets’ throughout his plant, and found that this technique dramatically lowered the ‘waste’ of in-process inventory.  He named these inventory supermarkets ‘kanban’.

Because Ohno was converting a spinning and weaving company to an automobile manufacturer, he already knew how to avoid making bad product.  Founder Toyoda Sakichi had invented an automatic shut-off mechanism that stopped a weaving machine the minute a flaw such as a broken thread was detected.  Ohno moved this concept to car manufacturing, where he insisted that each part be examined immediately after it was processed, and the line stopped immediately if a defect was found.

To maximize product flow, standard work sheets were created, but these were not developed at a desk by engineers.  They were developed on the shop floor by the workers who know the process.  Standard cycle times and kanban shelf space for each item was determined and workflow was leveled.  Production workers were like a relay team, handing off the baton (product) to the next person.  The handoff required 100% quality and tight timing.  If things got delayed, teammates were expected to help each other set up a machine or recover from a malfunction.

Ohno’s aggressive elimination of waste led him to the twin values of rapid product flow and built-in quality.  Over time, Ohno discovered that these two values led to the highest quality, lowest cost, shortest lead time products possible.

Total Quality Management
About the same time, Dr. W. Edwards Deming was teaching Quality Management in Japan.  In fact, the Total Quality Management (TQM) movement cannot be separated from Lean Manufacturing.  Demming’s photo is in the lobby of  Toyota’s headquarters, bigger than the photo of founder Toyoda Sakichi. Demming didn’t find an audience in the US after WW II, because managers at the time thought that poor quality was caused by people who just didn’t want to do a good job.  They didn’t think there was much managers could do to improve quality except exhort employees to do a better job.

Demming’s basic message was that quality is a management responsibility, and poor quality was almost always the result of systems imposed on workers which thwarted people’s desire to do high quality work.  He taught the Japanese managers how to empower production workers to investigate problems and systematically improve processes.  He taught that teamwork and long term, trust-based relationships with suppliers were far better than adversarial relationships.  He emphasized a culture of continuous improvement of both processes and products. 

In the 1980’s, Demming’s fourteen points (See Appendix 1) were studied by virtually every manufacturing manager.  Among these fourteen points are the well known mantra’s:

• Don’t Inspect Quality In.
• Constantly Improve the System.
• Break Down Barriers Between Departments.

But a few of Demming’s fourteen points might seem revolutionary even today, such as:

• Drive Out Fear.
• Eliminate Quotas, Numerical Goals and Merit Ratings.
• Don’t Award Business Based on Price; Minimize Total Cost.

Paradigm Shift
When we first heard about Lean Manufacturing, we thought it was a hoax.  Get rid of safety stock, don’t run machines at full capacity, have suppliers deliver small lots on a daily basis?  This was so counter-intuitive, so against the paradigm of the day, that the Japanese manufacturing techniques were widely discounted.  TQM concepts were more intuitive, but alone they were not enough to lift us out of our dire situation.  Desperate for a change, we decided to give Lean Manufacturing a try, and in the end, it saved our plant.

The critical step in implementing Lean Manufacturing in our plant was a carefully planned changeover from push scheduling to pull scheduling.  We decided that we could not do it part way, we had to switch plant-wide, cold turkey, over a weekend.  We devised a simple simulation which we taught to every one in the plant – managers, shift supervisors, and operators.  Using the simulation, teams of workers designed the layout and flow in their areas, including the kanban cards and rapid changeover methods.  The entire plant held its collective breath as the pull system went into effect, but the workers knew what to do – they had developed the methods themselves.  The first week packout accuracy was 92%, and it got better from there.  We were able to fill special orders in two weeks, so the vice president could stop expediting orders.  In a short time we were down to one week of inventory and could fill any order in the same amount of time.  We had lots of extra space, and quality had never been better.

The most difficult part of implementing Lean Manufacturing was the paradigm shift it required.  Everyone ‘knew’ that large lot sizes were necessary to keep expensive machines running at full capacity.   They also ‘knew’ that machine changeovers took a long time, and every minute a machine was idle its burden rate went up.  In addition, large warehouse inventories were necessary to make sure that when a customer ordered a product it could be shipped immediately.  After all, customers didn’t want to wait the month it took us to produce the product.

One of the reasons why Lean Manufacturing has been so difficult to implement is because people must question established, known truths, and this is not easy.  Another reason is that practices which create local optimization at the expense of the overall system are difficult to recognize, let alone change.  Local optimization points provide attractive points of measurement, and inevitably, what is measured is optimized.

Simple Rules
In a January 2001 article in Harvard Business Review titled ‘Strategy as Simple Rules’, Kathleen Eisenhardt describes how smart companies thrive in a complex business environment by establishing a set of simple rules which define direction without confining it.[2]  She suggests that instead of following complex processes, using simple rules to communicate strategy is the best way to empower people to seize fleeting opportunities in rapidly changing markets.

The 1980’s were a time of profound change in US manufacturing, and the change was guided by a set of simple rules.  Simple rules gave guidance to every level of the organization, and got everyone on the same sheet of music.  They empowered people at all levels of the organization, because the provided guidance for making day-to-day decisions.  With simple rules, work teams were able to continuously improve the processes and products without detailed guidance or complex processes.  

The basic practices of Lean Manufacturing and TQM in the 1980’s might be summed up in these ten simple rules:

   1. Eliminate Waste
   2. Minimize Inventory
   3. Maximize Flow
   4. Pull From Demand
   5. Empower Workers
   6. Meet Customer Requirements
   7. Do it Right the First Time
   8. Abolish Local Optimization
   9. Partner With Suppliers
  10. Create a Culture of Continuous Improvement

These Lean Manufacturing rules have been tested and proven over the last two decades.  They have been adapted to logistics, customer service, health care, finance, and even construction.  The application of the rules may change slightly from one industry to the next, but the underlying principles have stood the test of time in many sectors of the economy.

Lean Programming
Recent work in Agile Methodologies, Adaptive Software Development, and Extreme Programming have in effect applied the simple rules of Lean Manufacturing to software development.  The results, which we call Lean Programming, are as dramatic as the improvements in manufacturing brought on by the Just-in-Time and Total Quality Management movements of the 1980’s.

Lean Rule #1:  Eliminate Waste
The first rule of Lean Programming is:  Eliminate waste.  That is, eliminate anything which does not add value to the final product.  In Lean Manufacturing, waste is identified through a value stream analysis, a process which identifies all activities in the value stream and identifies the specific value they add to the final product. The value analysis process then attempts to find a different, more efficient way to add the same value,

The documents, diagrams, and models produced as part of a software development project are often consumables, aids used to produce the system, but not necessarily a part of the final product.  Once a working system is delivered, the user may care little about the intermediate consumables.  Lean principles suggest that every consumable is a candidate for scrutiny.  The burden is on the artifact to prove not only that it adds value to the final product, but also that it is the most efficient way of achieving that value.

Lean Rule #2:  Minimize Inventory (Minimize Intermediate Artifacts)
In our manufacturing plant, we communicated this message:  Inventory is waste.  Why?  Inventory consumes resources.  Inventory slows down response time.   Inventory hides quality problems.  Inventory gets lost.  Inventory degrades and becomes obsolete.  The ‘benefits’ of inventory are oversold.  The ‘cost’ of inventory almost always outweighs such ‘benefits’.

The inventory of software development is documentation that is not a part of the final program.  As inventory, this documentation should be subject to value analysis.  Take requirements and design documents, for example. How much value do the really add?  How important are they to the final product?  If you compare requirements and design documents to in-process inventory, then it is striking to note that the time it takes to produce these documents probably determines the cycle time of the project.  Just as inventory must be minimized to maximize manufacturing flow, so too requirements and design documents must be kept to a minimum to maximize development flow.

There are many wastes associated with this excess documentation:  The waste of time producing the documents, waste of time reviewing the documents, and the work that goes into change requests and associated evaluations, priority setting, and system changes.  But the biggest waste of all is the waste of building the wrong system if the documentation does not correctly and completely capture the user requirements.

The best approach for minimizing intermediate artifacts is to raise the level of abstraction of documentation.  Instead of a 100 page detailed specification, write a 10 page set of rules and guidelines, and document only the exceptions.  Instead of a few inches of specifics, produce a concise 25 page matrix which summarizes the effort.

We know that users are relatively poor at envisioning the details of a system from most documents, and are even less likely to correctly perceive how it should operate in their environment until they actually use it.  Even if users could predict exactly how the system should operate at the present time, it is unlikely that the way the system is supposed to work months before it is delivered will be exactly the way users need it to work for the rest of its useful life.  All of this must be taken into account when we determine how much value these documents actually add to the final product.

Lean Rule #3:  Maximize Flow (Drive Down Development Time)
During the 1980’s we learned how to make products in hours which used to take days or weeks.  We learned that very rapid product flow resulted in very short cycle times, often one or two orders of magnitude lower than before.  During the 1990’s, e-commerce projects were often able to accomplish in weeks what used to take months or years in the traditional software development world.  Yes, in some sense they cheated.  But the bottom line is, huge amounts of useful software was deployed in the last five years with extremely short cycle times by traditional standards.

In a recent paper titled ‘Reducing Cycle Time,’[3] Dennis Frailey  proposes reducing software development cycle time using the same techniques employed to reduce manufacturing cycle time.  He suggests looking for and reducing accumulations of WIP (Work in Process).  Just as in manufacturing, if WIP is reduced, and the cycle time will be reduced.  To reduce WIP, Frailey recommends using the ‘Small Batch’ principle and the ‘Smooth Flow Principle’, concepts straight from Lean Manufacturing.

Iterative development is basically the application of these principles to programming.  The basic premise of iterative development is that small but complete portions of a system are designed and delivered throughout the development cycle, with each iteration adding an additional set of features.  The cycle time from start to finish of any iteration varies from a couple of weeks to a couple of months, and each iteration engages the entire development process from gathering requirements to acceptance testing.

Lean Rule #4:  Pull from Demand (Decide as Late as Possible)
In our video cassette manufacturing plant, we used to think that it would be ideal if our marketing department could forecast exact market requirements.  A lot of work went into sophisticated forecasting techniques to more accurately predict the future.  Then  one day we realized that we were trying to do the wrong thing.  It would not be ideal if we had a perfect forecast.  Instead, it would be ideal if we could reduce our reliance on forecasts by reducing the system response time so dramatically that the system could respond to change rather than predict it.

In a market where volatile technology requires constant product upgrades, Dell Computer has a huge advantage over it’s keenest competitors because it doesn’t forecast demand, it responds to it by making-to-order in an average of six days.  While Dell holds about six days of inventory, it’s competitors maintain six weeks of inventory.  Dell’s ability to make decisions as late as possible gives Dell a significant competitive advantage in a fast-moving market.

Software development practices which keep requirements flexible as close to system delivery as possible can provide a significant competitive advantage in a changing market.  In a volatile business environment, users are not able forecast their future needs accurately.  Freezing the design early in a software development project is just as speculative as forecasting.  Software systems should be designed to respond to change, not predict it.  In software development, as in building computers, the ability to make decisions as late as possible provides a competitive advantage.

Lean Rule #5: Empower Workers (Decide as Low as Possible)
A basic principle of Lean Manufacturing is to drive decisions down to the lowest possible level, providing both the tools and the authority for people “on the floor” to make decisions. When Toyota took over GM’s manufacturing plant in Fremont, California in 1983, it inherited workers with the worst productivity and absenteeism record in the industry. Those same workers doubled their quality and productivity record in two years. This was accomplished through formation of teams that were trained in work measurement and improvement techniques and expected to develop and continually improve their own work standards and practices.[4]

One of the problems with heavyweight intermediate documentation is that it attempts to make all of the decisions for developers, rather than giving them a set of guidelines.  In general, raising the level of abstraction of intermediate artifacts will give guidance as well as freedom to the developers as they make the detailed design and programming decisions.  It is always better to tell developers what needs to be done, not how to do it.

Developers need to understand the goal of their work and how it fits into the overall flow, what it means to meet customer requirements, and the architectural structure and GUI standards of the system.  They also need to know what they must accomplish, by when, and how to tell when it is complete.  Finally, their work needs to be made visible in short iterative cycles to provide the feedback necessary for continual improvement.

Lean Rule #6:  Meet Customer Requirements (Now and in the Future)
In his 1979 book ‘Quality is Free’, Philip Crosby defines quality as ‘conformance to requirements’.   The Standish Group study of 1994[5] noted that the most common cause of failed projects was missing, incomplete, or incorrect requirements.   The software development world has responded to this risk by amplifying the practice of gathering detailed user requirements and getting user sign-off prior to proceeding with system design.   However, this approach to defining user requirements is deeply flawed.

I  worked on one project in which the customer wanted a complex system delivered in ten months.  Time was of the essence – 10 months or bust.  And yet, being a government agency, the contract required sign-off on an external design document before internal design and coding could begin.  Several users were involved, and they dragged their feet on signing the documents.  Why?  They were concerned that they might approve something which would prove to be a mistake later on.  Since there was no easy way to change things after the design documents were signed, they took two months to approve the design.  An who can blame them?  Their jobs depended on them getting it right.  So half way into a very tight schedule, over two months of time and a lot of paper was wasted producing and getting user sign-off on design documents. 

Instead of encouraging user involvement, user sign-off tends to create an adversarial relationship between the developers and the users.  Users are required to make decisions early in the development process, and are not allowed to change their minds, even when they do not have a full concept of how the system will work or how their business situation may develop in the future.  Users are understandably reluctant to make these commitments, and they will instinctively delay decisions to as late in the process as possible.  Note that this instinct on the part of the users is in line with Lean Rule #4.

The most effective way to accurately capture user requirements is found in the iterative approach to system development.  By developing core features early and obtaining customer feedback in a focus-group demonstration of each iteration, a far more correct definition of customer requirements can be obtained.  In addition, if we accept that the requirements will necessarily change over time, we must start with the essential requirement that the system must be designed to easily adapt to changes over its lifecycle.

Lean Rule #7:  Do it Right The First Time (Incorporate Feedback)
Before Lean Manufacturing arrived at our plant in the early 1980’s, we occasionally had output of marginal quality. We would test to find the good product and rework the bad product. After understanding the “Do it Right the First Time” rule, we closed down rework stations and stopped trying to test quality into the product. Instead, we assured that each component was good at every handoff.  This involved having tests and controls at every point of manufacture to detect a drift toward out-of-spec product and stop production before any bad product was made.

“Do It Right the First Time” did not mean “Freeze the Spec”.  On the contrary, product specs changed constantly, and lean discipline meant being able to flawlessly adapt to changing market conditions.  This was accomplished through a product architecture which facilitated manufacturing change, monitoring techniques which detected errors before they happened, and tests which were designed before manufacturing began.

In 1987 Barry Boehm observed that it costs 100 times more to find and fix a problem after software delivery than to find and fix it in early design phases.[6] This observation and the “Do it Right the First Time” rule have been widely used to justify the overhead of developing a detailed system design before code is written.

The problem lies in the assumption that it is possible to generate a detailed set of documents that correctly define customer requirements, and that those requirements will not change. The fact is that requirements do change, and frequently, over the life of most systems. “Do it Right” has also been misinterpreted to mean “don’t allow changes.” In fact, once we acknowledge that change is a fundamental customer requirement, it becomes clear that what “Do it Right” requires that we provide for change.

If we want to meet customer requirements, and we acknowledge that customers don’t really know what they want at the beginning of development, then we need to incorporate a method of obtaining customer feedback during development.  Instead, most software development practices include a “Change Control Process” which makes it so difficult to respond to user feedback that developers are discouraged from asking for it.  Far from insuring a quality result, these change-resistant practices actually get in the way of “Doing it Right”.

Lean Programming employs two key techniques that make change easy. Just as Lean Manufacturing builds tests into process so as to detect when the process is broken, Lean Programming builds tests into the development process in order to ensure that when changes don’t inadvertently break the code. In fact, the best approach is to write the tests first, and then write the code. An excellent unit and regression testing capability is the best way to encourage change late in the development process.

The second technique for allowing change to happen late in development is refactoring, or improving the design of existing software in a controlled and rapid manner. When refactoring is an accepted practice, early designs can focus on the issue at hand rather than speculate as to what additional design elements will be needed. As the additional features are actually added, refactoring provides a new, simplified design to handle the new reality. When refactoring is a part of the process, we reduce speculation as to what will be needed in the future by making it easy to accommodate the future if and when it becomes the present.

Lean Rule #8: Abolish Local Optimization (Sub-Optimized Measurements are the Enemy)
In the 1980s, the biggest enemy of Lean Manufacturing was often the accounting department. We had big, expensive machines in our plant, and the idea that they should not be run at full capacity was radical, to put it mildly. We compiled daily reports of work-in-process inventory, and the accountants didn’t want these reports abandoned just because there was virtually no WIP to report.

A generation of accountants had to retire before it was “OK” to run machines below their full capacity. Designing machines for rapid changeover rather than highest throughput remains a tough sell even today. After 20 years, Lean Manufacturing is still counter-intuitive to those who lack a broad view of the enterprise.

In this context, let’s examine the role of managing scope in a software development project. Project managers have been trained to focus on managing scope, just as we in manufacturing were trained to focus on maximizing machine productivity. However, Lean Programming is fundamentally driven by time and feedback. In the same way that localized productivity optimization creates a sub-optimized overall process, so too, does focus on managing scope create a sub-optimized overall project management process.

Think about it – holding the scope to exactly what was envisioned at the beginning of a project has little value for the user whose world is changing.  In fact, it adds anxiety and paralyzes decision-making.  It doesn’t add much to the ultimate system, which will be outdated by the time it is delivered. Managing to a scope that is no longer valid wastes a lot of time and takes up a lot of space on issue lists, trade-off negotiations and ultimate fixes to the system to get things right in the end. However, as long as keeping a project within its original scope is a key project management goal, this measurement will continue to be optimized—at the expense of the overall value delivered by the project.

Scope will take care of itself if the domain is well understood and there is a well-crafted, high-level agreement on what the system will do in the domain. Scope will take care of itself if the project is driven in time buckets that are not allowed to slip. Scope will take care of itself if both parties focus on rapid development and solving the user’s problem, and adopt waste-free methods of achieving these goals.

Lean Rule #9: Partner With Suppliers (Use Evolutionary Procurement)
Lean Manufacturing did not remain in the manufacturing plant. Once the idea of partnering with suppliers was combined with an understanding of the value of rapid product flow, Supply Chain Management was born. People began to realize that it took tons of paperwork to move material between companies, and this did not add value to the product. Moreover, the paperwork cost more than one might expect, not to mention the delay in product flow that it caused. Even today, predictions of billions of dollars of savings resulting from business-to-business Web portals are based on cutting the cost of transactions required to move goods between companies.

Supply Chain Management caused companies to take a close look at their business-to-business contracts. All too often, these contracts were focused on keeping the companies from cheating each other. In addition, it was common to pit one vendor against another to assure supply and obtain the lowest cost. Again, Lean Manufacturing changed this paradigm. Deming taught that trusting relationships with single suppliers create an environment that allows optimizing the overall value to both companies.

Throughout the 1980s, companies achieved the highest quality and lowest costs in their supply chains by reducing the number of suppliers and working with the remaining suppliers as partners. The quality and creativity which resulted from collaborating supply chains was demonstrated to far outweigh the apparent (and sub-optimal) benefits that came from competitive bids and rapid turnover of suppliers. Partnering companies helped each other improve product designs and product flows. They linked systems to allow just-in-time movement of goods across several suppliers with little or no paperwork. The long-term advantages of a collaborative supply chain relationships are well documented.

Wise companies realize that traditional software development contract practices generate hidden wastes. As manufacturers discovered in the 1980s, trusted relationships with a limited set of suppliers can yield dramatic advantages. Without the adversarial relationship created by a constant focus on controlling scope and cost, software development vendors can focus on providing the best possible software for customers, fixing requirements as late as possible in the development process and providing the most value for the available money.

Lean Rule #10: Create a Culture of Continuous Improvement
When software development seems to be out of control, one response has been to increase the level of “software maturity” of the organization. This might seem to be in line with good manufacturing practice, where ISO 9000 certification and Malcom Baldridge awards are sometimes equated with excellence. However, these process documentation programs indicate excellence only when the documented process is excellent in the context of it’s use.

In many current software development projects, excellence means the ability to adapt to fast moving, rapidly changing environments.  Process-intensive approaches such as the higher levels of Software Engineering Institute's (SEI) Capability Maturity Model (CMM) may lack the flexibility to respond rapidly to change.  In a recent e-mail advisor from Cutter Consortium, Jim Highsmith highlights the tension between such heavyweight methodologies and lightweight methodologies such as Lean Programming.[7]

The question becomes, do process documentation certification programs stifle, rather than foster, a culture of continuous improvement?  Deming would probably turn over in his grave at the thought of tomes of written processes substituting for his simple Plan-Do-Check-Act approach:

• Plan:     Choose a problem. Analyze it to find a probable cause.
• Do:        Run an experiment to investigate the probable cause.
• Check:   Analyze the data from the experiment to validate the cause.
• Act:     Refine and standardize based on the results.

Iterative development allows the use of the Plan-Do-Check-Act approach within a project. During the first iteration, the hand-off from design to programming or programming to testing may be a bit rough. It’s okay if the first iteration provides a learning experience for the project team, because there are more iterations to come, so the team can improve its process. In a sense, an iterative project environment becomes an operational environment, because processes are repeated and Deming’s techniques of process improvement can be applied from one iteration to the next.

Product improvement is also possible with iterations, particularly if refactoring is used. In fact, refactoring provides a tremendous vehicle to apply the principle of continuous improvement to the programming environment.

However, we need improvements that span more than a single project. We must improve future project performance by learning from existing ones. Here again, Lean Manufacturing can point the way. During the 1980s, a set of practices summarized in the ten rules of Lean Manufacturing were adopted widely across most manufacturing plants in the West. These practices then spread to service organizations, to logistics organizations, to supply chains, and beyond. They have withstood the test of time across multiple domains.

Following the simple rules of Lean Manufacturing has brought dramatic improvements to every industry in which they have been applied. These same rules can and should be applied to software development projects. The resulting Lean Programming practices will lead to the highest quality, lowest cost, shortest lead time software development possible.
_______________

Appendix 1:  Summary of W. Edwards Demming’s 14 points

   1. Create consistency of purpose.
   2. Adopt a win-win philosophy.
   3. Don’t depend on mass inspection; build quality in.
   4. Don’t award business based on price; minimize total cost; build long-term relationships of loyalty and trust with a single suppliers.
   5. Constantly improve the system of production, service, planning, etc.
   6. Train for skills.
   7. Provide leadership:  help people do a better job.
   8. Drive out fear and build trust so everyone can do a better job.
   9. Break down barriers between departments; abolish competition and build a win-win system of cooperation.
  10. Eliminate slogans, exhortations and zero defect targets; the cause of the bulk of problems lie in the system, and are beyond the power of workers to correct.
  11. Eliminate quotas, numerical goals and Management by Objectives; substitute leadership.
  12. Remove barriers that rob people of joy in their work; abolish the annual rating or merit system.
  13. Educate and improve individuals.
  14. Involve the entire organization.

There are many summaries of Demming’s 14 points, which he modified throughout the years, in the spirit of continuous improvement.  The above summary is based on his last version of the 14 points.[8].
_________________

References

[1] The Machine That Changed the World : The Story of Lean Production, by Womack, James P., Daniel T. Jones, and Daniel Roos, New York: Rawson and Associates; 1990

[2] Strategy as Simple Rules, by Eisenhardt, Kathleen M and Donald N. Sull, Harvard Business Review, Volume 79, Number 1, January 2001, pp 107- 116

[3] Reducing Cycle Time, by Frailey, Dennis, Software Development Magazine, August, 2000

[4] Time-and-Motion Regained, by Paul Adler, Harvard Business Review, January-February 1993 pp 97-108

[5]Charting the Seas of Information Technology – Chaos, by The Standish Group International, 1994

[6] Industrial Software Metrics Top 10 List, by Boehm, Barry, IEEE Software, Volume 4 Number 5, September, 1987, pp 84-85

[7] E-Projects in India, by Jim Highsmith, e-Project Advisor, Cutter Consortium's e-Project Management Advisory Service, March 1, 2001.

[8] Gone But Never Forgotten, by Brad Stratton, editor, Quality Progress Magazine, March 1994
______________

Annotated Bibliography (Chronological)

Quality Control Handbook, Joseph M. Juran, originally published in 1951, now in its forth edition
Considered the standard reference in the field of quality. Like Demming, Juran consulted mainly in Japan during the 1950’s and 60’s.
Managerial Breakthrough, Joseph M. Juran, originally published in 1964
Widely ignored when it was first published, this book is now considered a landmark treatise on continuous improvement.
Quality is Free, by Philip Crosby, New York:  McGraw-Hill, Inc. 1979
This is the book we used to launch the TQM program in our plant.  Demming never liked the zero defects approach advocated in this book, and it contains little about statistical quality control.  However, many corporate executives got the quality message from Phil Corsby.
Study of ‘Toyota’ Production System from Industrial Engineering Viewpoint, by Shigeo Shingo, Osaka, Japan, Shinsei Printing Co. Ltd.  1981
The title of this book is an indication of the quality of the translation, but that was not a barrier to our plant.  We studied the book cover-to-cover and got our introduction to Lean Manufacturing from this book.
Toyota Production System, Practical Approach to Production Management, by Yasuhiro Monden, Norcross, Georgia, Industrial Engineering and Management Press.  1983
This is the classic book on Lean Manufacturing.
Zero Inventories, by Robert W. Hall, Homewood, IL, Dow Jones-Irwin 1983.
Robert Hall, a professor at Indiana University, understood the concept of Just-in-Time earlier than most academics.  This book is still considered the definitive work on JIT.
A Revolution in Manufacturing, the SMED System, by Shigeo Shingo, Cambridge, MA, Productivity, Inc.; Originally published as Shinguru Dandori in 1983, English translation 1985.
The title of this book is an indication of the quality of the translation, but that was not a barrier to our plant.  We studied the book cover-to-cover and got our introduction to Lean Manufacturing from this book.
The Goal, by Eliyahu M. Goldratt, First Edition Published in 1984, Second Revised Edition Published in Great Barrington, MA, 1992
This book is a business novel. It is the easiest book to read for an introduction to Lean Manufacturing and is definitely a classic.  You will find this book on the reading list of most Operations Management courses.  Goldratt went on to develop the ‘Theory of Constraints’ and write several related business novels.
Out of the Crisis, by W. Edwards Demming; 1986
This is the book in which Demming outlined his famous 14 points.
The Demming Management Method, by Mary Walton and W. Edwards Demming; 1988
Probably the best book there is on Demming and his management approach.
Toyota Production System – Beyond Large Scale Production, Taiichi Ohno, published in Japanese in 1978 and in English in 1988 by Productivity, Inc.
This is an explanation of JIT by the inventor. The English version arrived after JIT had become widespread in the US.
The Machine That Changed the World : The Story of Lean Production, by Womack, James P., Daniel T. Jones, and Daniel Roos, New York: Rawson and Associates; 1990
This landmark book about the Toyota Production System was the first to associate the term ‘Lean’ with manufacturing.
Lean Thinking, by Womack, James P., and Daniel T. Jones, Simon & Schuster; 1996
A follow-up on the story of Lean Production, this book extends the concept of Lean throughout the enterprise.  It shows how the "lean principles" of value (as defined by the customer), value stream, flow, pull, and perfection can be applied to all areas of the enterprise.  (Software development is implicitly included.)

Originally Published in Software Development Magazine, May and June, 2001.

Screen Beans Art, © A Bit Better Corporation