When Demand Exceeds Capacity

We are getting solar panels on our roof.  Someday.  We signed a contract last November and the original estimate for installation was May. Since no one does outside work in the winter here in Minnesota, this was about two months after the earliest possible installation date. But May has come and gone, and we still do not have an installation date. Why? The electrical design of for our 40-year-old house is complex and the young solar company has limited electrical design capability. So does our mature electrical utility company. Between the two of them, our project has been held up for a long time. 

Solar systems have become widely popular since the electrical grid meltdown in Texas and the sharp rise in energy prices. So, it is not a surprise that the nascent solar industry is dealing with far more demand than the young solar companies and staid utility companies are equipped to handle. It is difficult to find enough talented people to do the complex system design, project management, and installation of solar systems. Thus, solar customers experience long delays that cost us months of potential savings, while they suffer from months of delayed revenue.  

Our solar company has an eager sales force and a handful of project managers. Once contracts are signed, sales people hand them off to project managers so they can focus on pursuing more sales. Since the rate of signed contracts greatly exceeds the rate of completed installations, the number of customers assigned to each project manager is constantly increasing. It is impossible for a project manager to manage dozens of complex solar projects at the same time, but that is what they are expected to do. 

Clearly, our solar company is unwilling to reduce its sales rate to match its project completion rate and unable to increase the completion rate to match the rate at which contracts are signed. Thus, the length of time a project takes continually increases, putting customers and project managers in a lose-lose situation. In fact, the solar industry in hardly the first industry in the world to experience far more demand that it has the capacity to supply; many other industries have navigated these waters. Typically, companies that do not learn how to manage excess demand end up failing because they generate annoyed customers faster than they generate revenue. Companies that became giants in rapidly growing industries – for example, Airbnb, AWS, Google, Netflix, and SpaceX – learned quickly to increase their delivery capacity to match the rate at which they generated sales. Successful solar companies will have to learn the same thing.

How do companies rapidly increase delivery capacity? Hiring more people might help, but it is not nearly enough. Young companies in a growth industry must figure out how to complete more jobs faster while doing less work or they will not be able to keep up with the rapid increase in demand. There are three essential strategies that successful industry giants have used to increase efficiency which other companies would do well to understand.

1. Focus on Customers Outcomes

The first strategy for increasing efficiency is to pay attention to the metrics that drive behavior. Startups tend to measure success by counting the number sales, while more efficient companies tend to measure the number of satisfied customers. When success is measured by the number of satisfied customers rather than booked revenue, several good things happen. The company learns not to accept work at a faster rate than it can complete work, because doing so does little more than stretch out the time it takes for its people to be successful while lengthening the queue of increasingly impatient customers. Work teams also learn to focus on completing installations as quickly as possible, because more installations will be seen as better performance. Finally, teams are unlikely to develop products that customers are not interested in – a very common source of wasted effort in startup companies.

It is extraordinarily difficult for an ambitious young company to turn away people that are beating a path to its door. Everyone’s instinct is to hire more salespeople and let the work pour in.  But let’s face it, the most important thing that young company needs to learn is how to COMPETE work quickly, not how to ACCEPT work quickly. If it opens the doors wide and lets everyone in, there will be so many customers milling about that workers will be distracted trying to deal with the chaos rather than focused on getting work done. The most important thing for that young company to do at this point is to stop spending so much time on managing excess work and start concentrating on how to get jobs done as fast as they arrive.

Every growth industry has gone through this learning curve. Once upon a time, mail order catalogs (Sears, for example) accepted orders as fast as they could and shipped them in about two weeks. Because of the delay, they had to deal with people changing their mind and a complex order change system was necessary. Then L.L.Bean had a novel idea – why not ship products within 24 hours of receiving the order? It worked – customers were delighted, and no order change system was needed. Today almost all fulfillment systems work on the same principle – focus on building the capacity to ship product at the same rate orders arrive. Once shipping capacity matches demand, the company can ship immediately after an order arrives, delighting customers and reducing congestion while giving customers little time to change their minds.  

2. Manage FLOW, not tasks

The second strategy for increasing efficiency is counterintuitive; it requires a shift in the company’s mental model of how work gets done. Consider our solar company. As a small startup, it thinks of every project as a separate entity and assigns a project manager to sequence the steps in an installation so all necessary work gets done. Unfortunately, this approach is not readily scalable, and even if it were, there are not enough experienced project managers available to hire. Industry leaders do things differently; they manage packages of items as they move through their system, not individual projects or tasks. 

What does it mean to manage groups of items?  Consider the shipping industry.  Until the 1950’s, ships were loaded with individual items packed carefully to keep the ship stable. Each item had to be traced through every port as it moved off the ship and onto another ship or a train or a truck. Then the shipping container was invented, and it was no longer necessary to manage individual items moving through the supply chain. Instead, shippers managed containers as they moved from origin to destination.  

Some years ago, we bought a sofa that was heavily discounted because of a missing headrest. A few months later the store called to say it was filling a container of furniture from their supplier in Norway and they could add our missing headrest for a small fee. It would have been too complex (and expensive) to order and track the headrest individually; but it was simply assigned to the container in which many items moved as a package from Norway to Minnesota. The cost of shipping was negligible. It's easy to see how containers created a reduction in complexity that dramatically reduced the cost of shipping goods and led to substantial changes in the global economy. 

How do you group and manage workflow in a growing solar company?  First, you think of a contract as a number of steps that need to be completed. Certainly, some of the steps can be done in parallel – for example solar panel layout can be done in parallel with electrical design. Other steps have prerequisites that must be completed first. You organize work so that each step has its own responsible team, its own queue, and its own completion rate. As each new contract is signed, it is coded with the various steps it must move through, including any required sequencing. Then you release the contract to the starting step (or steps). When a step is done, work should automatically move to the next step(s) until all work is complete. Your job is to manage the FLOW of work through the system rather that the progress of individual items through their various steps.

How do you manage the flow of work through a system? Start by managing the amount of time a job spends waiting for attention in the queue of each step. Queueing theory is clear: if you limit the length of a queue and send work through in FIFO (first-in-first-out) order, then time in a queue is the number of items in the queue divided by the average completion rate. For example, if you limit the items in a work queue to twenty and each item takes two days, work will wait in the queue – if it is FIFO – for no more than ten days. (Note: If a FIFO queue is not acceptable, then the queue is too long! Make it shorter.) 

Once a job reaches the end of a queue, it should flow quickly through the work step. The best approach is for a responsible team begin a job, focus on it until it is done, then go on to the next job. Focusing on one thing at a time improves the quality of the work, eliminates wasted time due to multitasking and gets everything done faster. (See Single Responsibility Teams, below.) 

If a step in the workflow is completed inside of your company, you have a lot of control over the rate at which work moves through the step (queue time + work time). Your goal is to create short, predictable completion rates for every internal step. If a step occurs outside your company (the utility company, for example, or the city permitting process) you do not have control of the time that takes, but you can measure it – you should track average time and variation. Note that it is important to separate internal steps from external steps – otherwise variations in the external completion rate will make it impossible to control internal rates of completion.

It is also important to ensure that downstream steps have capacity in their queues to handle work as it is completed upstream. If this sounds complicated, an analogy might be helpful: Consider an order coming into Amazon.  Once upon a time, orders were handled by a central database which directed each step in the processing of orders as they moved through the company.  But in the early 2000’s Amazon outgrew this process; they could not buy large enough database computers to keep up with surges in demand. Amazon needed a new approach, so it abandoned the central databases and thought of each order as moving through a series of independent services. When an order is placed, it is coded with an initial chain of necessary services (Do we need to do a credit check? Does the product require special shipping? ...) and sent on its way through the various processing steps. A service can divert the order or add additional steps (Oops, product is out of stock! ...) and downstream processes are expected to have the capacity to accept work from upstream processes. (If the product can be picked from a warehouse, it should be possible to package and ship it immediately.)

Armed with a set of steps each contract needs and the time each step will take, you, like Amazon, should be able to predict the expected installation date very early – ideally when the contract is signed. You should also have a handle on variation due to outside queues and dependencies such as parts availability and supply chain delays. You can send the contract into your workflow, track how well it is coming along and flag exceptions as soon as they occur. 

In addition, you can spot bottlenecks that need to be improved and generally manage the capacity of your organization. You can be alerted when you need to limit sales to avoid overcommitment: Do not accept work faster than the rate at which work moves through your bottleneck process. Focus on the bottleneck until it is improved, when of course a new bottleneck will appear. Find the next bottleneck and limit sales of projects that use it to the rate the bottleneck can handle. Repeat... 

By far the biggest benefit of managing the flow of groups of items is this: You will get a lot more done with a lot less effort. How can this be? When you no longer have half-done work clogging up your system, demanding attention, and distracting workers, everyone can spend much more of their time actually getting useful work done.   

3. Form Single-Responsibility Teams

The third big change you should make is to charter single-responsibility teams, organized and led by single-responsibility leaders. What is a single-responsibility team?  It is a team that has one and only one thing to do, a team that can focus all its efforts on accomplishing one thing at a time and doing that one thing well. Be careful to minimize dependencies among these teams; each team should be able to begin and complete its work independently; teams should not need to coordinate with or get permission from managers or other teams. Finally, work should not appear in the queue of a single-responsibility team until everything is in place for that work to be done, otherwise the team will have to finish someone else’s work before they can focus on their own. 

How is it possible to organize complex work into pieces that can be accomplished by single-responsibility teams? While there is no such thing as the “best” organizational structure for companies in diverse domains, it is easy to find examples that demonstrate the concept. Let’s start with AWS (Amazon Web Services). Over the past decade AWS has replaced most enterprise software companies with an array of independent services, each managed by a single-responsibility team and strung together through hardened interfaces. No one has ever seen an AWS roadmap; instead, teams identify (or are given) a customer problem and they start by writing a “press release” describing how their solution will solve the problem. Then they proceed to develop the solution, testing it with customers as often as possible. AWS has released multiple new enterprise-level services every year since 2012, each aimed at solving a specific customer problem. These services have gradually lured companies away from the complex, integrated packages of traditional vendors, often prompting AWS customers to organize their own work around autonomous, focused teams.

Now let’s look at SpaceX, which is organized very differently than AWS. SpaceX is fundamentally an engineering organization; its teams are organized around the components of booster rockets that launch things into space. There is a team for each stage of the booster (eg. payload, stage 1, stage 2, engine cluster…) with sub-teams for each sub-component of the stage (Merlin engine, landing legs…).  

You might wonder: How do teams know what to do and when to do it? How do they coordinate with other teams? How do they know they have done a good job?  SpaceX operates on the principle of responsibility, which says that each team / sub-team is expected to understand the role its component must play in the next launch and have it ready by the scheduled launch date (perhaps a couple of months away). The team, led by a responsible engineer, has one job: Ensure that the component works as well as possible during the launch and does its job as part of the overall system. 

How does SpaceX manage FLOW? For a large, multi-team development effort involving both hardware and software, managing flow is usually accomplished by scheduling a series of frequent integration events. The purpose of each event is to take a well-defined step forward in the overall development effort to learn what works, locate unintended integration effects, and discover ways to improve the system. A series of integration events sets the cadence for all teams involved in the development effort. In the case of SpaceX a test launch is scheduled every few months throughout a development cycle; importantly, launch dates Do. Not. Change. Each launch is an integration event that moves all teams forward together, creating a steady flow of real progress toward a working system. 

Of necessity, based on the structure of a booster, SpaceX teams have more dependencies than AWS teams. They also have a clear responsibility to understand and account for those dependencies. Since SpaceX teams work to meet fixed launch deadlines, their systems sometimes fail during a launch. Failures are part of the process, but teams must carefully document launch behavior to be sure that the cause of any failure is identified and eliminated. Believe it or not, this approach to development is much faster and less costly (and usually safer!) than trying to think through every failure mode and eliminate any possibility of failure before attempting a test launch. 

Does SpaceX focus on customer outcomes? The company undertook the difficult challenge of developing a reusable booster because it was clear that sending things into space was too expensive for many potential customers. Reusing boosters was an essential step in dramatically reducing the cost of a launch, significantly increasing the number of customers with access to space. This need to reduce launch costs has driven most major booster development decisions at SpaceX.

SpaceX’s reusable booster has reduced the cost of launching a kilo of payload into space by a factor of 7, while spending an order of magnitude less money on development than previous booster rockets.

Summary

A lot of companies have more work than they can possibly handle – and that is not a good thing. It creates long delays, excess complexity, and eventually, dissatisfied customers. The secret is to limit demand to capacity and then increase capacity to meet additional demand by creating a steady flow of work while teams focus on a single responsibility. You also need to stop doing work that does not contribute to great customer outcomes: Stop multitasking, stop prioritizing, get rid of backlogs, stop shepherding individual tasks. If companies as large and successful as AWS and SpaceX can organize work around customer outcomes, steady flow, and focused, single-responsibility teams – then so can you. 

Integration Does. Not. Scale.

In times past, there was a difference between the front office of a business – designed to make a good impression – and the back office – a utilitarian place where most of the routine work got done. The first (and for a long time the predominant) use of computers in business centered around automating back office processes, so of course, IT was relegated to the back office. 

As businesses grew, various back office functions developed their own computer systems – one for purchasing, one for payroll, one for manufacturing, and so on. The manufacturing system in vogue when I was in a factory was called MRP – Material Requirements Planning. As time went on, MRP systems were expanded to the supply chain, and then to the rest of the business, where they acquired the name ERP – Enterprise Resource Planning.

Over time it became obvious that the disparate systems for each function were handling the same data in different ways, making it difficult to coordinate across functions. So IT departments worked to create a single data repository, which quite often resided in the ERP system. The ERP suite of tools expanded to include most back office processes, including customer relationship management, order processing, human resources, and financial management. 

The good news was that now all the enterprise data could be found in the single database managed by the ERP system. The bad news was that the ERP system became complex and slow. Even worse, enterprise processes had to either conform to “best practices” supported by the ERP suite or the ERP system had to be customized to support unique processes. In either case, these changes took a long time.

ERP Systems Meet Digital Organizations  

As enterprise IT focused on implementing ERP suites and developing an authoritative system of record, the Internet became a platform for a whole new category of software, spawning new business models that did not fit into the traditional processes managed by ERP systems. Here are a few examples:

1. Many software offerings that used to be sold as products are now being sold “as a service”. However, ERP systems were designed to manage the manufacture and distribution of physical products; they don’t generally manage subscription services.
2. Some companies (Google for example) give away their services and sell advertising. Other companies (such as EBay and Airbnb) create platforms that unite consumers with suppliers, often disrupting traditional industries. In a platform business, the most critical processes focus on driving network effects by facilitating interactions between buyers and sellers. Although ERP systems can manage both suppliers and customers, they usually do not focus on the interactions between them.
3. The Internet of Things (IoT) brings real time data into many processes, changing the way they are best executed. For example, predictive maintenance of heavy equipment can be scheduled based on sensor data, resulting in better outcomes for customers and thus for the enterprise. ERP suites are intended to support standard practices; they struggle to support processes that change dynamically in response to digital input.
4. Capitalizing on the availability of data generated by products, companies are moving to selling business outcomes rather than individual products (GE is an example). When you are selling engine thrust or lighting costs, rather than engines or lightbulbs, processes need to be focused on the customer context. ERP systems generally focus on internal processes.
5. ERP systems are supposed to provide a single, integrated record of important enterprise data, but that data rarely includes dynamic product performance data, information about consumer characteristics and preferences, or other information that has come to be called “Big Data”. This kind of information is becoming an extremely valuable resource, but there isn’t room in ERP databases to store and manage the massive amount of interesting data that is available.

In summary, digitization is bringing the back office much closer to the front office, providing the data for dynamic decision-making, and substituting short feedback loops and data-driven interactions for “best practices.” Since enterprise ERP suites were not built for speed or rapidly changing processes, they are increasingly being supplemented with other systems that manage critical enterprise processes.

Postmodern ERP

In the last few years, in the wake of the success of Salesforce.com, many cloud-based software services have become available. Some target the entire enterprise (NetSuite for example), but many are focused on particular areas (e.g. human resources) or particular industries (e.g. construction). These services are finding an eager audience – even in companies that have existing ERP systems. Today, about 30% of the spend for IT systems is coming from business units outside of IT [1]. If they cannot get the software they need from their IT departments, business leaders are likely to purchase cloud-based services instead.

The cloud reduces dependence on a company’s IT department, so it has become quite easy for various areas of the enterprise to independently adopt “best-of-breed” solutions specifically targeted at their needs, rather than use a single ERP suite across the enterprise. These best-of-breed systems are usually selected by line business leaders and hosted in the cloud. They tend to be faster to implement and more responsive to changing business situations than the enterprise ERP suite – partly because they are decoupled from the rest of the enterprise. Gartner calls the movement from a single ERP suite to a collection of ERP modules from multiple vendors “Postmodern ERP”[2]. 

Gartner warns that a multi-vendor ERP approach can lead to significant integration problems, and recommends that multiple vendors should not be used until the integration issues are sorted out. Of course, business leaders want to know why integration is important. IT departments typically respond that the ERP’s central database is the enterprise system-of-record; other ERP modules – financial reporting, for example – depend on this database for critical data. Without an integrated database, how will the rest of the enterprise be able to operate? How will the accounting department produce its required financial reports? 

Integration Does. Not. Scale

But hold on. There are plenty of very large companies that work remarkably well – and produce financial reports on time – without an integrated system-of-record. In fact, internet-scale companies have discovered that integration does not scale. If we go back to the year 2000, we find that Amazon.com had a traditional architecture – a big front end and a big back end – which got slower and slower as volume grew.  Eventually Amazon abandoned its integrated backend database in the early 2000’s, in favor of independent services that manage their own data and communicate with each other exclusively through clearly defined interfaces. 

If we have learned one thing from internet-scale players, it’s that true scale is not about integration, it is about federation. Amazon runs a massive order fulfillment business on a platform built out of small, independently deployable, horizontally scalable services. Each service is owned by a responsible team that decides what data the service will maintain and how that data will be exposed to other services. Netflix operates with the same architecture, as do many other internet-scale companies. In fact, adopting federated services is a proven approach for organizations that wish to scale to beyond their current limitations. 

Let’s revisit the enterprise where business units prefer to run best-of-breed ERP modules to handle the specific needs of their business. This enterprise has two choices: 

1. Integrate the various ERP modules and store their data in a single ERP database.
2. Coordinate independently-maintained enterprise data through API contracts. 

The problem with the first option is that integration creates dependencies across the enterprise. Each time a data definition in the central database is added or changed, every software module that uses the database must be updated to match the new schema. This makes the integrated database a massive dependency generator; the result is a monolithic code base where changes are slow and painful. 

Enterprises that want to move fast will select the second option. They will move to a federated architecture in which each module owns and maintains its own data, with data moving between modules via very well defined and stable interfaces. As radical as this approach may seem, internet-scale businesses have been living with services and local data stores for quite a while now, and they have found that managing interface contracts is no more difficult than managing a single, integrated database.

What Scales

Assume that every team responsible for a process can choose its own best-of-breed software module and is responsible for maintaining its own data in appropriately secure data stores. Then maintaining an authoritative source of data becomes an API problem, not a database problem. When the system-of-record for each process is contained within its own modules, new modules can be added for handling software-as-a-service, two-sided platforms, data from IoT sensors, customer outcomes or other new business model that may evolve. These modules will exchange a limited amount of data through well-defined API’s with the credit, order fulfillment, human resources, and financial modules. Internally, the new modules will collect, store, and act upon as much unstructured data and real time information as may be useful. More importantly, these modules can be updated at any time, independent of other modules in the system. In addition, they can be replicated horizontally as scale demands. 

It is the API contract, not the central database, that assures each part of the company looks at the same data in the same way. Make no mistake, these API contracts are extremely important and must be carefully vetted by each data provider with all of its consumers. API contracts take the place of database schema, and data providers must ensure that their data meets the standards of a valid system-of-record. However, changes to an API contract are handled differently than most database schema changes. Each change creates a new version of the API; both old and new versions remain valid while other software modules are gradually updated to use the new version. A wise API versioning strategy eliminates the tight coupling that makes database changes so slow and cumbersome. The reason why federation scales – while a central database approach does not scale – is because with a well-defined API’s strategy, individual modules are not dependent on other modules, so each module can be deployed independently and (usually) scaled horizontally. 

When you think of Enterprise ERP as a federation of independent modules communicating via API’s (rather than a database), the problems with multi-vendor ERP systems fade because the system-of-record is no longer a massive dependency-generator that requires lockstep deployments. With a federated approach, business leaders can move fast and experiment with different systems as they become available, and still synchronize critical enterprise data with the rest of the company. In addition, similar processes in different parts of the enterprise can use different applications to meet their unique needs without the significant tailoring expense encountered when a single ERP suite is imposed on the entire enterprise.

What about Standardization?

Won’t separate ERP modules lead to different processes in different parts of the enterprise? Yes, certainly. But the question is – under what circumstances are standard processes important? In the days of manual back office processes, there was lot of labor-intensive work: drafting, accounting, phone calls, people moving paperwork from one desk to another. Standardization in this kind of operating environment made sense and could lead to significant efficiencies. But in a digitized world, the important thing is not uniformity; it is rapid and continuous improvement in each business area. Different processes for different problems in different contexts can be a very good thing.

Jeff Bezos agrees; he believes that the only path to serious scale is to have a lot of independent agents making their own decisions about the best way to do things. This belief was a key factor in the birth of Amazon Web Services, a $10 billion business that keeps on growing. Amazon began its journey away from a big back end by creating small, cross-functional teams with end-to-end responsibility for a service. These teams designed their own processes to fit their particular environment. Amazon then developed a software architecture and data center infrastructure that allowed these teams to operate and deploy independently. The rest is history.

In Conclusion

It is time for enterprise processes become federated instead of integrated. This is not a new path – embedded software has used a similar architecture for decades. Today, almost every successful internet-scale business has adopted some type of federated approach because it is the only way to scale beyond the limitations of the enterprise. 

As digitization brings back-office teams closer to consumers and providers, they must join with their front-office colleagues and form teams that are fully capable of designing and improving a process or a line of business. These “full stack” teams should be responsible for managing their own practices, technology and data, meeting industry standards for their particular areas. They should communicate with other areas of the enterprise on demand through well-defined interfaces. 

The good news is that you can gradually migrate to a federation from almost any starting point, including an enterprise-wide ERP system. Even better, as IT moves from enforcing compliance with the company’s ERP system to brokering interface contracts and ensuring data security, it becomes a business enabler rather than a bottleneck. And best of all, responsible full stack teams that solve their own problems will create attractive jobs for talented engineers and give business units control over their own digital destiny. 

Lean Software Development: The Backstory

We were in a conference room near the Waterfront in Cape Town. “I just lost a crown from one of my teeth.” my husband Tom declared just before I was scheduled to open the conference. Someone at our table responded, “You’re lucky, Cape Town has some of the best dentists in the world.” It didn’t feel very lucky; Cape Town was the first stop on a ten week trip to Africa, Europe, and Australia.

The situation was eerily familiar. A year earlier a chip had cracked off of my tooth as I ate a pizza in Lima, the first stop of a ten week trip to South America. I ate gingerly during the rest of the trip, worried that the tooth would crack further. Luckily I made it back home with no pain and little additional damage. Once there, it took three days to get a dentist appointment. The dentist made an impression of the gap in my tooth and fashioned a temporary crown. “This will have to last for a week or two,” she said. “If it falls out, just stick it back in and be more careful what you eat.” Luckily the temporary crown held, and ten days later a permanent crown arrived from the lab. Two weeks after we arrived home, my tooth was fixed.

We were scheduled to be in Cape Town for only two days. How was Tom going to get a crown replaced in two days? A small committee formed. Someone did a phone search; apparently the Waterfront was a good place to find dentists. A call was made. “You can go right now – the dental office is nearby. Do you want someone to walk you over?” As Tom headed out the door with an escort, I got ready for my presentation. Half way through the talk, I saw Tom return and signal that all was well.

“I lost a part of my tooth, not just the crown,” Tom told me after the talk. “I’m supposed to return at 3:30 this afternoon; I should have a new crown by the end of the day.” The dentist had a mini-lab in his office. Instead of making a temporary crown, he used a camera to take images of the broken tooth and adjacent teeth. The results were combined into a 3D model of the crown to which the dentist made a few adjustments. Then he selected a ceramic blank that matched the color of Tom’s teeth and put it in a milling machine. With the push of a button, instructions to make the crown were loaded into the machine. Cutters whirled and water squirted to keep the ceramic cool. Ten minutes later the crown was ready to cement in place. Ninety minutes after he arrived that afternoon and eight hours after the incident, Tom walked out of the dental office with a new permanent crown. It cost approximately the same amount as my crown had cost a year earlier.

Lean is about Flow Efficiency

The book This is Lean (Modig and Ahlström, 2013) describes “lean” as a relentless focus on efficiency – but not the kind of efficiency that cuts staff and money, nor the kind of efficiency that strives to keep every resource busy all of the time. In fact, a focus on resource efficiency will almost always destroy overall efficiency, the authors contend, because fully utilized machines (and people) create huge traffic jams, which end up creating a lot of extra work. Instead, Modig and Ahlström demonstrate that lean is about flow efficiency – that is, the efficiency with which a unit of work (a flow unit) moves through the system.

Consider our dental experience. It took two weeks for me get a new crown, but in truth, only an hour and a half of that time was needed to actually fix the tooth; the rest of the time was mostly spent waiting. My flow efficiency was 1.5÷336 (two weeks) = 0.45%. On the other hand, Tom’s tooth was replaced in eight hours – 42 times faster – giving him a flow efficiency of 1.5÷8 = 18.75%.

In my case, the dental system was focused on the efficiency of the lab’s milling machine – no doubt an expensive piece of equipment. But add up all of the extra costs: a cast of the crown for the lab, a temporary crown for me, two separate hour-long sessions with the dentist, plus all of the associated logistics – scheduling, shipping, tracking, etc. In Tom’s case, the dental system was focused on the speed with which it could fix his tooth – which was good for us, because a long wait for a crown was not an option. True, the milling machine in the dentist’s office sits idle much of each day. (The dentist said he has to replace two crowns a day to make it economically feasible.) But when you add up the waste of temporary crowns, the piles of casts waiting for a milling machine, and the significant cost of recovering from a mistake – an idle milling machine makes a lot of sense.

What does flow efficiency really mean? Assume you have a camera and efficiency means keeping the camera busy – always taking a picture of some value-adding action. Where do you aim your camera? In the case of resource efficiency, the camera is aimed at the resource – the milling machine – and keeping it busy is of the utmost importance. In the case of flow efficiency, the camera is on the flow unit – Tom – and work on replacing his crown is what counts.  The fundamental mental shift that lean requires is this: flow efficiency trumps resource efficiency almost all of the time.

Lean Product Development: The Predecessor

During the 1980’s Japanese cars were capturing market share at a rate that alarmed US automakers. In Boston, both MIT and Harvard Business School responded by launching extensive studies of the automotive industry. In 1990 the MIT research effort resulted in the now classic book The Machine that Changed the World: the Story of Lean Production (Womack et al., 1990), which gave us the term “lean.” A year later, Harvard Business School published Product Development Performance. (Clark and Fujimoto, 1991) and the popular book Developing Products in Half the Time (Smith and Reinertsen, 1991) was released. These two 1991 books are foundational references on what came to be called “lean product development,” although the term “lean” would not be associated with product development for another decade.

Clark and Fujimoto documented the fact that US and European volume automotive producers took three times as many engineering hours and 50% more time to develop a car compared to Japanese automakers, yet the Japanese cars had substantially higher quality and cost less to manufacture. Clearly the Japanese product development process produced better cars faster and at lower cost that typical western development practices of the time. Clark and Fujimoto noted that the distinguishing features of Japanese product development paralleled features found in Japanese automotive production. For example, Japanese product development focused on flow efficiency, reducing information inventory, and learning based on early and frequent feedback from downstream processes. By contrast, product development in western countries focused on resource efficiency, completing each phase of development before starting the next, and following the original plan with as little variation as possible.

In 1991 the University of Michigan began its Japan Technology Management Program. Over the next several years, faculty and associate members included Jeffrey Liker, Allen Ward, Durward Sobek, John Shook, and Mike Rother. This group has published numerous books and articles on lean thinking, lean manufacturing, and lean product development, including The Toyota Product Development System (Morgan and Liker, 2006), and Lean Product and Process Development (Ward, 2007). The second book summarizes the essence of lean product development this way:

1. Understand that knowledge creation is the essential work of product development.
2. Charter a team of responsible experts led by an entrepreneurial system designer.
3. Manage product development using the principles of cadence, flow, and pull.

It is important to recognize that even though lean product development is based on the same principles as lean production, the practices surrounding development are, quite frankly, not the same as those considered useful in production. In fact, transferring lean practices from manufacturing to development has led to some disastrous results. For example, lean production emphasizes reducing variation – exactly the wrong thing to do in product development. The western practice of following a plan and measuring variance from a plan is often justified by the slogan “Do it right the first time.” Unfortunately, this approach does not allow for learning; it confines designs to those conceived when the least amount of knowledge is available. A fundamental practice in lean product development is to create variation (not avoid it) in order to explore the impact of multiple approaches. (This is called set-based engineering.)

The critical thing to keep in mind is that knowledge creation is the essential work of product development. While lean production practices support learning about and improving the manufacturing process, their goal is to minimize variation in the product. This is not appropriate for product development, where variation is an essential element of the learning cycles that are the foundation of good product engineering. Thus instead of copying lean manufacturing practices, lean product development practices must evolve from a deep understanding of fundamental lean principles adapted to a development environment.

Lean Software Development: A Subset of Lean Product Development

In 1975, computers were large, expensive, and rare. Software for these large machines was developed in the IT departments of large companies and dealt largely with the logistics of running the company – payroll, order processing, inventory management, etc. But as mainframes morphed into minicomputers, personal computers, and microprocessors, it became practical to enhance products and services with software. Then the internet began to invade the world, and it eventually became the delivery mechanism for a large fraction of the software being developed today. As software moved from supporting business process to enabling smart products and becoming the essence services, software engineers moved from IT departments to line organizations where they joined product teams.

Today, most software development is not a stand-alone process, but rather a part of developing products or services. Thus lean software development might be considered a subset of lean product development; certainly the principles that underpin lean product development are the same principles that form the basis of lean software development.

Agile and Lean Software Development: 2000 - 2010

It’s hard to believe these days, but in the mid 1990’s, developing software was a slow and painful process found in the IT departments of large corporations. As the role of software expanded and software engineers moved into line organizations, reaction against the old methods grew. In 1999, Kent Beck proposed a radically new approach to software development in the book “Extreme Programming Explained” (Beck, 1999). In 2001 the Agile Manifesto (Beck et al., 2001) gave this new approach a name – “Agile.”

In 2003, the book Lean Software Development (Poppendieck, 2003) merged lean manufacturing principles with agile practices and the latest product development thinking, particularly from the book Managing the Design Factory (Reinertsen, 1997). Lean software development was presented as a set of principles that form a theoretical framework for developing and evolving agile practices:

1. Eliminate waste
2. Amplify learning
3. Decide as late as possible
4. Deliver as fast as possible
5. Empower the team
6. Build quality in
7. See the whole

Although the principles of lean software development are consistent with lean manufacturing and (especially) lean product development, the specific practices that emerged were tailored to a software environment and aimed at the flaws in the prevailing software development methodologies. One of the biggest flaws at the time was the practice of moving software sequentially through the typical stages of design, development, test, and deployment – with handovers of large inventories of information accumulating at each stage. This practice left testing and integration at the end of the development chain, so defects went undetected for weeks or months before they were discovered. Typical sequential processes reserved a third of a release cycle for testing, integration, and defect removal. The idea that it was possible to “build quality in” was not considered a practical concept for software.

To counter sequential processes and the long integration and defect removal phase, agile software development practices focused on fast feedback cycles in these areas:

1. Test-driven development: Start by writing tests (think of them as executable specifications) and then write the code to pass the tests. Put the tests into a test harness for ongoing code verification.
   
2. Continuous integration: Integrate small increments of code changes into the code base frequently – multiple times a day – and run the test harness to verify that the changes have not introduced errors.
   
3. Iterations: Develop working software in iterations of two-to four weeks; review the software at the end of each iteration and make appropriate adjustments.
   
4. Cross-functional teams: Development teams should include customer proxies and testers as well as developers to minimize handovers.

During its first decade, agile development moved from a radical idea to a mainstream practice. This was aided by the widespread adoption of Scrum, an agile methodology which institutionalized the third and fourth practices listed above, but unfortunately omitted the first two practices.

The Difference between Lean and Agile Software Development

When it replaced sequential development practices typical at the time, agile software development improved the software development process most of the time – in IT departments as well as product development organizations. However, the expected organizational benefits of agile often failed to materialize because agile focused on optimizing software development, which frequently was not the system constraint. Lean software development differed from agile in that it worked to optimize flow efficiency across the entire value stream “from concept to cash.” (Note the subtitle of the book Implementing Lean Software Development: From Concept to Cash (Poppendieck, 2006)). This end-to-end view was consistent with the work of Taiichi Ohno, who said:

“All we are doing is looking at the time line, from the moment the customer gives us an order to the point when we collect the cash. And we are reducing that time line by removing the non-value-added wastes.” (Ohno, 1988. p ix)

Lean software development came to focus on these areas:

1. Build the right thing: Understand and deliver real value to real customers. 
2. Build it fast: Dramatically reduce the lead time from customer need to delivered solution. 
3. Build the thing right: Guarantee quality and speed with automated testing, integration and deployment. 
4. Learn through feedback: Evolve the product design based on early and frequent end-to-end feedback.

Let’s take a look at each principle in more detail:

1. Understand and deliver real value to real customers.

A software development team working with a single customer proxy has one view of the customer interest, and often that view is not informed by technical experience or feedback from downstream processes (such as operations). A product team focused on solving real customer problems will continually integrate the knowledge of diverse team members, both upstream and downstream, to make sure the customer perspective is truly understood and effectively addressed. Clark and Fujimoto call this “integrated problem solving” and consider it an essential element of lean product development.

2. Dramatically reduce the lead time from customer need to delivered solution.

A focus on flow efficiency is the secret ingredient of lean software development. How long does it take for a team to deploy into production a single small change that solves a customer problem?  Typically it can take weeks or months – even when the actual work involved consumes only an hour. Why? Because subtle dependencies among various areas of the code make it probable that a small change will break other areas of the code; therefore it is necessary to deploy large batches of code as a package after extensive (usually manual) testing. In many ways the decade of 2000-2010 was dedicated to finding ways to break dependencies, automate the provisioning and testing processes, and thus allow rapid independent deployment of small batches of code.

3. Guarantee quality and speed with automated testing, integration and deployment.

It was exciting to watch the expansion of test-driven development and continuous integration during the decade of 2000-2010. First these two critical practices were applied at the team level – developers wrote unit tests (which were actually technical specifications) and integrated them immediately into their branch of the code. Test-driven development expanded to writing executable product specifications in an incremental manner, which moved testers to the front of the process. This proved more difficult than automated unit testing, and precipitated a shift toward testing modules and their interactions rather than end-to-end testing. Once the product behavior could be tested automatically, code could be integrated into the overall system much more frequently during the development process – preferably daily – so software engineers could get rapid feedback on their work.

Next the operations people got involved and automated the provisioning of environments for development, testing, and deployment. Finally teams (which now included operations) could automate the entire specification, development, test, and deployment processes – creating an automated deployment pipeline. There was initial fear that more rapid deployment would cause more frequent failure, but exactly the opposite happened. Automated testing and frequent deployment of small changes meant that risk was limited. When errors did occur, detection and recovery was much faster and easier, and the team became a lot better at it. Far from increasing risk, it is now known that deploying code frequently in small batches is best way to reduce risk and increase the stability of large complex code bases.

4. Evolve the product design based on early and frequent end-to-end feedback.

To cap these remarkable advancements, once product teams could deploy multiple times per day they began to close the loop with customers. Through canary releases, A/B testing, and other techniques, product teams learned from real customers which product ideas worked and how to fine tune their offerings for better business results.

When these four principles guided software development in product organizations, significant business-wide benefits were achieved. However, IT departments found it difficult to adopt the principles because they required changes that lay beyond span of control of most IT organizations.

Lean Software Development: 2010 - 2015

2010 saw the publication of two significant books about lean software development. David Anderson’s book Kanban (Anderson, 2010) presented a powerful visual method for managing and limiting work-in-process (WIP).  Just at the time when two week iterations began to feel slow, Kanban gave teams a way to increase flow efficiency while providing situational awareness across the value stream. Jez Humble and Dave Farley’s book Continuous Delivery (Humble and Farley, 2010) walked readers through the steps necessary to achieve automated testing, integration and deployment, making daily deployment practical for many organizations. A year later, Erik Reis’s book The Lean Startup (Reis, 2011) showed how to use the rapid feedback loop created by continuous delivery to run experiments with real customers and confirm the validity of product ideas before incurring the expense of implementation.

Over the next few years, the ideas in these books became mainstream and the limitations of agile software development (software-only perspective and iteration-based delivery) were gradually expanded to include a wider part of the value stream and a more rapid flow. A grassroots movement called DevOps worked to make automated provision-code-build-test-deployment pipelines practical. Cloud computing arrived, providing easy and automated provisioning of environments. Cloud elements (virtual machines, containers), services (storage, analysis, etc.) and architectures (microservices) made it possible for small services and applications to be easily and rapidly deployed. Improved testing techniques (simulations, contract assertions) have made error-free deployments the norm.

The State of Lean Software Development in 2015

Today’s successful internet companies have learned how to optimize software development over the entire value stream.  They create full stack teams that are expected to understand the consumer problem, deal effectively with tough engineering issues, try multiple solutions until the data shows which one works best, and maintain responsibility for improving the solution over time. Large companies with legacy systems have begun to take notice, but they struggle with moving from where they are to the world of thriving internet companies.

Lean principles are a big help for organizations that want to move from old development techniques to modern software approaches. For example, (Calçado, 2015) shows how classic lean tools – Value Stream Mapping and problem solving with Five Whys – were used to increase flow efficiency at Soundcloud, leading over time to a microservices architecture. In fact, focusing on flow efficiency is an excellent way for an organization to discover the most effective path to a modern technology stack and development approach.

For traditional software development, flow efficiency is typically lower than 10%; agile practices usually bring it up to 30 or 40%. But in thriving internet companies, flow efficiency approaches 70% and is often quite a bit higher. Low flow efficiencies are caused by friction – in the form of batching, queueing, handovers, delayed discovery of defects, as well as misunderstanding of consumer problems and changes in those problems during long resolution times. Improving flow efficiency involves identifying and removing the biggest sources of friction from the development process.

Modern software development practices – the ones used by successful internet companies – address the friction in software development in a very particular way. The companies start by looking for the root causes of friction, which usually turn out to be 1) misunderstanding of the customer problem, 2) dependencies in the code base and 3) information and time lost during handovers and multitasking. Therefore they focus on three areas: 1) understanding the consumer journey, 2) architecture and automation to expose and reduce dependencies, and 3) team structures and responsibilities. Today (2015), lean development in software usually focuses on these three areas as the primary way to increase efficiency, assure quality, and improve responsiveness in software-intensive systems.

1. Understand the Customer Journey.

Software-intensive products create a two-way path between companies and their consumers. A wealth of data exists about how products are used, how consumers react to a product’s capabilities, opportunities to improve the product, and so on. Gathering this data and analyzing it has become an essential capability for companies far beyond the internet world: car manufactures, mining equipment companies, retail stores and many others gather and analyze “Big Data” to gain insights into consumer behavior. The ability of companies to understand their consumers through data has changed the way products are developed. (Porter, 2015) No longer do product managers (or representatives from “the business”) develop a roadmap and give a prioritized list of desired features to an engineering team. Instead, data scientists work with product teams to identify themes to be explored. Then the product teams identify consumer problems surrounding the theme and experiment with a range of solutions. Using rapid deployment and feedback capabilities, the product team continually enhances the product, measuring its success by business improvements, not feature completion.

2. Architecture and Automation.

Many internet companies, including Amazon, Netflix, eBay, realestate.com.au, Forward, Twitter, PayPal, Gilt, Bluemix, Soundcloud, The Guardian, and even the UK Government Digital Service have evolved from monolithic architectures to microservices. They found that certain areas of their offerings need constant updating to deal with a large influx of customers or rapid changes in the marketplace. To meet this need, relatively small services are assigned to small teams which then split their services off from the main code base in such a way that each service can be deployed independently. A service team is responsible for changing and deploying the service as often as necessary (usually very frequently), while insuring that the changes do not break any upstream or downstream services. This assurance is provided by sophisticated automated testing techniques as well as automated incremental deployment.

Other internet companies, including Google and Facebook, have maintained existing architectures but developed sophisticated deployment pipelines that automatically send each small code change through a series of automated tests with automatic error handling. The deployment pipeline culminates in safe deployments which occur at very frequent intervals; the more frequent the deployment, the easier it is to isolate problems and determine their cause. In addition, these automation tools often contain dependency maps so that feedback on failures can be sent directly to the responsible engineers and offending code can be automatically reverted (taken out of the pipeline in a safe manner).

These architectural structures and automation tools are a key element in a development approach that uses Big Data combined with extremely rapid feedback to improve the consumer journey and solve consumer problems. They are most commonly found in internet companies, but are being used in many others, including organizations that develop embedded software. (See case study, below.)

3. Team Structures and Responsibilities.

When consumer empathy, data analytics and very rapid feedback are combined, there is one more point of friction that can easily reduce flow efficiency. If an organization has not delegated responsibility for product decisions to the team involved in the rapid feedback loop, the benefits of this approach are lost. In order for such feedback loops to work, teams with a full stack of capabilities must be given responsibility to make decisions and implement immediate changes based on the data they collect. Typically such teams include people with product, design, data, technology, quality, and operations backgrounds. They are responsible for a improving set of business metrics rather than delivering a set of features. An example of this would be the UK Government Digital Service (GDS), where teams are responsible for delivering improvements in four key areas: cost per transaction, user satisfaction, transaction completion rate, and digital take-up.

It is interesting to note that UK laws makes it difficult to base contracts on such metrics, so GDS staffs internal teams with designers and software engineers and makes them responsible for the metrics. Following this logic to its conclusion, the typical approach of IT departments – contracting with their business colleagues to deliver a pre-specified set of features – is incompatible with full stack teams responsible for business metrics. In fact, it is rare to find separate IT departments in companies founded after the mid 1990’s (which includes virtually all internet companies). Instead, these newer companies place their software engineers in line organizations, reducing the friction of handovers between organizations.

In older organizations, IT departments often find it difficult to adopt modern software development approaches because they have inherited monolithic code bases intertwined with deep dependencies that introduce devious errors and thwart independent deployment of small changes. One major source of friction is the corporate database, once considered essential as the single source of truth about the business, but now under attack as a massive dependency generator. Another source of friction are outsourced applications, where even small changes are difficult and knowledge of how to make them no longer resides in the company. But perhaps the biggest source of friction in IT departments is the distance between their technical people and the company’s customers. Because most IT departments view their colleagues in line businesses as their customers, the technical people in IT lack a direct line of sight to the real customers of the company. Therefore insightful trade-offs and innovative solutions struggle to emerge.

The Future of Lean Software Development

The world-wide software engineering community has developed a culture of sharing innovative ideas, in stark contrast to the more common practice of keeping intellectual property and internally developed tools proprietary. The rapid growth of large, reliable, secure software systems can be directly linked to the fact that software engineers routinely contribute to and build upon the work of their world-wide colleagues through open source projects and repositories like GitHub. This reflects the longstanding practices of the academic world but is strikingly unique in the commercial world. Because of this intense industry-wide knowledge sharing, methods and tools for building highly reliable complex software systems have advanced extraordinarily quickly and are widely available.

As long as the software community continues to leverage its knowledge-sharing culture it will continue to grow rapidly, because sophisticated solutions to seemingly intractable problems eventually emerge when many minds are focused on the problem. The companies that will benefit the most from these advances are the ones that not only track new techniques as they are being developed, but also contribute their own ideas to the knowledge pool.

As microstructured architectures and automated deployment pipelines become common, more companies will adopt these practices, some earlier and some later, depending on their competitive situation. The most successful software companies will continue to focus like a laser on delighting customers, improving the flow of value, and reducing risks. They will develop (and release as open source) an increasingly sophisticated set of tools that make software development easier, faster, and more robust. Thus a decade from now there will be significant improvements in the way software is developed and deployed. The Lean principles of understanding value, increasing flow efficiency, eliminating errors, and learning through feedback will continue to drive the evolution, but the term “lean” will disappear as it becomes “the way things are done.”

— Case Study —

Hewlett Packard LaserJet Firmware 

The HP LaserJet firmware department had been the bottleneck of the LaserJet product line for a couple of decades, but by 2008 the situation had turned desperate. Software was increasingly important for differentiating the printer line, but the firmware department simply could not keep up with the demand for more features. Department leaders tried to spend their way out of the problem, but more than doubling the number of engineers did little to help. So they decided to engineer a solution to the problem by reengineering the development process.

The starting point was to quantify exactly where all the engineers’ time was going.  Fully half of the time went to updating existing LaserJet printers or porting code between different branches that supported different versions the product. A quarter of the time went to manual builds and manual testing, yet despite this investment, developers had to wait for days or weeks after they made a change to find out if it worked. Another twenty percent of the time went to planning how to use the five percent of time that was left to do any new work. The reengineered process would have to radically reduce the effort needed to maintain existing firmware, while seriously streamlining the build and test process. The planning process could also use some rethinking.

It’s not unusual to see a technical group use the fact that they inherited a messy legacy code base as an excuse to avoid change. Not in this case. As impossible as it seemed, a new architecture was proposed and implemented that allowed all printers – past, present and even future – to operate off of the same code branch, determining printer-specific capabilities dynamically instead of having them embedded in the firmware. Of course this required a massive change, but the department tackled one monthly goal after another and gradually implemented the new architecture. But changing the architecture would not solve the problem if the build and test process remained slow and cumbersome, so the engineers methodically implemented techniques to streamline that process. In the end, a full regression test – which used to take six weeks – was routinely run overnight. Yes, this involved a large amount of hardware, simulation and emulation, and yes it was expensive. But it paid for itself many times over.

During the recession of 2008 the firmware department was required to return to its previous staffing levels. Despite a 50% headcount reduction, there was a 70% reduction in cost per printer program once the new architecture and automated provisioning system were in place in 2011. At that point there was a single code branch and twenty percent of engineering time was spend maintaining the branch and supporting existing products. Thirty percent of engineering time was spent on the continuous delivery infrastructure, including build and test automation. Wasted planning time was reclaimed by delaying speculative decisions and making choices based on short feedback loops. And there was something to plan for, because over forty percent of the engineering time was available for innovation.

This multi-year transition was neither easy nor cheap, but it absolutely was worth the effort. If you would like more detail, see (Gruver et al, 2013).

A more recent case study of how the software company Paddy Power moved to continuous delivery can be found in (Chen, 2015). In this case study the benefits of continuous delivery are listed: improved customer satisfaction, accelerated time to market, building the right product, improved product quality, reliable releases, and improved productivity and efficiency. There is really no downside to continuous delivery. Of course it is a challenging engineering problem that can require significant architectural modifications to existing code bases as well as sophisticated pipeline automation. But technically, continuous delivery is no more difficult than other problems software engineers struggle with every day. The real stumbling block is the change in organizational structure and mindset required to achieve serious improvements in flow efficiency.

— End Case Study —

Credit

This essay is a preprint of the author’s original manuscript of a chapter to be published in Netland and Powell (eds) (2016) "Routledge Companion to Lean Management"

References

Anderson, David. Kanban, Blue Hole Press, 2010

Beck, Kent. Extreme Programming Explained, Addison-Wesley, 2000

Beck, Kent et al. Manifesto for Agile Software Development, http://agilemanifesto.org/, 2001

Calçado, Phil. How we ended up with microservices. http://philcalcado.com/2015/09/08/how_we_ended_up_with_microservices.html

Chen, Lianping. "Continuous Delivery: Huge Benefits but Challenges Too" IEEE Software 32 (2). 50-54. 2015

Clark, Kim B. and Takahiro Fujimoto. Product Development Performance, Harvard Business School Press, 1991

Gruver, Gary, Mike Young, and Pat Fulghum. A Practical Approach to Large-Scale Agile Development, Pearson Education, 2013

Humble, Jez and David Farley. Continuous Delivery, Addison-Wesley Professional, 2010

Modig, Niklas, and Par Ahlstrom. This is Lean, Stockholm: Rheologica Publishing, 2012

Morgan, James M and and Jeffrey K Liker. The Toyota Product Development System, Productivity Press, 2006

Ohno, Taiichi. Toyota Production System, English, Productivity, Inc. 1988, published in Japanese in 1978

Poppendieck, Mary and Tom. Lean Software Development, Addison Wesley, 2003

Poppendieck, Mary and Tom. Implementing Lean Software Development, Addison Wesley, 2006

Porter, Michael E. and James E. Heppelmann. How Smart, Connected Products are Transforming Companies, Harvard Business Review 93 (10), 97-112, 2015

Reinertsen, Donald G. Managing the Design Factory, The Free Press, 1997

Ries, Eric. The Lean Startup, Crown Business, 2011

Smith, Preston G. and Donald G. Reinertsen. Developing Products in Half the Time, Van Nostrand Reinhold/co Wiley, 1991

Ward, Allen. Lean Product and Process Development, Lean Enterprise Institute, 2007

Womack, James P., Daniel T. Jones, and Daniel Roos. The Machine That Changed the World; the Story of Lean Production, Rawson & Associates, 1990

Artist Island

Once there was an island called Artist Island. The people on this island made a very good living by gathering stones from Stone Island, forming them into stunning jewels and selling the jewelry to people in the Land of Festivals. At first, people from Artist Island visited the Land of Festivals to see what kind of jewelry would be popular and then scouted out the right kinds of stones on Stone Island. But that was not a very efficient use of the time of these excellent artisans, so after a while the people from Stone Island were asked to bring their stones to Artist Island. Later, to further improve efficiency, the Stone People were asked to deliver the finished jewelry to the Land of Festivals. Since the Stone People were now the ones in contact with the Festival People, they became responsible for ordering the jewelry as well.

The people of Artist Island continued their search for efficiency. It took a long time to shape a pile of stones into a pile of jewels, and by the time a pile was done, the Stone People complained that the designs were out of date. So the artisans learned about Lean and decided to reduce work-in-progress. They worked on stone piles for no longer than two weeks, and brought smaller piles of jewels to the boat dock for shipment to the Land of Festivals.  However, the transport boats were still quite large and took some time to fill up, so it still took a long time for jewelry to get to the Festival People, and the designs were still out of date.

Then the Stone People came up with an idea. If they invested in smaller boats, they could deliver jewelry to the Land of Festivals in much smaller batches, shortly after it was made. Gradually they switched from large boats to small boats, and as they did, they started using the same small boats to deliver stones to Artist Island. Now a stone could be dug out of a mine, taken to Artist Island, formed into a jewel and delivered to the Land of Festivals in about a month. The jewelry designs were much more up-to-date, and sales improved.

But there was still a lot of unsold jewelry in warehouses in the Land of Festivals, because sometimes the jewels had flaws and sometimes the artisans didn't understand what the Stone People were asking them to make. Quite often the jewelry was boring because the Stone People weren't aware of the many marvelous kinds of jewels that were possible, so they didn't order many interesting designs.

The Earthquake
About this time an earthquake shifted the rocks holding water in the lake, and the water level receded.  Artist Island became a peninsula connected to the mainland. Some enterprising artisans decided to walk to the mainland and talk to the Festival People. They discovered that the Festival People didn't really like their jewelry designs very much, and so the artisans went back home and produced some new designs. After a few trips, they learned more and more about the festivals, their timing, their themes, and the kind of jewels that would be best for each one. They began to produce jewelry specially designed for each festival, and sales soared.

Of course, the artisans weren't using their time quite so efficiently anymore, because some of the best craftspeople spent part of their valuable time walking over to the mainland and talking to the Festival People. But then again, contact with customers energized the artisans, and they brought their enthusiasm back to Artist Island. The new designs were wildly popular, so none of them ended up stored in a warehouse. Thus the people of Artist Island were able to sell more jewelry and charge higher prices than before.

The artisans found that the Festival People were looking for unique jewels, so they asked the Stone People to look for new kinds of stones. But the new stones that the Stone People brought were not suitable for forming into jewels, and the artisans began to wish they had kept a few of their old boats so they could go and look for new stones themselves. One day some of the artisans went exploring their new peninsula and discovered a shallow sandbar connecting Artist Island to Stone Island. So they were able to wade over to Stone Island to help the Stone People look for better stones.

Because they knew what they were looking for, the artisans soon discovered new types of stones that could be easily formed into jewels. Actually, the new stones were much bigger than the previous ones -- they barely fit in the small boats -- so the Stone People had ignored them. The large stones gave the Artist Island people a novel idea: perhaps they could make cups and bowls from the new stones. Several different kinds of stones were brought to Artist Island and artisans eagerly tried their hand at making household items. This turned out to be easier than the intricate work of making jewelry, so even apprentice artisans were able to shape the new stones. The Festival People loved the new dishes and bought many pieces in addition to the jewels they always enjoyed.

Of course, the artisans weren't using their time quite so efficiently now that some of the best craftspeople were working with the Stone People as well as the Festival People. But then again, dishes involved less intricate work so more items could be produced and it was much easier to avoid flaws. Furthermore, the Festival People were eager to buy every single item the artisans could produce. No longer was finished work gathering dust in warehouses. Thus the artisans were able to make and sell many more products than before. Finally, since the household items were considered a necessity, business remained good even during tough economic times.

The New Landscape
Let’s take a tour of Artist Island a few years after the waters receded and turned it into a peninsula. The former island has three different kinds of artisans. First of all there are the enterprising artisans who learned to empathize with customers and look for new stones to shape in novel ways to solve customer problems. They are deeply engaged in their work and have created tight feedback loops so they can continue to develop products that customers love and bring innovative new stones to the market.

This group of artisans has revised the definition of efficiency.[1] They don’t worry too much about resource efficiency -- that is, keeping every artisan busy. They focus on flow efficiency -- that is, keeping each stone moving from Stone Island all the way to customers in the Land of Festivals with as little delay as possible. They have found that with greater flow efficiency, they get higher quality, more rapid customer feedback, and thus they are more likely to make products that delight customers. The enterprising artisans are doing very well.

But not everyone on Artist Island noticed that the waters receded, or if they noticed, they were not eager to abandon their comfortable routines. The traditional artisans believe in resource efficiency -- that is, making the most efficient use of their valuable time. So they continue to receive boatloads of stones from the Stone People, form them into the kind of jewels they are asked to make, and send the jewelry to be sold in the Land of Festivals. It’s not their problem if the Stone People order the wrong jewels, or if the boats are so large that their work is out of date by the time it reaches the Land of Festivals, or if half of their jewelry ends up in warehouses, unused by the Festival People. Their job is to deliver what they are asked for in a timely manner, and to continually reduce their costs.

Of course, the work is not very challenging and it’s difficult to get enthusiastic about making piles of jewelry that no one is likely to use. Because of this, many traditional artisans are leaving to join the enterprising artisans, attracted by the opportunity to think for themselves, the challenge of improving their artistic skills, and the satisfaction of seeing their work appreciated by the Festival People.

There is a third area of Artist Island -- one that wasn't mentioned earlier -- an area that has been around ever since stones began to be used for practical items. Here we find the parts-makers, artisans who form stones into parts for automobiles and airplanes and medical devices and control systems and things like that. They make up almost half of the population of Artist Island. The parts-makers work with vehicle and device designers to make sure their parts fit and operate properly. Recently they have learned a few things from the enterprising artisans, such as focusing on flow efficiency (moving stones through their work area without delay), understanding the needs of their customers (both the device designers as well as device customers), and constantly looking for new stones that can better meet these needs.

The Future
As the years go by, the enterprising artisans will move to the mainland to work side-by-side with the Festival People. The parts-makers will also migrate to the mainland and join forces with the device designers. All that will be left on the former island are the cost centers housing traditional artisans, but even these will gradually shrink and eventually disappear. Because in the end, while the talent of the artisans will remain essential, Artist Island will not matter anymore.

Navigating the New Landscape
We have spent a lot of time over the past decade working to make life better on Artist Island (a.k.a. Software Development Land). We promoted Lean principles such as small batches and steady flow and quality at the source. But over the past few years we have watched the waters recede and marveled as the island turned into a peninsula. The best and brightest of the artisans have abandoned the boat system and learned to talk directly with customers, work as partners with other disciplines, and seek out new approaches to solving problems.

We have written three books about Artist Island, but we couldn't write a fourth, because the island has largely disappeared. In its place is a new landscape, one in which integrated product teams are expected to ask the right questions, solve the right problems, and deliver solutions that customers love. So we wrote our fourth book -- The Lean Mindset: Ask the Right Questions -- about thriving in the new landscape, a land without islands, a land that doesn't have quite enough artisans, a land that’s full of endless possibilities.
_____________________________
Footnote:
[1] Thanks to Niklas Modig and Pär Åhlström, for introducing us to these two viewpoints on efficiency. See their excellent book -- This Is Lean: Resolving the Efficiency Paradox.