Sunday, March 17, 2002

Is Agile Software Development Sustainable?

Agile software development practices are often criticized as being suitable only for small, co-located teams of experts working on modest sized projects. If agile development is truly limited to these perceived boundaries, then it is probably not sustainable. Software development practices must address large projects, multiple geographies, and a general population of developers if they are to become the basis of a thriving new paradigm.

On the other hand, in The Innovator’s Dilemma, [HarperBusiness, 2000], Clayton Christensen notes that all disruptive technologies start by addressing a market which is considered small and “down-market” from existing technologies. So if agile practices are a “disruptive technology” compared to traditional software development processes, then it would be quite in character for them to start by addressing small systems. The questions is, can agile processes grow to address the needs of large systems?

The answer lies in recognizing that the needs of large systems are changing in a way that is best addressed by agile practices. Over the last decade, corporations have tended to use commercially available systems to address more and more information needs. Initially, these systems tended to be monolithic and proprietary, but this is changing. Corporations have found that any monolithic system, even one which is commercially supplied, rapidly becomes a legacy system as new technologies combine with mergers to obsolete any system that cannot adapt to change.

The result has been a tendency for developers of large systems to prefer the use of commercially supplied components, from API’s to Web Services, for a sizeable portion of any application. In Building Systems from Commercial Components, [Addison-Wesley, 2001] Kurt Wallnau and coauthors point out that large systems cannot be built from software components using traditional development processes. The marketplace in which component vendors compete dictates that components are complex, they change frequently, and customers pretty much have to take whatever is offered.

System development in the component marketplace is distinctly different from traditional software development. To begin with, one does not start by capturing user requirements, but by understanding the capabilities of available components. The challenge is to bring the user requirements into line with available software, rather than the other way around. System architecture is generally dictated by the available components and the key architectural task is to create an ‘ensemble’ of components that work well together. Finally, designing a system to be able to deal with unpredictable change is a fundamental skill when working with commercial components. If nothing else is certain, the fact that these components will constantly change can be guaranteed.

The bottom line is that the problems that used to be addressed by traditional software processes have changed, and those processes are no longer up to the task of addressing large project development. Meanwhile, the agile practices being honed in small projects are just the ones needed in the a large project environment which deals with legacy systems and commercial components. So don’t be surprised to see agile practices move up-market, as disruptive technologies always do, and take over the large projects as well.

A quote from a version of Chapter 1 of Building Systems from Commercial Components, [Addison-Wesley, 2001] by Kurt Wallnau, Scott Hissam, and Robert Seacord.
Whatever position one takes regarding the CMM versus ISO-9000 one thing is clear: these software management standards have, for over a decade, established the context for improving the practice of software development. Indeed, to many people software engineering and software process are one and the same thing. An entire industry has emerged to support the adoption of CMM or ISO-9000 models, and process improvement incentives have played a dominant role in defining roles and behavior within software development organizations. The resulting roles and behaviors constitute what we refer to as the process regime.

The process regime… established roles and behaviors rooted in a manufacturing metaphor, where software processes are analogous to manufacturing processes, programmers are analogous to assembly-line workers, and the ultimate product is lines of code. When viewed in terms of software manufacturing, improvements in software engineering practice are equated with process improvement, which itself is centered on improving programmer productivity and reducing product defects. Indeed, the manufacturing metaphor is so strong that the term software factory is still used to denote the ideal software development organization….

Today it is inconceivable to contemplate building enterprise systems without a substantial amount of the functionality of the system provided by commercial software components such as operating systems, databases, message brokers, Web browsers and servers, spreadsheets, decision aids, transaction monitors, report writers, and system managers.

As many organizations are discovering, the traditional software factory is ill equipped to build systems that are dominated by commercial software components. The stock and trade of the software factory—control over production variables to achieve predictability and then gradual improvement in quality and productivity—is no longer possible. The software engineer who deals with component-based systems no longer has complete control over how a system is partitioned, the interfaces are between these partitions, or how threads of control are passed or shared among these partitions. Traditional software development processes espoused by the process regime and software factory that assume control over these variables are no longer valid. The process regime has been overthrown….

Screen Beans Art, © A Bit Better Corporation