During a chilly February in 2001, 17 eminent software developers met up at the Snowbird ski resort in Utah. But they weren’t there for the 125-passenger aerial tram, the 89 different ski runs or the legendary powder. They were on an agile software mission.
One of those in attendance, Robert Martin (a.k.a Uncle Bob), described the hook-up in Utah as one of the only successful meetings he had ever been to. What this group of software thinkers came up with is what they called the Manifesto for Agile Software Development. It turned many of the accepted practices of software development on their head. But is it still as relevant today as it was then, now that we are in an era featuring megatrends like cloud computing, big data and social networking?
Agile Manifesto
One of the most impressive things about the Agile Manifesto was its brevity, given the complexity of its subject matter. The Agile Manifesto reads, in its entirety, as follows:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
– Individuals and interactions over processes and tools
– Working software over comprehensive documentation
– Customer collaboration over contract negotiation
– Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
The meanings of the manifesto items on the left within the agile software development context are described below:
– Individuals and Interactions – in agile development, self-organization and motivation are important, as are interactions like co-location and pair programming.
– Working software – working software will be more useful and welcome than just presenting documents to clients in meetings.
– Customer collaboration – requirements cannot be fully collected at the beginning of the software development cycle, therefore continuous customer or stakeholder involvement is very important.
– Responding to change – agile development is focused on quick responses to change and continuous development.
Twelve principles underlie the Agile Manifesto, including:
– Customer satisfaction by rapid delivery of useful software
– Welcome changing requirements, even late in development
– Working software is delivered frequently (weeks rather than months)
– Working software is the principal measure of progress
– Sustainable development, able to maintain a constant pace
– Close, daily co-operation between business people and developers
– Face-to-face conversation is the best form of communication (co-location)
– Projects are built around motivated individuals, who should be trusted
– Continuous attention to technical excellence and good design
– Simplicity- The art of maximizing the amount of work not done – is essential
– Self-organizing teams
– Regular adaptation to changing circumstances
On a recent visit to London, I met up with one of the authors of the Manifesto, Jim Highsmith. Highsmith is also now an Executive Consultant at Thoughtworks, a global IT consultancy. I asked how the gathering at Snowbird ski resort came about. "I’d started writing a book on adaptive in the mid-Nineties as an independent, and I wasn’t particularly wired into the OO thing [object oriented]. Kent [Beck] was doing his thing in XP [Extreme Programming], and while I was looking more at the management side of software development he was looking at the development track; we had started exchanging emails.
"Myself and Alistair Cockburn were working on something that we called ‘light’ methodologies; in fact there were a number of us out there doing our own thing in isolation. Alistair hadn’t liked it being called ‘lightweight’ as it happens, and I think it was Bob Martin [Uncle Bob] who had the idea of us all meeting up in Utah," Highsmith continues. "We got all the Manifesto guys together and came up with the four values of the Manifesto and 12 principles. It evolved a little bit more after a series of emails. We thought it had the potential to expand but we didn’t anticipate it would take off how it did."
Yet the Agile Manifesto authors’ views on a more lightweight approach to software development at organisations – favouring as it did working software over over-zealous or over-complicated documentation before development gets underway – has not been without its critics. One common criticism of agile software development methods is that it is developer-centric rather than user-centric. Agile software development focuses on processes for getting requirements and developing code and does not focus on product design. Mike Gualtieri, principal analyst of agile software development at Forrester Research, published a widely read criticism stating that software developers are not coders, but experience creators.
Searching for experience
In his blog, Gualtieri described agile software as a cop-out, and here’s why:
"Using ‘working software as the measure of progress’ is narcissistic," Gualtieri wrote. "Rush to write code is oft the translation of this misguided principle. Software developers want to be measured on what they know — code — instead of what they don’t know: the domain, business requirements, and the user. The idea is that you’ve got nothing if you don’t have working software. The cop-out is thinking that code is the measure of everything. Software is always part of a bigger picture. It works in conjunction directly or indirectly with other software, technologies, the environment, business processes, and people to create customer experience and/or improve operational efficiency. Like most of the Agile principles, this one is developer-centric rather than user-centric."
Other critics have said agile methodologies can also be inefficient in large organisations and certain types of projects. Agile methods seem best for developmental and non-sequential projects. Many organisations believe that agile methodologies are too extreme, and adopt a hybrid approach that mixes elements of agile and plan-driven approaches.
I put these criticisms to Highsmith. "I think what can happen is people get stuck at ‘agile 101’," he says. "Yes there are some basic rules that state you should do this and not this, and it’s natural to start with the basics. But you need to go beyond the basics. I think there is a difference between ‘doing agile’, for example adopting some stand-up meetings, and actually being agile in your methods.
"The other thing I see happening is that the development team has convinced management to be agile, but management isn’t really on board with it or encouraging it. It’s almost like an organisation’s antibodies come out and attach the agile team. If you have a CIO or VP of engineering saying we are going to be agile it’s very different; it’s being driven from the top."
From humble beginnings…
Highsmith says agile was launched and then grew from people passionate about software development, but that the push was largely bottom-up. "To expand agile in the typical organisation you need to have both," he says. "You need bottom-up push but also top-down support. Agile may mostly address software development but studies also find that the world is a hectic place, and CEOs say agility is a major factor. Is it critical to be efficient and responsive? Firms may say, ‘agile is good, let’s do it everywhere’; or they may say, ‘agile is good in some areas, let’s drive it depending on the business need.’"
Many organisations have experienced success with agile methods. One of the early studies reporting gains in quality, productivity, and business satisfaction by using Agile methods was a survey conducted by Shine Technologies from November 2002 to January 2003.
A similar survey conducted in 2006 by Scott Ambler, the Practice Leader for Agile Development with IBM Rational’s Methods Group reported similar benefits.
In a survey conducted by VersionOne (a provider of software for planning and tracking agile software development projects) in 2008, 55% of respondents answered that agile methods had been successful in 90-100% of cases.
Martin Cheesebrough, CTO at CRM, XRM and software development firm Digiterre, told me in a recent interview that the firm believes strongly in agile. "Agile means you can show what you have built so far and get on and give feedback, which helps to avoid mistakes from misunderstandings, or going down a blind alley. We also use pair programming a lot because two heads are often better than one. But a lot of our method is about pragmatism and common sense."
But others remain unconvinced. Forrester’s Gualtieri says, "The lack of empirical evidence for the benefits of Agile methods is telling." Not only that, but he has a serious issue with one of the main underlying principles of Agile, namely the Manifesto’s view that, "continuous customer or stakeholder involvement is very important".
According to Gualtieri, "Business people and developers working together daily is lazy. This principle is a capitulation by developers who say it is impossible to understand the true requirements. Developers would much rather stick with what they know: typing class definitions, if-then-else statements, and looping constructs into glorified text editors.
"The absurd Agile solution to getting the requirements right is to meet with the business every day (or frequently). Often the business people who are involved are not the same as the users, and they are not domain experts or design experts. Seriously, can your best people be available to help developers do their job?"
There are other development methodologies, of course. Before agile or lightweight methodologies, many companies used what they described as waterfall approaches – generally considered more rigorous and more document-focused than agile. The waterfall model is a sequential design process, in which progress is seen as flowing steadily downwards (like a waterfall) through the phases of Conception, Initiation, Analysis, Design, Construction, Testing, Production/Implementation and Maintenance.
But according to Highsmith, Waterfall is a disaster. "Waterfall was a mistake in the first place!" he says. "Walker Royce wrote a paper saying, ‘don’t do this’, and everybody saw this diagram of a waterfall model and said, ‘this is how you do it!’" Royce had presented the diagram as a depiction of a flawed, non-working model.
But even Highsmith concedes that the Waterfall method has been known to work in some situations. "Obviously there have been a lot of Waterfall projects that have delivered a lot of value. You can’t deny that it can work," he says. "NASA has such high reliability requirements that testing and validation just must play a big part. On the other hand there are medical device companies that also need a high degree of control and they have successfully used agile."
What about the so-called Three Amigos — James Rumbaugh, Grady Booch and Ivar Jacobson, developers of Unified Modeling Language and the Rational Unified Process or RUP — did they have the right idea? "RUP came out at a time when formalization was in vogue," Highsmith says. "In the 1990s, with things like business process re-engineering; it was the idea if you can get the process down you can plug and play. It’s probably not what Grady [Booch] intended but it got lost in a more traditional methodology. You start with a set of things and you add more as you see them. But there is an incentive to be safe and risk averse; to sort of say, ‘you can’t fault me because I can show all this documentation if it goes south’."
And Kent Beck’s Extreme Programming [XP], which advocates frequent releases in short development cycles? "I talked to Kent Beck early on and I felt the term XP put people off," Highsmith says. "What if you had called it Moderate Programming? XP became about pushing the dials to 10, and sometimes it broke. For many organisations a 6 or 7 is better. But sometimes XP [which is a form of agile] has been misinterpreted.
"It’s not about documentation, it’s about collaboration. If you simply substitute documentation for collaboration you may have a problem, but if you document your collaboration, that might be fine. Indeed if done correctly agile is much more disciplined than traditional waterfall teams. That was often very formal but not very disciplined," he adds.
Digiterre’s Cheesbrough seems to agree with this view, saying, "Some clients want more of a waterfall approach and we can fit in with that. Agile doesn’t say never do documentation, it says whatever you do make sure you are adding value. Also if a client wants a fixed price you are going to have to do more up-front to know where to set the bar. But even closer to waterfall you can still do a show and tell on a regular basis."
Agile criticism
But commentators like Forrester’s Gualtieri are not at all convinced. He and his research collegues have been arguing for a new approach to software development, which as he says, "Asks, ‘What software development process prepares us to be experience creators?’ I call this new approach: parallel, immersive software studio… I just call it STUDIO".
Gualtieri outlines four pillars of his new approach, and says this parallel, immersive software studio is different from other methodologies in five important ways:
1. Software is not code. It creates experience.
2. Development teams are not coders. They are experience creators.
3. Technical talent is table stakes. Great developers must be design and domain experts.
4. Process is bankrupt without design. You get what you design, so you better get the design right.
5. Software is a creative endeavor, not an industrial process like building automobiles. The methodology is structured to support the creative talent.
"It is just a new idea in a sea of many," Gualtieri opines. "But, the accelerating demand for applications that ‘wow’ businesses and users will create new winners and losers in the world of software development. Agile-mania is ending. The evidence: the Agile gurus are now telling us what is wrong with Agile and how to fix it. I say: Throw the baby out with the bath water. Now it is time to get serious."
Another of Gualtieri’s colleagues, Dave West, reported on issues with some agile implementations and wrote a paper called "Water-Scrum-Fall Is The Reality Of Agile For Most Organizations Today." The report also shows Agile adoption was at 38.6% in 2010 and notes that interest in Agile remains very, very high.
Digiterre’s Cheesbrough again: "Agile helps a lot, but I do believe the biggest factor is having good people; I think that outweighs the process. People are first and foremost."
It’s unlikely we have seen the end of agile software methodologies – indeed we still haven’t seen the end of waterfall. And after all, the Agile Manifesto was only penned eleven years ago and is still a relatively nascent body of ideas. So while Forrester and others may find much to criticize agile, its authors deserve some kudos for putting aside personal ambitions to jointly draft one of the most important software development manifestos of all time. Even if, as some critics suggest, it may be about time to update some of its principles.
The Agile Manifesto authors in full:
Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Stephen J. Mellor, Ken Schwaber, Jeff Sutherland, and Dave Thomas
