Agile’s Second Chasm (and how we fell in)

Some years back, I remember the excitement in the Agile community when the general public started taking us seriously. The common reaction went from “Get away from me, crazy people,” to “That’s interesting; tell me more.” It felt like the car we had been pushing uphill had finally reached the crest.

At the time, it was very exciting. In retrospect, perhaps that was the beginning of our doom: the car passed the crest, picked up speed, and careened out of control. Now the state of the Agile world regularly embarrasses me, and I think I’ve figured out why.

The first chasm

Geoffrey Moore’s seminal book “Crossing the Chasm” is about a gap between a product’s early adopters and the rest of a market. Writing in 1991, he looked at why disruptive technology products fail, and concluded that there was an chasm between the early market for a product and everyone else. Some companies fall in, some cross it to success.

crossing-the-chasm

The essence of this is that the early market, who are circa a sixth of the total market, are different. They are comfortable with changing their behavior, and are eager to do it to get a jump on the competition. In contrast, early mainstream purchaserswant evolution, not revolution. They want technology to enhance, not overthrow, the established ways of doing business.”  Later mainstream purchasers don’t want to change at all; they do it because they have to. For a physical technology product, that mainly shapes marketing and product accessories like support and documentation. But what does that mean for something as ethereal as a development process or a set of values?

The second chasm

You can’t buy an idea. Instead, people buy things like books, talks, classes, and consulting services. By and large, those buyers don’t measure any actual benefit. When wondering whether they got their money’s worth, they go by feeling and appearance.

For early adopters, that can be enough. As people who seek sincerely and vigorously to solve a problem or gain an advantage, they are strongly motivated to find ideas of true value and to make them work. But for the later adopters, their motivation for change isn’t strong. For many, it’s merely to go along with everybody else, or to appear as if they are.

Because those later adopters are numerically greater, they’ll be the majority of the market. And unlike early adopters, who tend to be relatively self-sufficient, they will consume more services, making them an even bigger influence. They’ll also tend to be the least sophisticated consumers, the least able to tell the difference between the product they want and the product they need.

Putting that together, it suggests that when people want an idea, the most money can be made by selling products that mainly serve to make later adopters look or feel like they’re getting somewhere. Any actual work or discomfort beyond that necessary to improve appearances would be a negative. Bad products and bad ideas could drive out the good ones, resulting in a pure fad. The common perception of the idea would come to be defined by the majority’s behavior and results, burying the original notion.

That’s the second chasm: an idea that provides strong benefits to early adopters gets watered down to near-uselessness by mainstream consumers and too-accommodating vendors.

Agile fell in

From what I can see, the Agile movement has fallen in the second chasm. That doesn’t mean that there aren’t bright spots, or people doing good work. You can, after all, find old hippies (and some new ones) running perfectly fine natural food stores and cafes. But like the endless supply of grubby, weed-smoking panhandlers that clutter San Francisco’s Haight-Ashbury district, I think there is a vast army of supposedly Agile teams and companies that have adopted the look and the lingo while totally missing the point.

Originally, I based this notion on the surprising number of people I talked to who were involved in some sort of Agile adoption that was basically bunk: Iterations that weren’t iterating. Testing that wasn’t testing. Top-down power relationships. Cargo cult Agile. Fixed deadlines and fixed scopes. Teams that aren’t teams. Project managers who pretend to be Scrum Masters. Oceans of Scrumbut.

That could have just been me, of course. But over the past year or so I’ve talked to a number of Agilists that I met in the 2000-2003 timeframe. They have similar concerns. “This isn’t what we meant,” is something I’ve heard a few times, and I feel the same way. Few these days get the incredible improvements we early adopters did. Instead, they get little or no benefit. The activity of the Agile adoption provides a smokescreen, and the expected performance gains become a stick to beat employees with.

Honestly, this breaks my heart. These days, every time an acquaintance starts off saying, “So now we’re ‘doing Agile’ at work…” I cringe. I used to be proud to be associated with the Agile movement. But although I still love many of the people involved, I think what happened is a shame. Looking back, I see things I wish I had done differently, things I intend to do differently next time. Which makes me wonder:

  • Have other early Agilists noticed this?
  • If you had it to do over again, what would you do differently?
  • To what extent do you think the problem is solvable?

I’d love to hear what other people think, be that via email, on Twitter, or in the comments below.

50 Comments

  1. Rasheed Abdul-Aziz:

    Don’t lose heart.

    q: Have other early Agilists noticed this?
    a: sure. I don’t want to leave the company I’m in for fear of facing ‘not agile but says its agile’. I wouldnt join a not-agile shop, but joining one that on the surface looks like it, that would be worse. However, I’m not sure we’ve fallen into the chasm, just a lot of adopters have.

    q: If you had it to do over again, what would you do differently?
    a: no.

    Q: To what extent do you think the problem is solvable?
    a: Continuous delivery from real agile teams will win, because it’s sustainable. I don’t think the race is run, not by a long shot. Ruby (especially RoR) adoption is still on the up and up. That’s a community that backs craftmanship (which I take to be what Agile is about, in the long run). Some of the more amazing Java projects, including Jenkins and IntelliJ/Jetbrains all have continuous delivery and amazing craft. There’s even a movement among Flex developers toward this end (www.robotlegs.org).

    I don’t think you can stop the force of performance (with clear metrics) that real Agilists bring.

    That’s my two cents.

  2. Bob Marshall:

    William,

    Yes, it’s very depressing to see so many good intentions of so many folks come to naught (or next to naught). I’ve been in the Agile space for more than 17 years now, and I have certainly noticed. In fact, this is what drives my work on Rightshifting – trying to understand the root cause(s) of WHY this is happening.

    And there is hope, I believe. Hope that folks will come to see that the problems you mention – with e.g. Agile adoption into mainstream companies – ARE recognisable, and amenable to change. Rightshifting explains why these disappointments happen (root cause), how to recognise whether there’s an opportunity for progress (on a per-case basis) and how to frame the challenges so as to maximise the chances for agile adoption success.

    - Bob

  3. Chris Parnin:

    This is a great model to explain how software engineering philosophies can be adopt and grow.
    But I’m not sure if that’s the only thing going on here.

    I can’t help but be reminded of many Mac users who freaked about the new iPhone (see MacHeads documentary). It wasn’t there movement anymore, too mainstream. Sometimes things grow up. Reality is ugly. People have work to do.

  4. Will Cain:

    The product marketing perspective applied to agile makes a lot of sense. (Disclaimer: I was already a fan of Crossing the Chasm.) My experience would tend to support your observation. I’ve worked as a developer on “agile” projects that fit pretty neatly into either the early adopter or mainstream camps. I would posit that it’s hard to have mainstream adoption of disciplined agile processes without support at the top, and without a few true believers leading the transition. This also seems true of nearly any significant process change, such as test-driven development or pair programming. Habits and behaviors can be difficult things to break. Agile may be especially challenging, however, because it places a premium on transparency. It tends to shine a bright spotlight on product management as well as on potential skeletons in the closet like technical debt and developers’ personal velocities. This can prove an uncomfortable exercise. However, I think the true believers of agile value this honesty and openness. Perhaps the more thoughtful and open organizations will be the more competitive ones in the end. I hope it’s not naive to think so.

  5. Tristan Kromer:

    Good observation.

    I can’t make any useful suggestions, but it makes me wonder about certification programs like they do for Fair Trade, various MS tools, or energy efficiency. They all seem to pop up when these sorts of questions come to the forefront.

    I wonder under what circumstances certifications are useful, what it takes to popularize them, and so forth.

  6. whatsthebeef:

    There was always talk of agile being customised for specific teams and companies to suite the way they worked, this provided a ‘get out clause’ for managers, customisation was clearly what was required, but much greater customisation, and not customisation which just meant removing the difficult concepts.

    A much tighter definition of agile is required, for example, define the 10 commandments of agile but unlike the actual 10 commandments, if they aren’t followed then the adopter can no longer claim to be a agilist and they can start a new movement and call it something else.

  7. jeffa00:

    I’ve yet to work in an environment that *really* implemented an agile methodology, but have worked in a few who claimed to. My favorite comment from a manager (paraphrased and probably poorly remembered on my part) was along the lines of “We’ll get things done faster because we’ll SPRINT!” LOL

  8. David Rader:

    Very accurate observations. A lot of clients want “Agile” because they have heard about it or someone in the executive team is championing it. But upon digging in, they also want Gate Reviews, multiple levels of design documents, long term detailed project plans, up front requirements analysis, and no more than quarterly product releases – because their organization is not built to handle a faster release schedule. It’s important to realize that “Iterative” is not the same as “Agile” – and RUP with short iterations could be a good choice for an organization that is not yet ready to give up the command & control ways.

  9. Rob Myers:

    Having recently worked with a few very large organizations who were “going Agile” recently, I have to agree, William: The “Cargo Cult” mentality often overwhelms the efforts to make lasting, humane, profitable changes.

    In my assessment, this is caused by two things (perhaps it’s really one thing, but I can’t help but separate these):

    (1) One is the difference in goals: The companies that do well with “Agile” transition are looking to improve the flow of value *by* applying humanistic, reality-based practices. The cargo cult companies are only trying to increase the flow of value.

    The other is culture: Are the execs, themselves, prepared to alter their own lifestyles to live within the values that Agile espouses? Far too frequently (even within tiny organizations) “Agile” is something being done to the workers, because they are seen as the constraining element of the system.

    Growing systems tend to create hierarchies; the problem starts when a few at the top think that the whole system exists to serve their needs. Effectively, the organization has created a Corporate Ego, which assumes It needs to continue unchanged. A dysfunctional corporate culture will adopt prepackaged “Agile” (and pay top dollar) as long as that flavor of “Agile” supports the continued dysfunction.

    Many of the Agile Vendors may have the best intentions, but they go about it the wrong way (mostly, “One Size Fits All,” or “Please Check the Desired Box: [ ] Kanban [ ] Scrum [ ] XP”).

    Three things that keep me going:

    (1) Often within the transitioning organization, a single real Agile team will emerge, and that’s a start. I’ve helped those people see the dysfunction, and the needs that hide beneath. I’d rather leave an org with at least one high-performing team among dozens of moderately functioning teams. The external coaches can then help implement some way to share the new knowledge this team has obtained. (I didn’t say it was an unarguable success, just that it keeps me from spiraling into a pit of despair.)

    (2) There are teams and whole orgs who are just quietly doing it, without announcing it, or getting involved in the cacophony of political arguments.

    (3) “Agile,” as it’s generally defined as applicable to software teams, is just a piece of the whole picture. It’s no longer enough for me to be a coach of Agile principles and practices. I (and the team of coaches necessary for larger transitions) need to explore and help adjust the whole system, perhaps at first in subtle ways. Transitions are painful. Misdirected transitions are doubly so. There’s no reason Agile principles and practices cannot provide great value to a huge organization, but if something doesn’t solve the root constraints, it’s just wasted effort.

    I’m not saying “There’s another way! Abandon Agile!” but that Agile may have to wait a while. Does the org want to “Go Agile” or do they sincerely want to improve the flow of value?

    So we learn from and draw from XP, Scrum, Kanban, Lean, Theory of Constraints, Systems Thinking, etc. I’ve come to learn that the fundamentals–humanistic principles and systemic goals–are the same in each. Again, this doesn’t help the world of business as quickly as we’d like, but it helps me to see when there’s a much bigger system, with proportionally bigger opportunities for improvement.

    I’m still learning how to have patience, and to be happy with whatever gains occur that would not have occurred without my “external” coaxing.

    So don’t despair. This happens to every good idea. Once the self-proclaimed thought leaders have finished diluting the word “Agile” until it’s meaningless (which may have already happened), we’ll rename it or add a modifier. ;-)

  10. Matt:

    Midst all the hand-wringing about the demise of Agile I can’t help but wonder “what’s the problem?”

    If you are doing Agile right in your shop then presumably things are going OK and you are producing good quality software on a timely basis.

    If someone is doing Agile wrong then perhaps they are or aren’t producing (after all, lots of good quality software got produced prior to 2000 if I remember right.) If that’s the case then the worst case scenario is they aren’t doing well and they are blaming Agile. But if they are blaming Agile…why is that bad? If not Agile, they will blame the economy, the weather or perhaps the Aztec Sun God. In any case, their loss and who cares?

    I can honestly say I don’t care if I work in an Agile shop, a pseudo-Agile shop or a non-Agile shop. What I care about is having the opportunity to produce high quality software that solves real-world problems in a timely manner. You can have your labels, I want to write software. So far I am not convinced Agile (real or otherwise) has a corner on that market.

  11. The Agile manifesto, 10 years later – Steven Willems:

    [...] Reading this article of ten years ago again, you might understand why. As it is well explained in this blog by William Pietri a lot of the agilists out there have missed the point completely. The industry picked up the Agile [...]

  12. Patrick Steyaert:

    Today one hears the same kind of reflections in both the agile community and e.g. the CMMI community. The tool is right but it is wrongly used by the majority… If the tool is wrongly used by the majority then maybe, just maybe, we should also examine the tool.

  13. Andy Brandt:

    I think you miss one key factor here – agile was created by people who really and deeply care about what they do – software is (or at least was at the time their passion). However, this is not what drives most people at work – most people just come there to do something they don’t care about to get money for what they care about, namely their lives, hobbies etc. Agile won’t work with people who don’t care and it is its internal problem that was revealed once it crossed the chasm (or, as you call it, the first chasm).

    BTW – I wrote about it like two months ago… http://www.andybrandt.net/653/is-agile-for-everyone

  14. Bob MacNeal:

    You’re dead on target. The mechanics of Agile become a blunt weapon when the principles of Agile are disregarded. I don’t think this can be fixed. Too many corporate ass-clowns looking for an easy fix, rather than transformative change.

  15. euromix:

    My experience with agile not working is wealthy organization that cannot provide a product owner.
    It was a way to bring awareness of that problem. It did not change much the situation as there was a large agreement in management that people have to deal with what they have.
    This being said, when the company pays and pays well, i have not seen suppliers willing to announce that the situation does not meet the minimal requirement to start a project properly.
    The project becomes an agile crisis management project. money flows, crisis come and go, and agile is seen as not doing any better than other way….

  16. Marty Nelson:

    Turning to blame for Agile adoption failure is not the agile way. This includes:
    * People aren’t doing it right
    * People aren’t doing it the way we told them to.
    * People don’t care enough
    * People don’t have the right values
    * People are just followers
    * People are watering down *our* ideas to make a buck

    Clearly there are people who have success with agile, can you think of other reasons? Consider:

    * What is implicit in those successes that we need to make explicit?
    * Are there areas were agile depends on exceptional talent that need to be made into teachable proficiencies?
    * Do we need to expand our notion of team to include others who are critical to our success?

    Take heart, but get off the pity-potty and roll up your sleeves and think about how to change things for the better.

  17. Jonathan Kern:

    This is the beauty of freedom to choose. For those that (loudly or otherwise) fail at Agile, there are just as many (quietly or otherwise) succeeding. The market will sort things out. It allows for new shoots of green grass to grow up as the old companies die from under their own ineptitude.

    The biggest “failure” — if there is one — is the lack of education in the underlying, foundational, principles of just what it means to *be* agile. (Too many think that it is sufficient if you just *do* agile.)

    I personally do not lament the misunderstanding of companies when attempting to live up to the agile tenets.

    Besides, Agile will never die : http://technicaldebt.com/?p=673

  18. Vin D'Amico:

    What we’re seeing is normal. Agile development is maturing. It has gone from a niche technique used by a minority of development teams to a mainstream approach used in many corporations. I’d like to suggest that agile development has reached it’s adolescence. It’s no longer a child but not yet an adult.

    I know it can be frustrating to hear of people “using agile” and failing, when we know that they merely modified their waterfall approach. They incorporate iterations, daily meetings, and backlogs into waterfall. Instant agile, right?

    We need them to fail. We need them to learn. We need them to adopt real agile development. It will happen. Give it time.

  19. Rob Myers:

    Marty,

    You make some excellent points. I can’t actually speak for William, but I didn’t get the impression he was on a self-pity trip.

    What hurts me is the fact that, by accepting mediocre versions of agility, people on teams continue to suffer ridiculous demands for overtime, excessive administrative gymnastics, and other forms of wasteful business practices that have been proven time and again (over at least 50 years) to be counterproductive from a whole value-stream perspective.

    I’m not (and William isn’t, I believe) one of the suffering team members. My job is to help teams and organizations to uncover constraints in–and improve the flow of–a system. The fact that the people living in the system will benefit from increased pride in work, better work/life balance, and less rework, is what gets me up in the morning. When I see my efforts misused for individual gain (of a consultant or someone within the organization), then I feel my own time there has been wasted.

    You asked the following questions. They seemed possibly rhetorical, but there are answers. Here are mine:

    * What is implicit in those successes that we need to make explicit?

    Passion. The desire to be involved in building something that is valuable, usable, and high-quality can inspire developers, testers, managers, Product Owners, UX, everyone on the whole team.

    People who want to learn. If the team consists of people who want to better their skills, and the process allows them to explore their interests and sate their spongy minds, then great things can happen.

    * Are there areas were agile depends on exceptional talent that need to be made into teachable proficiencies?

    Yes. If the product requires code-writing, then we hire developers who know how to write good code. By good, I mean maintainable. By maintainable, I mean comprehensible to the other developers, and fully tested (checked) via automation.

    I say “if” because I’ve run into far too many IT app projects that involve mere marshaling of data back and forth between screens and database. That doesn’t require a software team; there are shrink-wrapped packages to do that for you. No one feels passionate about data-entry apps.

    * Do we need to expand our notion of team to include others who are critical to our success?

    Yes! Isn’t this a given??? On Lean teams, the team forms around the building of a product. The goal is to release a high-quality product. Even IT teams can approach their activities in this way. Who needs to be involved in getting that product working for the customer? How much involvement, and when? Are there gates, and if so, which one is the constraint? How can we ease the log-jam behind that constraint?

    Thanks,

    Rob

  20. Nick Bauman:

    This is an ancient problem with semiology: People confuse the sign for what’s signified. Agile is now just a brand name to those people.

    I think the key is to start at the beginning again. What do you value? What do they value? I think a simple vocalization of these values are the beginning. Without the values, the principles don’t make sense. Without the principles the practices are hollow. And then you have Scrum, which is a placeholder for people who don’t get it and never will because they have the wrong values to start with.

  21. Rob Myers:

    PS to Marty:

    Darn, but I can’t edit my own comments, and I didn’t intend to be purely dev-centric in my answer to your 2nd question…

    * Are there areas were agile depends on exceptional talent that need to be made into teachable proficiencies?

    Devs: Good OO coding practices, and other engineering practices. The XP practices are a nice starting point. So are Design Patterns. Something that gets them beyond what they learn about modular programming in most Universities.

    Testers: Good testing heuristics and paradigms (e.g., pair-wise). A variety of automation tools. Exploratory testing. Deep familiarity with the product. Anything that gets them out of the point-click-type follow-the-test-script mentality.

    PMs/SMs/POs: Communication techniques such as active listening, or non-violent communication. Systems Thinking. Collaborative leadership techniques. Basic neuroscience. Whatever gets them beyond the fallacy of command-and-control.

    Orgs who assume that they can send everyone to two days of Scrum training and they’ll all start “doing Agile” are in for deep disappointment. An Agile Transition is a journey.

  22. Luke Arno:

    There is a more fundamental problem here. Agile methodology is a methodology. Humanism is not a methodology. Agile is a good methodology, a good tool, but I think the disappointment we share stems from a false hope: that the methodology would transmit a humanistic perspective. In fact, maybe promoting (and selling) this methodology has only confused the point. Maybe we should just collectively focus on clearly expressing our values, if that is what we are actually want to communicate. Promoting a departure from utilitarianism on a utilitarian basis is self defeating. It has only sown the bumper crop of hypocrisy we now reap.

    Agile is great, but what really matters to me are human beings. I value people for who they are, not as resources to be used. And they are all invaluable. I value craft because anything less would disrespect the people who do the crafting _and_ the people who experience what is crafted. Maybe you can be more effective and make more money if you actually have good values (in fact, I think you can) but that is not the point. When that is the point, you already have shitty values. Being an artisan is about something more than cash or winning. It is timeless and needs no buzzwords, no sales pitch. It’s about valuing your own life and the lives of others as invaluable. That is what self respect boils down to. If you have that, whatever methodology a team decides on or comes up with will probably be great.

    @Rob Meyers: Did you just invent the term “Corporate Ego?”

  23. Malcolm Anderson:

    Ouch, this hits very close to the bone, but at the same time explains a lot of what I’ve been dealing with out in the market.

    I finally just answered this over at my blog (complete with link-back)

    http://geekswithblogs.net/geekusconlivus/archive/2011/02/22/response-agiles-second-chasm.aspx

    here’s the summary
    Yes
    Nothing
    Implement agile engineering practices.

  24. Marty Nelson:

    Rob -

    The pity part was meant to be a goad to action, hope nobody is offended. Beyond our disappointment, it’s just change, right? Inspect and adapt.

    No, they were not meant to be rhetorical questions. I agree that lack of passion is a big problem, or as Esther Derby says “teams Share a compelling work goal”. The odd thing is our tendency is to still focus on dev changes (dev-centric as you put it). We’re really hard on ourselves, and we tend to think everything can be solved by working ‘smarter’ (our version of ‘harder’). Technically excellent crap is still crap and that can be demoralizing.

    Or we look over the other side of the fence to ‘the business’ and throw out terms like “Systems Thinking”, “Value Stream”, “Working Product for the Customer”. That seems like a lot of mumbo, jumbo, high-priced hand-waving. XP practices were (are) so cool in that there is a simplicity (does not mean easy) and it’s something you can show someone how to do.

    I think it’s that ‘fence’ between dev and the biz that’s the problem. We all talk and want the whole ‘whole team’, but that’s going to mean upheaval and change for everyone and things on the dev side are not going to stay the same. Current agile arose in a gated world were dev was intended to react to feature requests and that attitude is still largely intact.

    My .02 is that means letting go of *translating* customer desires and outcomes into features assumed to deliver those outcomes, and instead address the satisfaction of customer desires head-on and as our top priority. What does this mean exactly? Asking the customer a question directly framed an inquiry to what they want to be shown that proves they got the value at a moment in time after the system has been used in the traditional sense of a piece of functionality.

  25. kiwi:

    Agile fanatics are partly to blame. When they preach it like a religion there’s huge signs of hidden agendas and then failure is imminent. As the buzz was spinning higher years ago, people got lost in the process. The real lesson here is: be real, deliver to the customer in an iterative, phased approach, act as if you only code it once. Programmers get lazy because of “less design”, teams too large, and “you can always refactor it later”. Agile can either be a great big mask to hide behind, or a small sharp tool that provides structure.

  26. Grant (PG) Rule:

    I hypothesise that part of the problem stems from transferring agile practices from (relatively) small teams producing software products that can released to a marketplace of more-or-less independent end-consumers, to the corporate environment where ‘applications’ are commissioned on behalf of ‘end-users’ who then use those applications to deliver a service to the end-consumers.

    In product-oriented firms there can be more direct relationships between the Product Owner, the Development Team and the end-consumers for whom the product is intended. At least the Product Owner has a clear responsibility to obtain a deep understanding of what the end-consumer values. (NB: I use terms such as Product Owner etc. for convenience. It’s the role & responsibilities that matter, not the name of the role, nor the exact cut of responsibilities).

    In larger firms and corporations, the entrepreneurial leadership aspect of the Product Owner role is often very greatly weakened. It may go entirely unrecognised or be misunderstood. Both by the (often relatively junior) person assigned the role of ‘product owner’ and by senior management.

    Additionally, the length of the value stream (i.e. the end-to-end network of processes) in a typical software product company is significantly shorter, and hence feedback is more rapid, than it is in many large companies (in practice, it doesn’t have to be). The result is that many ‘agile implementations’ amount to local optimisation. Often led bottom-up by an ‘agile enthusiast’. It may maximise the efficiency of the software development group, but the failure to engage upstream and downstream stakeholders in the transformation results in de-optimisation of the end-to-end value stream. So waste is developed faster.

    Local optimisation and ‘point improvements’ can only go so far. They are very difficult to sustain. In my +38 years in software development I have seen this pattern repeated over and over again.

    It’s not that executive leaders don’t want more effective processes, and a happier, more fulfilled workforce. Of course they do. And IME the creative, customer-facing workforce ‘get it’. They know that current processes can be improved and are not averse to trying new practices when their effectiveness can be demonstrated. However, there is (often) a significant ‘middle layer’ of managers who exhibit all the characteristics of Geoffrey Moore’s ‘mainstream’.

    Achievement of a sustainable tranformation requires understanding, vision, purpose and active support from the executive. What often works most effectively is a bottom-up, top-down alliance to promote change, that encourages the ‘middle layer’ to change ‘cos they ain’t got no choice. But don’t expect them to thank you.

  27. David Koontz:

    I think if one applies the Bridges Transition model (Ending Event that signals Change, Neutral Zone of uncertainty and Chaos that allows for creativity, New Beginning) locally or globally one finds that we early adopted think we are ready for the New Beginning – but the newly converted (majority) are still in the Zone of uncertainty. We need to guide them with compassion toward the creativity that becomes their New Beginning. Yes it takes much more energy to transform the masses than the early adopters – what did we expect?

  28. Steve Powell:

    I think it has been said (almost) before in the comments, but let me make it explicit:

    The early adopters correlate highly with success; this is no surprise — they are the ones that have the passion to succeed. That is why they were the early adopters: they are self-selected.

    It is not beyond the bounds of possibility that Agile has nothing to do with this phenomenon. It is possible that the early adopters will always do well, simply because the sort of people that adopt early are the sort of people that succeed.

    If you take a group of intelligent, impassioned people (in any field) and give them enough support (and room to manoeuvre), they will produce great stuff. Agile is a meme-hook that attracts those types of people — ergo, Agile looks good. So did Six-sigma, TDD, Top-down design, formal methods, OOP.

    Does that mean that Agile isn’t cool? No! However, it might mean that Agile succeeds more because of what it allows than what it prescribes. It allows good people to find a good way. It is no surprise that the majority, corporate-driven masses don’t find a way of excelling, even with excellent advice.

    The real software crisis (with us since before 1980) may never be solved by better techniques in the hands of greater numbers of people; it might only be solved by greater care by a fewer number.

    Or it may never be solved at all.

  29. Chris Morris:

    I don’t think the problem is solvable**. How many people do you know that have exercise equipment in their house and how do you think fitness enthusiasts feel about it? (… or people with a grand piano in their house or a shiny toolbox in their garage or financial software on their computer or …) I’ve been a Christian most of my life and the same pattern you describe has been in the church for … ever. There’s seems to always be a vast majority of [X] who claim allegiance to an idea, but their actions aren’t exactly convincing. Pick anyone who’s commented on this page and go through enough areas of their lives and you’ll see the same pattern somewhere, myself included.

  30. Chris Morris:

    What would we do different? I don’t think it’s in our power to control this. The best we can do is attend to the work we’re given and struggle our best with it. As far as it breaking your heart, I think that’s inevitable as well, but a good sign. Breaks my heart, too.

  31. Rob Myers:

    @Luke Arno: RE “Corporate Ego”: Not likely. And, even if it originated in my brain, I’m not sure it’s worthy of the term “invent.” I just put together words to try to convey meaning. A hopeless endeavor, but I continue to try. ;-)

    I found your comment to be wonderfully inspiring! You are right, the “Agile methodologies” cannot themselves convey values to an organization. Only people can convey values to other people. And, as you say, it has to be about people.

    It’s a compromise: Often we have to find the “selling point” so we can get in and try to help an organization. I make it as clear as possible that I’m there to help them with what they need, not what they think they want. If they don’t like it, they’re free to let me move on. Organizations that have the right values are easy: We may go in, provide basic training, and leave shortly after, perhaps returning on occasion to make sure they’re following their own values, and to help them see the newest constraint. These orgs are fun to work with, but there’s a risk for any consultant of “hasty generalization,” or of assuming that I made it happen. So, I’m both relieved and dismayed that so much of my work is for orgs that aren’t living their values. In those places, the real, hard work exists, and the deeper learning occurs.

  32. Rob Myers:

    Marty:

    I’ve been told (by an Agile “Thought Leader”) that I was “too XP-centric.” What that actually means is that–if I feel that the key constraint can be alleviated by teaching the team TDD–I will not shy away from teaching the team TDD. It’s one tool in my toolbox (a double-barreled megawatt power-tool, perhaps…).

    IMHO…

    In 1996, XP was an amazing revelation of a balanced set of techniques for building software. (Scrum was an amazing revelation in team-management techniques, and many of those were applied to XP.) I strongly believe we need to take another look at that collection of concrete practices we call “Extreme Programming.”

    There are many who argue that XP is too rigidly defined, and requires too much discipline (whatever *that* means) for many teams. Perhaps. Nevertheless, it’s a concrete manifestation of the values and principles put forth by the Agile Manifesto, as well as the seven Lean principles. And I now see it as brilliant “Systems Thinking” (within the boundaries that Kent Beck was always quite clear about, but naysayers chose to ignore. In Systems Theory, Lean, or the Theory of Constraints, if you ignore the boundaries, you may miss the whole point.) Whether through logic or intuition (or both), Kent included numerous rapid feedback loops, and eased key constraints around communication. You touch on those in your “$0.02″ paragraph. :-)

    So, as an old XP programmer and coach, I’m with you 100% on the XP thing.

    The reason I’ve broadened my studies to include Lean, Theory of Constraints, and now Systems Thinking (and all three of those look oddly similar to me) is that very fence you speak of (and those gates).

    In particularly large organizations, or orgs whose primary business is not software-related; *the* problem is frequently not the software team at all. We can help the dev team(s), but if we miss the Bigger Problem that lies somewhere else (upstream or downstream) in the organization, then the org won’t see any benefit from the Agile Transition, the Agile teams are still squeezed at both ends, and we’ve generated more anti-Agile sentiment (and fueled the notion that Agile Consultants are all money-grubbing fanatics). It’s important that someone (an external coach, mid-level manager, or exec) be aware of the broadest business context, and–through one or more of those “high-priced hand-waving” strategies–look for areas of trouble.

    Having said that, the right answer may be to apply some disciplined Agile techniques within a struggling Scrum team. (In smaller orgs, that is very frequently what is needed.) The trick is to get planners *and* builders to see (and create) the thread of logic, so that everyone can support (and own) the changes to their daily behaviors.

    Inspect and adapt, and vary your focus between the overarching system and various smaller systems within.

  33. William Pietri:

    Hi! Thanks everybody for the great comments. And thanks especially for the stories of personal experiences; I love those. A few responses:

    Why do I care? Because I’m part of a profession that I care about. And because the early Agile world was a community of practice that helped me a lot. Falling in the second chasm harms both.

    For those who think degradation is inevitable: I disagree. Some ideas and movements haven’t lost their mojo, at least not to the same extent. E.g., medicine.

    I agree with a number of commenters that large companies are much more likely to be problematic.

    People have asked if I’ll write more about this, including on what I’d do differently next time. As it happens, I’ve drifted away from the Agile movement toward the Lean Startup movement, so my asking is not entirely idle. Hopefully my muse will deliver some useful prose on these topics!

    Thanks again, one and all!

  34. Ian Chan:

    Fantastic article William, I couldn’t help but nod along as I read through it.
    I’ve recently come to accept that the holistic/principles/foundations of Agile methodologies are often interpreted as fanatic dogma by many casual Agile adopters and Agile critics. This is unfortunate as I (like everyone here) has seen the Agile name deteriorated down to nothing but a buzzword. I avoid criticizing casual/loose adoption patterns, but people labeling any non-waterfall process as Agile is a bit frustrating.

    Agile ISN’T the silver bullet, and people who understand the process know this. But people needed to cling onto something when it was announced that Waterfall is “uncool”, and Agile was there to (apparently) save the day. I left IBM just before they mandated an “Agile” development practice across the entire 300+K employee company. The thought of that wakes me up at night. hah!

  35. John Kordyback:

    “The only thing worse than being talked about is not being talked about.” – Oscar Wilde

    Agile was either going to be “corrupted” or ignored. It wasn’t ignored. The question is what do we do next? To continue innovating agile we’ll need a lot more disciplines than the traditional developer focused communities. That’s the problem. We need teachers, marketers, carpenters, writers and a whole host of others I can’t even guess at. We’ll need to listen to them very, very carefully and not fear change.

    Evolve and surprise. Be fearless. Be humble. Be foolish. But most of all, don’t look backwards. That’s a sure path to stagnation.

  36. William Pietri:

    Hi, John. Claiming that “Agile was either going to be ‘corrupted’ or ignored” is a bold statement, but I’m not convinced yet. I think there are plenty of ideas that haven’t been entirely corrupted. That includes, as I said earlier, medicine.

    Also, I think that telling people they never should look back is foolish. One shouldn’t only look back, of course, but one can certainly learn useful lessons from history, which is what I’m trying to do here. As Santayana said, “Those who cannot remember the past are condemned to repeat it.”

  37. Dave Rooney:

    Hello John,

    I think a lot of this has to do with personality types. Have a look at this relatively ancient page from the C2 wiki:

    http://www.c2.com/cgi/wiki?MyMyersBriggsTypeIs

    Look at the number of people who are an ‘s’ vs. the number who are an ‘n’. Intuitors (n) don’t require much hard data to decide if something is a good idea, and often rely on gut feel. They make up about 25% of the general population, but they’re over-represented in the early adopter group.

    Intuitors

  38. Matt:

    Dave’s comment made me think that the premise of this article is that Agile is a product that is being sold to whoever will buy in. Perhaps the product was designed for only 25% of the population?

    There are many reasons for a product to fail in a market, not the least of which is that maybe it just isn’t that good of a product. The paradoxical aversion to change exhibited by many folks in the Agile world might be what keeps it from becoming a product that works across the board, not just in some shops.

    (I am not flaming, I think it’s a good product and I already bought mine.)

  39. William Pietri:

    Matt: I think there were in effect two products. I think the pre-chasm product delivered basically as promised. The post-chasm product was very well suited to what the post-chasm audience most wanted to buy, with little consideration as to what they needed. It did not deliver much value, and drove the pre-chasm product into obscurity. Whether or not you consider that a good product depends on your philosophy of business.

    I’m told that the people who originated the butter compartment in the refrigerator carefully thinned the door insulation near it, so that the butter was kept cool but not cold. That way the butter stayed spreadable. But all most people understood from looking at it was that there was suddenly a special place for butter, which seemed neat, so that’s what other manufacturers built, and that’s now what we have today. The form was preserved even as the point was lost.

  40. Michael Sahota:

    Hi William. I am totally on the same page. I think that we as a community are beginning to wake up and smell the odour. I just wrote a related post: Shhh! Agile Failure in the Large – http://www.agilitrix.com/2011/02/agile-failures/

  41. Scott:

    William, great post but IMO you miss your own point. You lament, “…Looking back, I see things I wish I had done differently, things I intend to do differently next time…” However, you fail to mention what those things actually are, showing (philosophically at least) that again Agile is too soft to learn from it’s own lessons. You’ve created this great post saying that there are lessons you’ve learned (that could help avoid falling into the Chasm [I assume]) but then you don’t enumerate any of them. Instead you do the frustratingly Agile act of asking for feedback.

    It is this exact point which, IMO, weakens Agile. Where the “Agilist” should step up, they step back. Where leadership should be shown we wait for things to “self-organize.” I’m not talking about command and control anymore than I’m waiting for the million monkeys to deliver my copy of the encyclopedia (actually, now that I think about it – wikipedia fits that experiment – ha). What I’m talking about is leadership. [I know these are gross over-simplifications - please don't spam me for these claims.]

    I was coding some error handling the other day and my partner thought I was putting in too much, but when I looked at what he wanted to do (bubble all errors up to the UI) I pointed out that that removed all possibility of “handling the errors.” This is a common mistake. If you don’t provide a place for things to be dealt with, you can’t blame the higher levels for doing it their own way. If all you do is throw, something up the stack will eventually find it’s own way of dealing with the error. This may include falling into the chasm.

    In Agile, I’ve noticed that the processes are designed to allow things to “bubble up” somewhat organically. But by not offering enough concrete guidance, Agile opens itself to many options which, in and of themselves, can kill Agile.

    Why? Why not provide the guidance and leadership to steward the Agile process along. Waterfall sucks because it’s too much of this and all one way. Agile (IMO) sucks because it’s too little. It’s sitting around the scrum fire singing campfire songs while management burns. However, I see not much wrong with Agile outside of a lack of leadership.

    If adding a leadership component to Agile would kill it, then it’s already dead. Scrum, which predates Agile knew this. It build the leadership role right in (scrum master). It would be nice in Agile if the product owner could fill this role but as we all know, most product owners are not versed in Agile (nor is it their business). IMO, Agile is missing this role (I’ll coin it the AgileMaster in honor of the ScrumMaster: apologies to the 330 hits google returns on AgileMaster).

    Perhaps you have shared your insights in another blog post that I missed. If so, would you be kind enough to share a link. If not, would you consider making such a post?

    Please, share what you would do differently.

    Thanks, Scott

  42. William Pietri:

    Hi, Scott! I haven’t written that post yet, although I would like to. I didn’t mention any of my proposed solutions here because I wanted any replies to be uninfluenced by my notions. Also, the lessons I’m learning about the past of the Agile world aren’t really applicable to it today; I want to apply them to the much younger Lean Startup movement.

    I don’t see Agile approaches as incompatible with leadership. Indeed, Wikipedia is a fine example of how far you can get with self-organization, which allows individuals to lead as they feel moved, and as others see them as able. If anything, I think Agile approaches allow for more true leadership, something that traditional dominance hierarchies tend to crush.

    The Agile movement, however, lacks the unifying purpose of a software development team, which is to make a specific thing for a specific audience. Although the Agile Manifesto was a great statement of common principles, it was definitely not a statement of a common goal. Since they didn’t aim to get anywhere in particular, we shouldn’t be surprised that’s where they got to.

    I appreciate the suggestion for the future article, and it’s one I definitely want to write. Whether it ends up here or elsewhere, I’ll definitely make sure there’s a pointer from our RSS feed so that you can be notified.

  43. Bob Gower:

    Excellent post William and very thought provoking. The post generated some good internal conversation here at Rally, so thank you.

    I started off applying Agile in startups (as you know – you heavily influenced my early thinking over Thai curry one afternoon in Oakland) and I now apply it mostly to larger companies as a coach for Rally Software.

    It occurs to me that these two groups map somewhat to Moore’s distinctions of early and mainstream markets pretty well. That said I’m sure there are many startups that are more conservative and mainstream than early adaptors.

    I think both groups have significantly different needs and in a sense Agile is a different product for each market.

    For early adopters it’s relatively easy to keep the “why” of the Agile conversion in sight. The teams are small, the pain is well understood (or acutely felt) and they are open to experimentally applying solutions. They learn naturally through inspection and adaption and their cultures are less entrenched and calcified.

    For larger companies the catalyst for (and champions of) an Agile adoption is often difficult to pinpoint. Is it predictability, speed-to-market, defect reduction or simply keeping-up-with-the-Googles that fuels the push to transition. Without solid management backing it can be difficult to get things to move. And even with it the organizational antibodies are a force to be reckoned with. Once things are moving the pace of adoption can be slow – painfully slow.

    To my mind though as long as you are incrementally improving your process, through regular periods of inspection and adapting, you are essentially doing Agile. No process change in any organization is perfect over night and we, as Agilists, need to keep focused on steady incremental improvement. This process benefits of course from a serious look in the mirror, however Agile maturity, like emotional maturity, exists on a gradient.

    As long as we are making steady progress does it matter if we are not perfect? The question is of course, are we making progress?

  44. William Pietri:

    Hi, Bob! Thanks for writing.

    We agree that there are basically two products. Where we disagree is whether both of them are truly Agile. I think the post-chasm product — the one that people have purchased when they say they are “doing Agile” — is not in fact what any of the early adopters had in mind. In particular, I don’t think continuous improvement is enough to be Agile.

    The Agile Manifesto comes with a very specific set of values and principles. A key one is, “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” Many, many of the alleged Agile adoptions these days don’t qualify. Other priorities are primary, but nobody calls this out as a problem.

    Which I understand: pointing out uncomfortable truths is hard and rarely profitable. Hell, even noticing the uncomfortable truths is hard. As Upton Sinclair wrote, “It is difficult to get a man to understand something when his salary depends upon his not understanding it.” And right now, the salary of a lot of Agile consultants and certified-in-two-days Scrum Masters depends on not understanding that what they’re doing is not Agile as meant by the people who created it.

  45. Bob Gower:

    True, I see many many organizations calling themselves Agile that I would call something else. The challenge I see however in taking Agile into larger organizations is that they do not change overnight, and even the best intentioned transitions can fail.

    I’m curious what the road to Agile looks like and believe in assessing maturity along a gradient more than making a binary distinction of Agile or not. That said I think you make an important point that we can easily get lost in the weeds if we are not keeping a focus on the creation of valuable software.

  46. William Pietri:

    I agree that creating a binary Agile/not-Agile distinction is uninteresting for the kind of people who are never going to cross a line like that, and I think that includes most people currently buying Agile adoption help, and nearly every large company. Who pays to be told they’ll never make the cut?

    My problem with a gradient model is that it implies they will get there eventually. I don’t think there’s much evidence to support that, and there are a number of reasons to think otherwise, including the Crossing the Chasm model and the extent to which Agile approaches require different power structures. So I would rather we have a model that conveys the truth of the situation as best we can discover it, rather than what we have now, which is the one most useful for promoting adoption efforts.

  47. Rob Myers:

    One tongue-in-cheek approach, if you have the patience to wait for change:

    1. Read “What Got You Here Won’t Get You There” by Marshall Goldsmith.
    2. Give the book, and these instructions, to your boss.

    ;-)

  48. Toby James:

    Great post William.

    A couple of months ago I was talking to a guy about the software development practises in the company he contracted to. He was saying that it was a mess and that the teams hadn’t delivered anything in months. He was brought in to “sort it all out”, and was looking at Scrum as a way to do this.

    He had his work cut out for him, management “fully” supported agile, loved the concept, seemed to know a bit about it and its successes. So what was the problem? Unfortunately they saw it existing only within the development team, and did not feel they needed to change at all. And this is where he was fighting his biggest battle.

    We continued talking for a while and I finally asked him the name of the company he was contracting to. Reluctant at first, when he told me I could have died. This company had been promoting for some time that they were agile.

    The team I work with live and breathe the agile principles, but we do not yet call ourselves “agile”. We have external forces that constantly challenge us and strongly believe that our practises are not yet to the standard of what would be considered agile. We are happy to tell people we are working hard towards agile and I believe we have the right people and company structure to get there.

    We do have a greater dream though, that one day we won’t need to be branded “agile” or “waterfall” or “Scrum” etc… and just be known as a team that consistently delivers kick-ass software that exceeds the expectations of the market.

  49. An Agile article from an early adopter | My Blog:

    [...] early adopter of Agile laments the current state of Agile he sees in many organizations. That’s the second chasm: an idea that [...]

  50. Marquis:

    Great article, you hit the nail on the head here.

    One thing that really bothers me is when a company wants to go agile, but they put the process in a position to fail. Either not allowing it to be used to its fullest potential (afraid of the short term risks/slow down as their teams learn the process and make it their own) or they only allow it to be used by a small team on a project that’s not overly important to the company. In both cases I’ve seen upper management state that agile isn’t all that effective because it didn’t produce the results they were expecting. Sadly at that point their minds are made up and it becomes even more of a struggle to introduce agile.

Clicky Web Analytics