Agile Focus

There seems to be a lot of confusion these days about whether something or other is really Agile, and what that means. Here’s my take on how to sort that out.

Growing up, I lived in a city called Grand Rapids. As you’d expect, there’s a river running through it. Does the river actually have rapids, and if so, are they truly grand? Do other rivers have rapids more rapid, or perhaps more grand? Those questions are interesting, but from the perspective of the name, it doesn’t matter. A long time ago, somebody thought that was a good name for the place, and it stuck.

Capital-A Agile

A decade ago and more, a bunch of people were working on new software processes. They were very different than what had come before, but they all had something in common. It was hard to put a finger on exactly what that was, but eventually they got together and came up with four value statements and twelve principles. And they came up with a single word: Agile. As in “Agile Manifesto” and “Agile Software Development”.

Was this perfect? No. Was it meant to explain everything about software development for all time? No. Was it a software development process on its own? Definitely not. But it was a declaration of common purpose, a list of things they could agree on.

That’s what capital-A Agile is: a bunch of people seeing that they had something in common, and attempting to say what that common thing was. They gave it a name and a partial definition. And most importantly, they formed a community that is still working out what that means and how best to do it.

Small-a agile

It’s important to note that they weren’t saying that they had an exclusive lock on agility, or even what made software development agile. As with the naming of Grand Rapids, they were pointing at a particular spot in the landscape of ideas and naming it. The word agile has a variety of meanings, and there are a lot of aspects to software development to which you could apply those meanings. They weren’t trying to lay claim to agility as a whole, any more than Grand Rapids is claiming all the rapids in the world, especially the grand ones.

That also means that there are plenty of ways to develop software that aren’t Agile. After all, software got made long before the Agile Manifesto was written. And there are surely ways of being agile that aren’t included in either the Manifesto or in the current practices of the Agile community. Heck, that’s part of why we get together every year, and talk so extensively on line. Processes based on continuous improvement gives you a real taste for continuously improving that process.

Saying “that’s not Agile”

So given this, what does it mean when somebody says “that’s not Agile”? To me, it just means that the thing they’re pointing at is a different spot in the landscape of ideas.

Some people get upset when they hear that, because they believe they’re doing well at making software, or because they think they’re being pretty small-a agile. They may or may not be right, but that doesn’t matter. If people in the Agile community say that something isn’t Agile, then it probably isn’t, the same way the city of Grand Rapids gets to decide where the city limits are.

If it bothers you to get told that something isn’t Agile, you have three basic choices:

• Find out more about what we mean about Agile. We’re generally a friendly bunch, glad to show you around. Join a mailing list, come to an event, or even ask in the comment box below.
• Persuade us we’re wrong. If Agile is the city we’ve built on the landscape of ideas, it’s a city that’s grown a lot over the years. It in effect started with a number of different little towns growing together. More recently came Lean, but these days it’s getting hard to even tell where the boundaries used to be. We’re very open to new construction, and your idea might be the next big development.
• Start your own thing. Agile may be the big thing of the moment, but it will eventually be as obsolete as the cavalry charge. Just because we say that something isn’t Agile doesn’t mean that it’s not a good idea. If you’re sure of yourself, do what the Agile founders did: stake out your own territory in the landscape of ideas and give it a name.

Regardless, there’s no need to get upset. Having different approaches or coming from different schools of thought doesn’t mean we don’t share the same goals in the end.

Much has been written on the subject of how to estimate the effort involved in developing software products. Some Agile teams use Story Points as their estimation units. Others estimate in Ideal Days. Still others use one of these plus Task Hours to help refine their estimates. Planning Poker is a popular method for quickly arriving at group consensus on estimates.

My own earliest attempts at estimating predate the Agile era by a couple of decades. In the beginning I used a technique that has probably been tried (with varying degrees of success) by many others:

1. Break the development effort into as many small tasks as you can think of
2. Estimate each of the tasks in hours
3. Add all the estimates together
4. Apply sufficient “padding” for comfort
5. Hope the boss buys it

Estimation’s main purpose

There’s no doubt that the more recent Agile estimating techniques are much better than what I started out with, and I’m glad that so much effort has been put into developing these methods. Even so, I believe it’s easy to lose sight of the purpose behind estimating, which can be summarized as follows:

The purpose of estimating is to provide the customer with a predictable development schedule.

Predictable is the operative word here. With a truly predictable schedule, the customer can derive the information that is most important from his or her point of view. Given a predictable schedule, the customer can answer important questions, such as:

• How much will this development cost?
• Will we be able to hit the market window?

In other words, the customer is probably not interested in the accuracy or precision of your estimates in and of themselves. The customer is more likely interested in the predictive power of your estimates. Given that, you want to focus your estimating efforts on things that provide predictability.

Duration should be derived, not estimated

I like to encourage teams to focus on estimating User Stories in Story Points, and to discourage them from estimating tasks at all, largely because coming up with Task Hours doesn’t seem to contribute much to the goal of providing the customer with a predictable schedule. Admittedly, estimating Task Hours may help to inform the team about how good or bad it is at estimating duration, but that’s a metric that’s not directly useful to the customer.

What is directly useful is duration that is derived from a team’s demonstrated velocity. Given a velocity in units of Story Points per iteration, plus knowledge of how many Story Points remain in a development effort, the customer can quickly and easily predict the development schedule.

Upon hearing about the short cycles of agile processes, some people are afraid that long-term planning never happens. They fear that teams could paint themselves into corners. That’s a risk: some teams do get fixated on each upcoming delivery, never sparing a thought for the future. But that’s not necessary, and below I tell you how to avoid it.

The secret

One of the secrets of agile processes is that they don’t tell you precisely what to do. Agile methods create contexts where people learn to do the right thing naturally.

For example, leaving loose ends is bad. You could make rules about exactly what constitutes a loose end and how to get rid of them. Many try.

Instead, you could start each week with a new set of work that you expect to fill the week. Then you release at the end of every week. That leaves no time for last week’s loose ends, which will turn up again as this week’s urgent bug fixes. In that context, teams have a strong incentive to learn how to tidy up all the loose ends that matter.

The secret, applied

So how do we create a context where your team will naturally spend the right amount of time doing long-term planning?

Step 1: Create a strong need for regular short-term plans. The best way to take care of this is through regular, frequent releases. If you are releasing something every week, then you are never more than 7 days from needing to have something to ship.

Step 2: Make a home for the long-term plan. There are always more good ideas than time to do them in. Instead of just throwing away the excess, you should keep them in a backlog. When you get enough of them that you need to organize them, make sure you do: that’s how the plan’s structure will emerge. For maximum effect, your backlog should be publicly visible, easy to update, and visually interesting. I usually use index cards and a rack or board. Do what works best for you, but if your plan becomes stale or gets ignored, try something more obvious and easier for everybody change.

Step 3: Establish feedback loops. It isn’t enough to release. You have to pay attention to what happens when you release. The right way depends on your environment. But observing what happens is the only way you can separate your good notions from your bad ones. The more often you do that, the better you’ll get.

Step 4: Frequently go over the plan. People with varied backgrounds should often walk through the plan together. Regular meetings are one way. But every question that starts with “why” is an opportunity to take a fresh look. As is the arrival of new data from customer feedback, user testing, market analysis, or economic forecasts.

Step 5: Be accountable to each other. A good team is in it to win, and win together. Mutual accountability helps a lot with that. Developers, for example, are responsible for building things that work for the long term, and the whole team should hold them to account for that. Equally, those managing the product are accountable to the team for the decisions they make.

Step 6: Be accountable externally. Almost every team has executives or investors that they are responsible to. On a regular basis (e.g., quarterly) tell them what you will do, what you have done, and how that compares to what you said you’d do. If they don’t ask good questions, get better advisors, even informal ones. As Doug Carlston, founder of Broderbund, has said, “The number one reason for bad software is too much money.” A lack of accountability is a big part of that.

Step 7: Keep time free for idle thought. There are many reasons to avoid being habitually overcommitted. But I think the biggest one is that tired or panicky people underinvest in the long term. We’ve all seen people come back from vacation filled with thoughtful observations and new ideas. Having good ideas once a year isn’t enough!

Some readers may feel like I’ve cheated them a bit. I haven’t told you how to break a project down into features. Or how to calculate value for units of work. There’s nothing about how to prioritize, estimate, organize, evaluate, or execute features. There’s nothing on market penetration or customer satisfaction or pleasing early adopters. Nothing on revenue models or organizational politics.

But that’s ok. I trust that given thoughtful practice, your team will figure all that out. The important thing is to start your practice right away.

Every time I’ve had the chance to talk in detail with people who hate pair programming (where two developers work jointly on the same task, usually sharing one computer, screen, and keyboard), I discovered that they were doing it in a way I too would have hated.

Below is a list of mistakes I’ve heard or seen real people make, including many that I’ve made myself. If you think pairing is not for you, first check to see if you and your team have solved all of these problems.

How to pair badly

1. Hog the keyboard – You’re obviously superior to your partner. Control the action at all times.
2. Ignore your partner – That noise that sounds kinda like useful suggestions? Probably a mosquito.
3. Ignore yourself – Speaking up for your own interests can’t possibly be helpful. Suppress, suppress, suppress!
4. Sit where you can’t see – You’ll be able to read the code when it’s your turn. If you still care then.
5. Sit where you can’t reach – Why be ready to take action? There’s probably not much you can do anyhow.
6. Take a back seat – Don’t sit next to your partner. Sit behind her. Then she doesn’t know when your eyes are closed.
7. Use a regular desk - Desks set up for one person work even better for two. Plus, the furniture police might catch you!
8. Don’t explain – Your partner, who is psychic, can just read your mind. Just keep on typing!
9. Don’t listen – You already know what your partner is thinking. And it’s boring.
10. Don’t ask – No, there isn’t a better way to do it, and your partner never forgets anything. So hush.
11. Interrupt frequently – ZOMG! They made a typo! You’d better tell them right now. They’d never notice otherwise.
12. Get distracted – Look, new mail! Hey, an IM! And check out Slashdot! Wait, there’s coding going on?
13. Daydream - Since you’re not really there to help, it’s ok to figure out exactly how many light bulbs there are in the room.
14. Be a back-seat driver – Your partner always wanted to be a stenographer. Keep telling them just what to type.
15. Don’t take breaks – Everything is better when you’re tired and cranky. Especially teamwork!
16. Work too much – It’s not what you create. It’s how many hours you spend at your desk.
17. Ignore ergonomics – The hunchback look is back in fashion. And who doesn’t like pain?
18. Betray trust – During, taunt about minor knowledge gaps. After, tell everybody how dumb your partner is.
19. Refuse to learn other tools – Emacs is obviously superior to whatever junk your team uses. To pair with you, they should all learn it. Now.
20. Don’t rotate pairing partners - If pairing for two hours is good, then six weeks with the same person must be way better!
21. Be arrogant – Seriously, what could you learn from that other guy? You know already: nothing.

Which ones are your (un)favorites? And what ones are missing from this list?

The last couple of years, I’ve run a big San Francisco race, Bay to Breakers, and it’s coming up again in May. I’m not much of a runner, and I’ve slacked off lately. But if I’m going to survive 12k with hills, now’s the time to start training.

How doing too much hurt me

The first part of my training is pretty simple. For 30 days, every day I do 30 minutes of mixed walking and running. (Want to start yourself? Use Hal Higdon’s plan.) But two days ago, I did too much. I met up with a more fit friend, and I let her set the pace. I did a lot more running than usual, and because it was fun to run with a friend, I did it a lot longer than usual: 60 minutes. I felt great, but like I had run a race, not gotten a workout.

It was fun, but I paid a price. The rest of the day I was a little loopy, a little unproductive. And then the next day, I was sore and tired enough that I didn’t work out at all. I knew I was supposed to go, but I just couldn’t bring myself to get out the door. And heck, maybe I should have taken a day off. Maybe my body needed the break. Doing extra seemed like it would be better, but it caused me to break training, and in the end, I got no more running in.

How it hurts teams

All that reminds me of a common novice mistake: overestimates causing productivity crashes.

Imagine a team just getting started with an Agile method like Extreme Programming. The first week, they guess they can do 20 points. However, they only get 10 done. How many should they pick to do the second week?

• 30, because they’re behind and want to catch up;
• 20, because the first week was an anomaly, and now they’re sure they can do 20;
• 10, because the evidence says that’s what they can do in a week; or
• 7, because they bit off too much and want to recover.

The most common answers are 30 and 20. They are wrong.

When people say that they are “behind”, that’s an indicator to me of a non-Agile culture. Their initial guess was just a guess. It is always a mistake to turn guesses into promises or plans, no matter how much stakeholders want promises or plans. A team picking 30, trying to catch up, is prone to spiraling out of control. They will repeatedly up the ante to gain back lost credibility, and repeatedly fail, either by not meeting goals or by producing junk.

Trying again to meet an unrealistic goal, as in 20, is a more insidious mistake. Especially early in an Agile adoption, a team is trying to learn how to do things right. But if they are continuously over-committing, they won’t have time to learn. Failing repeatedly to meet goals will discourage them. Regular over-commitment prevents successful Agile adoption.

The right answer

In the example above, I’d be happy if the team picked either 10 or 7. Why?

Guessing that they’d do next week what they did last week, picking 10, is an agile practice known as Yesterday’s Weather. It’s a realistic, data-driven answer, and assumes that you can get done about what you got done. There’s a risk that the team has still bitten off too much, in which case they’ll miss the goal again, but Yesterday’s Weather will sort that out in an iteration or two.

But what if they pick too little and say 7? Well, too little may not be too little. After blowing an iteration, the team can use a little slack to recover, to figure out what the problem is, and to fix things. But if it really is too little? That’s fine. It sets a team up to exceed the goal by pulling in something extra. That makes everybody happy, and it’s a habit you want your teams to develop.

Until a team has a strong record of regular quality output and consistently meeting goals, always err on the side of biting off too little. Challenging your limits is great, but first be sure you know what those limits are and how to recover from pushing them. When making plans or promises, do it based on a demonstrated track record, not on what you or your employers would like to be true.

I just read George Dinwiddie’s interesting take on developer productivity, and I wanted to throw in my own two cents.

You can’t measure it

I agree with a number of others who say that there’s no good measure for developer productivity. There are several basic approaches people use, and all of them have flaws:

• time spent - This is a classic way to measure productivity. How long did people work? If the number is large, things must be good, right?
• apparent effort – Although this is even more flawed, it’s very popular. The “if you ain’t sweatin’, you ain’t workin’” metric is a favorite of seagull managers. But it’s easy to manipulate, and even when people are honest, it’s terribly misleading.
• technical output – This includes things like keystrokes or lines of code produced. As Bill Gates says, “Measuring programming progress by lines of code is like measuring aircraft building progress by weight.”
• functional output – Instead of counting lines of code, you can count features, through mechanisms like function points. Counting fields and data elements is a lot more work than counting lines of code, but it’s not clear the results are much better.
• business value – That’s what we’re after, so it seems like it would be great to track this. And you should. But it’s incredibly difficult to assign that value to individual bits of work, and especially to individual players.

Over the years, I have seen a lot of places try to numerically measure how productive their developers are. I’ve never seen anybody have much success, butI have seen a lot of wasted effort. And worse, I’ve seen a lot of harm. Try to measure individual productivity, for example, and you create a disincentive to help others. Since some of your most productive developers are the ones who mentor others and keep them from wasting time or making messes, it’s easy to drastically reduce productivity just by trying to measure it.

But everybody knows

Does this mean that it’s impossible to know how well a team is doing, or who the top performers are? Not at all.

The team knows

On agile projects, it isn’t individuals who are responsible for delivering. It’s teams. If your team is working together in a room, has tight feedback loops, and delivers frequently to end users, everybody will be forced to work together and interact frequently. Every team member will have a good idea of who is a top performer and whether somebody isn’t pulling their weight. They can’t not know. Whether they’ll tell anybody else is a different thing, which leads to my next point.

Embed reporters and distribute power

There are many advantages to having business people, like product managers and business analysts, in the room with developers. Your products will be better designed, better built, and more efficient. But a side benefit is that there will be somebody management trusts to give them an honest opinion on developer productivity. For this to work well, the business representative should be — both culturally and organizationally — not part of the engineering organization.

It’s also important to give the team matched responsibilities and power relating to this. Involve the team extensively in interviewing and hiring — and also in firing. Make sure they know they’re responsible for total productivity, and give them the authority to make changes they need in that regard. Hint: if they’re not allowed to change the furniture or order new RAM for their machines, they sure won’t think they can pressure or fire a poor performer.

Focus on delivering value

The key though, is to get the whole team focused on whatever purpose the team exists for. As frequently as possible, measure key indicators, like sales, usage, or customer satisfaction. And don’t automatically measure some numbers that go on a web page nobody looks at. Metrics don’t matter unless somebody cares about them. In an Agile context, caring about something is contagious. You should visibly care about the numbers that matter, possibly through a hand-drawn big visible chart. Others will pick up the habit.

If people really care about achieving shared goals, then you won’t have to worry about their performance. They’ll be doing it themselves.

I think most Agile advice boils down to two things:

• Take small steps.
• Work close together.

When I’m struggling, I try to see if there’s any possible way to take smaller steps, or to work closer together with others on the project.

Take small steps to get to done early and often

In software development we work at many different scales, from high-level business goals to specific user actions all the way down to lines of code. Agile methods emphasize getting to done early and often at all of these levels:

• We take one business goal at a time and make sure it’s met before tackling the next one (exemplified by the “sprint goals” of Scrum and “minimum marketable features” as described in the book Software by Numbers).
• We implement one user-level feature in an end-to-end slice (from persistence to user interface) before going on to the next (the “user stories” of Extreme Programming).
• We write and test a few lines of code at a time (the “test-driven development” of Extreme Programming).

At every level, the point is to get one thing “really done” before starting on the next thing.

How do we know how small our steps should be? Experience and intuition are a starting point. If things are flowing along nicely it’s tempting to take slightly larger steps. But as soon as troubles start cropping up, a good bet is to take smaller steps: Breaking work down into smaller pieces while maintaining, or even strengthening, our definition of “done” helps to get things done cleanly and consistently.

On a recent project, the original plan was over-ambitious, allowing only a few months for what was turning out to be a couple years of work with the available resources. So we got together with the project champion and identified some specific short-term business goals: First, we needed to make sure that the first big client, on the verge of signing the contract, would get what they were promised. Second, we wanted to make a marketing announcement about the general availability of the system.

It turned out that meeting both of these goals required much less than the total vision for the software. In that case, they were both driven more by the content that we were going to be publishing through the system than by the end-user software functionality. We went through all the user stories and put a big red dot on each story card that was essential for the first goal, and a red circle on each story card that was essential for the second goal. Taking our velocity into account, it turned out that the “red dots” fit comfortably into the time available before launching with the client, and it was quite satisfying to have such clarity about what to focus on and what not to focus on for those few iterations.

Work close together to enhance communication

On every software project, we need to communicate with the customer about what to do and what’s been done, and we need to communicate with our teammates about how to do what needs to be done. Agile methods suggest frequent and lightweight communication by finding ways to work closer together, rather than staying far apart and communicating through impersonal means.

Barriers to communication can be both physical and process-related. Physical barriers can be anything from cubical walls and long office hallways to separations of hundreds or thousands of miles. Being separated by thousands of miles leads to both time zone challenges and cultural differences.

Extreme Programming (XP) recommends that the customer and the entire team work together in the same physical location. When this seems impractical for a project, it is often tempting to abandon XP or Agile methods altogether. It is also possible to miss the point, by working nearby but avoiding contact with each other. However, keeping in mind the advice to work closer together, we can be creative: We can do video conferencing, we can pair program remotely using screen-sharing tools, and we can at least visit each other from time to time, especially at the start of the project. Meeting each other face-to-face is an extremely powerful way to cut through cultural differences and interpersonal misunderstandings.

One example of a process-related barrier is individual task assignments, since if I’m on the hook to finish a task myself I will be less inclined to take a break to help you. Agile methods usually recommend whole-team estimating and planning, so that the whole team is interested in working close together to get all the tasks done. If one person is stuck, the whole team feels stuck and is quick to rally around the issue.

Regarding another process-related barrier, Agile methods have a reputation for throwing away documentation. But documentation really isn’t the point.

I was once in a meeting with a product manager talking about the requirements documents he liked to write, where at one point he leaned in close, pointed toward the QA group (not represented in the meeting) and whispered, “Those guys don’t even read the spec!” He clearly thought that was the QA group’s problem, because his spec document was clear and complete — he had put a lot of effort into it.

Agile methods do not recommend, “Throw away the requirements documentation.” Agile methods do recommend, “If the requirements documentation is causing pain, find ways to work closer together with the intended audience.” If the QA group isn’t reading the spec, maybe that’s because it’s not helping them do their job. Maybe it’s just not communicating effectively.

One of the most common novice questions about Agile methods is, “When do we do X?”

People always ask this about something important, and I’ve heard it asked about a bunch of things, including design, research, testing, architecture, and optimization. I think that’s a very reasonable question, as other methods have phases for things like that. From the novice perspective, we’ve taken away the phases, so it sure can look like we’ve just discarded everything but the coding.

The agile approach

My answer is that if something is important, you should never stop doing it. Make sure a little of it happens every week, and maybe every day. Close those plan-act-evaluate loops as often as possible:

• Is the user experience important to you? Then have a designer in the room, giving continuous feedback to developers. Do regular user testing. Instrument your production software and release it often, so you can see how well your theories play out.
• Is reliability important to you? Then involve QA as part of the definition of the story. Build your acceptance tests alongside the production code. Write your code test first.
• Is maintainability important to you? Then never stop improving the design of the code through refactoring, promiscuous pairing, and collective ownership of the code.
• Is making the right product important to you? If so, always have a product manager in the room. Make sure people talk about the why of a feature, not just the what.
• Is productivity important to you? Then have the team look back every week and find some way to improve your process. Encourage everybody to always be on the lookout for ways to do things better.

And so on, for everything that you think is important to your project.

The error of phases

A big mistake of phased processes is to think that you can get a great product by ignoring important activities for weeks or months at a time. But people only stay good at things that they do frequently. Even worse, spending months with your back turned on some important facet of your project inevitably harms it. As humans, it’s inescapable that we forget the past and insufficiently understand the future. And new information is always coming in, no matter what phase you’re in.

In truth, creating a cohesive product for an ever-evolving environment and audience requires regular attention to every aspect that matters. Instead of treating some important perspective as in the past or part of the future, bring them all into the present. If some activity really matters to your project, make sure somebody on your team is thinking about it every day.

Anyone with even the slightest exposure to eXtreme Programming has heard the phrase “do the simplest thing that could possibly work.” While this precept is typically applied during the process of writing code, I’ve also heard it mentioned during design discussions. I bring it up myself during estimation sessions as a reminder to estimate the simplest implementation of a user story one can imagine.

I’ve also heard it misquoted as “do the easiest thing that could possibly work” and even “do the quickest thing that could possibly work.”

It’s not necessarily the easiest thing

Although the simplest thing that could possibly work might sometimes also be the easiest thing, it often is not. For example, the easiest thing might be to copy and paste a section of code in order to duplicate its functionality in another part of a program. But is that the simplest thing? I believe it is not.

It’s not necessarily the quickest thing

The simplest thing that could possibly work might sometimes also be the quickest thing, but there aren’t any guarantees. The use of short variable and method names might be quicker (because it saves typing time) than the use of longer, more meaningful names, but is it simpler? Again, I don’t think so.

What does simple mean?

Here’s my working definition: Something that is simple is neither complex (meaning having more parts) nor complicated (meaning difficult to understand).

In the first example above, the easiest thing is not the simplest thing because doing the easiest thing (rubber-stamping the code) introduces complexity.

In the second example above, the quickest thing is not the simplest thing because doing the quickest thing (using short names) introduces complication.

Conclusion

If you want to do the simplest thing that could possibly work, look for a solution that is both easy to understand and lacking in unnecessary complexity.