On June 24th Mary Poppendieck gave a free talk at the IT University in Copenhagen arranged by BestBrains. The title of the talk was “Is Agile a Fad?” with the subtitle “Will Agile Software Development End Up On the Dumping Grounds of History?”. This dramatically titled talk gathered more than a 100 agile enthusiasts in a packed auditorium, and I spotted several members from Danish Agile User Group and Copenhagen Ruby Brigade, as well as both students and teachers from Copenhagen Business School and, of course, ITU.

Mary Poppendieck started her talk with a story about the dot-com bubble of the 1800′s: Plank roads. She ended the story by asking: Is agile development a plank road? Is agile development just another fad that gets everyone exited for a time, but doesn’t really solve anything in the long run? Poppendieck never really answered this question, but she went on to go through software development from the 1960′s up to our century, concluding that for each decade and each new software development paradigm, large software projects (over a year) was still late, over budget and full of bugs – even today this is the case. She did, however, point out that the opportunity to do small software projects (around 3 months) has increased greatly during these decades, and that large projects is falling out of favor.

Continue reading →

Agilists believe that good practices and processes can improve consistency but that repeatability – in the statistically controlled feedback loop sense of the term – is a fantasy. Unfortunately, it is a fantazy that many corporate executives believe in, and that belief exacerbates the dysfunctionality between product development and management. Executive management has been told that they can have it all, and they want to believe it. They are then disappointed when plans don’t work out. Their solution: more processes and more standards. Are traditionalists willing to commit to this view of repeatability as a problem rather than a solution? Agilists believe that change must be adapted to, that it can’t be planned away. You can have flexibility, or consistency, or some blend of both. But expecting a process or methodology to provide ultimate flexibility and complete predictability at the same stretches the limits of credulity.

Jim Highsmith, Agile Software Development Ecosystems

Happy new year!

This post describes nine steps you company or department can take towards more agile development. If the title seems familiar, it is because the suggestions in this post are not my own, but rather the wisdom of an American woman who calls herself a pragmatic consultant; Johanna Rothman. Rothman did a presentation at Agile Israel user group in March 2005 with this very title, and was generous enough to let the user group record and publish her presentation as a podcast. Recently I did a presentation in my Agile Methods class at school summarizing Rothman’s nine steps, and since it seems she hasn’t written a specific article herself about this good advice, I thought I would do it. I know the podcast is almost two years old, but believe me; this stuff is still highly relevant.

The gap between the agile practices and where you are now, feels so big, that it seems impossible to get there.

If you are working with some kind of software development, you have probably considered going agile. Maybe you are in a big, rigid, bureaucratic development methodology today, maybe you are just doing the good old code’n'fix, or perhaps you are working in some kind “agile-ish” way. If you are perfectly happy with how your development process is today, or if you are already doing the real agile thing with XP, Scrum or something else, then this post is not for you.

The nine steps are steps that even very big and heavy organizations can take take towards becoming agile. You don’t become agile overnight – it takes time; especially because agile development, as opposed to many other methodologies, is all about having the right values embedded in the company culture, and less about making documentation in a specific way or using the right tools.

The steps are ordered so they should get harder and harder to implement, but still possible in most organizations. The first four steps are mostly technical, while the other five steps are mostly about better project management. All of them are steps to become agile.

1. Daily builds

Also known as continuous integration, daily builds are all about integrating, linking and compiling the entire code base of the development project automatically. If a full build is very time consuming and resource demanding; this can be done overnight, but if its lighter; it can be done even more often, preferable every time new files are committed to the code base.

Developers develop and develop and develop and 6 months later they finally try to put everything together.

The advantages of daily builds are great for projects of a size where individual developers can’t build everything themselves at their leisure. It results in very early feedback to the developers, and the less time that has passed since the developer wrote code that broke the build, the less expensive is the error to fix, because the developer is still available to the project and has the code in fresh in mind. This is also why big bang integration is such a bad thing – if you only test everything together near the end of the project, you are asking for trouble.

2. Automated smoke tests

If you don’t do test-driven development, if you don’t do any serious unit testing; then automated smoke tests are a place to start. A smoke test can run automatically and will always return either passed or failed. Passed doesn’t mean that code is fantastic or has all the correct logic, it just signals that:

This software is not bad enough to reject

It’s not easy to understand how automated smoke are different from unit tests, but the difference is really just this: They are simpler. If you are already unit-testing with JUnit, NUnit or similar testing frameworks, you are already past automated smoke tests. If you are only testing from you GUI, smoke tests can be a place to start.

Smoke tests are the next early warning layer building upon the daily build. You software might build, but then a smoke test might fail, and you get rapid feedback once again with all its benefits. There is a rule of thumb that says that customers mainly use 20 % of the functionality in the software – your smoke tests should cover these 20 %, and nothing more, really.

3. Implementation by feature

While many large software companies organize and implement each layer in the typical 3-tier architecture by itself, implementation by feature (or by slice) will often be much more efficient. In the typical bottom-up development cycle, architects and developers try to predict every possible need the upper layers might have, which in the end most often results in some of the functionality not being used, and other functionality still lacking, since requirements are constantly changing.

By implementing by feature, that is; developing a slice of user functionality through all layers in the software, you get something done, finished and working, and you don’t write a lot of code you don’t need. If your company is organized around architectural departments, you can try to setting up cross-functional teams for implementing specific features.

4. Test earlier and more often

All serious software developers test their software to a certain extent, but too often, testing is something that doesn’t get started until just before implementation ends, because implementation just keeps going on. This doesn’t give testers a fair chance to test to the software thoroughly, and when they find bugs, the developers will have forgotten most of the rationale behind the buggy code.

While agile methods suggest pair programming as one way to produce code of much higher quality, Rothman suggest a much less time-consuming technique that can be used if pair programming is not a viable option; the develop-test technique:

1. Take two developers that works on similar code (i.e. they can understand each others code)
2. Let them work independently as usual for, say, two hours
3. Then let them hand off the code to each other for a review
4. The developers must each create unit tests for the other persons code, and comment on its general quality
5. Now the code is all unit tested, and each developer has gotten some constructive and early feedback

Again, much more subtle techniques of course exists, but this can be a place to start.

5. Timeboxing requirements

This is one of my favorites. A typical project might take 10 months, and out of that, developers will spend 5 months on requirements and 5 months of design, implementation, testing and all the other tasks that are part of development. It is wrong to use all that time on requirements.

Why? Because even when you spend that much time on the requirements, you will never be a 100 % right. Probably not even 50 % right. Even after 5 months. The solution? Timebox your requirements phase.

Timeboxing: You spend as much time as you have decided to spend on a particular activity, and by definition; you are done.

So make a decision. Set a deadline. If you allocate 3 weeks to gathering requirements, then you are done after these 3 weeks, and hopefully, this has forced you to focus on the important requirements. You can’t gather requirements by yourself either, you have to do it collaboratively with – preferably – representations from all stakeholders present. Perhaps in a cross-functional team.

6. Rank the requirements

It is much better to rank you requirements (1, 2, 3, 4, 5 …) than to just categorize them (high, medium, low). Why? Because when the requirements are just categorized, we often even doesn’t get through all the requirements in the “high” category, and we might just do few from the “medium” category, just because they are easier to implement.

Ranking requirements explains exactly the order the requirements has to be implemented.

A good technique for setting your requirements list in a prioritized order, is to use pairwise ranking. You simply sit down with the important stakeholders and compare the requirements in pairs – is this requirement more or less important than the other? Then, when you have the prioritized list, you could, for instance, freeze requirements 1-10 and implement these, allowing the rest of list to be rearranged and changed. If you go in that direction, you are in fact already in your way to use Scrum.

7. Rolling wave planning

I wish I had a crystal ball. I wish I could see out past two or three weeks. But I’m not very good at that.

With rolling wave planning, you put up major milestones as far as you have to, but you only do detailed planning for the next 3-6 weeks. Rothman recommends 4-5 weeks of detailed planning, since most people like to know what they are doing for the next month or so. The benefit is that you have much easier time when your environment and requirements suddenly change. Let me put in a few more quotes from Rothman’s presentation, since she really explains it much better than I:

You can’t adapt a change, if your plan is set in concrete, but you can adapt a change, if you plan for change

What if the client wants to see a plan? You can make up fiction! You can give the client anything they want – it’s not accurate!

8. Milestones based on deliveries – not functions

When you setup you milestones for the project, they should based on deliveries, that is; code that satisfies given requirements, instead of functions, that is; code that completes certain aspects of the software architecture. This way, you know exactly how far you are from customers point of view, since the customer doesn’t really care about your architecture and layers, but just wants to see his or hers requirements implemented.

The worst thing you can do is to do reporting based on how many percent done the project is. 90 % done is very bad place to be, and you can end up setting there for a long time. So look at deliveries instead. Say: We have delivered code that satisfies the demands for feature XX.

9. Cross-functional teams

Cross-functional teams has been recommended as a solution several times in this post. Functional teams has blind spots, i.e. architectural teams doesn’t think about how their code interfaces with other peoples code, not because they are bad developers – they just don’t think about it.

Cross-functional teams are very helpful at preventing other people from ignoring what’s really going on in the system. When you put people with different areas of expertise together, they are able to create a more holistic solution and do a full delivery.

While it’s probably unrealistic to try to convince the top management in your company that all the development teams should be reorganized to fit to agile development, it is not that hard to put cross-functional teams together across a locked organization structure.

Conclusion

Agile methods sounds great when you read about them – and they probably are great once they are fully implemented into an organization. But it’s not easy to get there. These nine steps are a place to start. I’ll try to convince the smart guys and girls at my company that we should become agile, and these steps can hopefully help me.

I will also be attending the next meeting in the Danish Agile User Group which takes place at November 27th. Join the group if you are in the Greater Copenhagen area and finds this stuff interesting

I have a great class in school called Agile Methods, but while our teacher keeps the lessons interesting by letting us discuss a lot, I find learning about agile development really inspirational when I get the opportunity to hear about actual agile development projects from real-life practitioners. I’ve had two such opportunities recently and have met Obie Fernandez, a Rails and agile practitioner from ThoughtWorks, and Ole Jepsen, freelance agile consultant and co founder of Agile Project Leadership Network.

When Obie Fernandez was in Denmark to speak at JAOO, he also met up with a bunch of us from Copenhagen Ruby Brigade at a caf̩ in central Copenhagen. While most of the evening was spent completely relaxed just talking, Obie also gave his presentation from RailsConf Chicago, which I missed when I was there Рglad I got the chance to hear it in my own hometown.

The presentation was mainly a collection of case stories about projects, where ThoughtWorks has successfully used Ruby on Rails and agile development, with a lot of good points mixed in. To be honest, I can’t recount too many of these points, since Jesper Rønn-Jensen kept buying rounds of drinks to everyone and it was very late in the evening, but I think most of what he said related to Rails can be found in his own summary.

I don’t think the actual presentation material can be found online, since it contains information about clients that’s not 100 % cleared, and it’s a pity, since a lot of what I got out the presentation was from seeing the images of real developers working on real agile development. When I read texts by Kent Beck, Jim Highsmith or some of the other agile gurus, I keep thinking that it sounds almost to good to be true. That no one could actually be doing this stuff for real, because it is just so wonderfully different from how we normally develop software. That’s why it’s nice to see images of people pair programming or having stand-up meetings, while stories on post-its hangs on the whiteboard in the background – people are doing this!

Now, two valuable things I do remember Obie saying, are that:

1. The only really good way to do pair programming, is have a setup with one computer, two mirrored screens and two sets of keyboards and mice that works simultaneously. This way none of the two developers has to sit and twist their neck to see the screen, they can select their own distance to each other and most importantly; they can switch between the coder and observer roles instantaneously. (This later made me think of how ironic the image on the front page of www.pairprogramming.com is.)
2. Traditional unit testing is not a very good fit for many Rails-applications, at least not if they are basically just CRUD-applications without any advanced business logic. This is due to the fact that ActiveRecord – which does the actual crudding – is already thoroughly tested, and that mainly controllers and user interface is added. For this reason, Obie recommended Selenium as the way to test your application, or – better yet – as a way to do test driven development. Selenium allows you test your user interface and application flow in an automated way through your browser, using JavaScript to control the tests.

Ole Jepsen, a Danish expert in agile development (which I don’t have a picture of, so I put myself in instead – just imagine that I’m talking notes while Ole is talking), gave a lesson at my Agile Methods class in a very agile way. After a short introduction of himself and his view on development, he asked us – the students – to come up with subjects about agile development we wanted him talk about. Then each of use got three votes, and the subjects where prioritized after number of votes. “These are our stories as you – the customers – has prioritized them, and I – the supplier – will try do as many of them in this time box as possible, starting out with the most important.” Ole said. Although I have pretty comprehensive notes, I will only dig into what I found was Ole’s most important points:

Pitfalls of agile development in practice
Just like waterfall methods doesn’t work like the text books say they will (at all!), agile methods of course also have their problems and pitfalls when you take them out of (say) Kent Beck’s comfortable descriptions and into the Real World^TM. Ole had an example from a big Danish telecommunication company, where a project which had failed several times, switched to use agile methods (“it has to hurt before they go agile!” was another of Ole’s points) – but not without problems:

• The customer (that is; the user of the system being developed) and the developers do not automatically communicate well, just because you put them in the same room. In this project, the customer and the lead architect held long meetings where customer explained the problem domain. Then the architect would spend two days writing Word documents and afterwards show them to the customer (which was in the same room), asking: “Is this right?”, and would get the answer: “No no, that’s completely wrong!”. In the end, the project manager had to ask the customer to take responsibility of really explaining the needs so the developers could understand it, and the developers, in turn, has to really listen.
• The project was planned in three main iterations, and near the end a fourth was added to catch up with postponed stories. After each iteration, a demonstration was done for the customer so they could see the progress. However, when the third iteration was nearing its end, the developers on the team persuaded the project manager to skip the demonstration, because work at that point was very hectic to finish up the project. They skipped the demonstration, but as a result; the customer started asking questions: Had they made progress? The point is; that in order work without a specific requirements specification, customer and supplier has to trust each other, and the customer has to be able to track the progress.

When go agile, and when to use waterfall
Ole explained that many people like to view the options for choosing a development method as two extremes:

• A large, predictable, complex project where requirements can be specified up front: Waterfall
• A smaller, dynamic (often web-) project: Agile

In Ole’s experience, though, there are no projects that won’t benefit from agile methods one way or another, while there are projects that will just never work with a waterfall model, because requirements are changing all the time. He also pointed out that if your project might fail, it should fail as early as possible.

Architecture in agile development projects
One of the big concerns of experienced developers considering going agile, is what happens to the architecture of the system being developed. Traditionally, one or more architects have laid the groundwork in designing the entire object model before the programmers where allowed to begin their part of the work. A traditional architect would consider it an embarrassment if his design has to be changed because of flaws or unforeseen things, but agile development does not allow much time for BDUF (Big Design Up Front).

The argument against BDUF is another acronym; YAGNI (You Ain’t Gonna Need It), which stresses the point that in trying to make a design that covers any conceivable aspect of the system, you are actually wasting a lot time designing – and often also implementing – stuff that will never be used. However, Ole explained, agile methods are in no opposed to architecture, it is just that the architect design, drawings and diagrams becomes less important, while the architects themselves becomes much more important.

Some agile practitioners are of the opinion that no dedicated architects are needed at all; all programmers are architects and visa versa. The purpose is that everyone “owns” the entire code base, so everyone is capable of making changes anywhere. Ole, however, does not like the idea of everyone making architecture decisions jointly – he prefers that a chief architect makes the final decisions and is responsible for keeping the design consistent and clear.

The above is just a fraction of all the exciting stuff I’m learning about agile development at the moment, and remember: You are not agile just because you code ad-hoc without planning your stuff in a code-and-fix way. Agile development requires both discipline and experience.

Anyway, I’m off to London for the next week and I’m not bringing my laptop, since I’m afraid of something happening to my precious, yet fragile Macbook Pro – so I’ll be offline for a week…