We did that anyway, obviously, but mostly we were learning, tweeting, chatting, coding, emailing and taking notes at a conference where the wifi actually worked and power strips were easily available. I’m not going to reference all the talks that I went to here, but I have uploaded my notes here. Also, I’ve written a detailed summery of David Heinemeier Hansson’s keynote and a few select talks that I attended for RailsMagazine, but the free PDF-edition won’t be available for another two weeks now freely available for download … slides from all the presentations can be found here.

So what did I take away from RailsConf from the top of my head? The first that springs to mind, is that I got to try RubyMine, a new IDE for Ruby on Rails. At first glance this might look like another Aptana or Netbeans, but the Rails Envy guys gave a stunning demonstration of the power of this new IDE: From TextMate keymapping and navigation, over code completion and refactoring that actually works, to automatic diagramming of models and their relationships, this product looks like a real productivity booster. I’m going to try it out the minute I get home.

Rack kept popping up everywhere. Lots of talks on Rack and Rails Metal (which is really just a wrapper for a “raw” Rack handler), and now Rack suddenly doesn’t feel so scary anymore. It just a way to do cool things on a raw request and/or response without the (full) overhead of the Rails stack. JRuby was another popular topic, as well as clouds – how to deploy to server environment that will seamlessly scale with your needs.

I have been following IronRuby with great interest since the project was started 2-3 years ago, and I was a bit disappointed to hear that the .NET implementation of Ruby is still nowhere near completed. I asked Jimmy Schementi from the Microsoft IronRuby team what took them so long (to be honest, I was a bit rude, sorry Jimmy), and asked that things simply took time. He said that JRuby had been 6 years underway. The positive news, however, was that the IronRuby team is still very committed to making their implementation fully Ruby 1.8 compliant, and that are using Rails to measure their success against.

I’d love to tell you more, but my plane is leaving soon, so you will have to make do with my notes.

Community Day 2009 is free 1-day event May 28 in Copenhagen sponsored by Danish developer communities. Besides my two cents, it also features presentations on Android, Flex, Air, Silverlight, LINQ, Drupal, jQuery and ASP.NET MVC – in other words it looking to be a lot of buzzwords and popular technologies explained. As if that wasn’t enough, the day also include three geek-friendly meals, friendly competitions and lots of free beer.

The course I will be teaching at Copenhagen Business School is called Agile Development in Practice, and builds upon a theoretic platform of agile methods. The course focus is on giving the students practical experience with Scrum and Ruby on Rails through a combination of traditional teaching and working with real-life examples. The course is free to attend for all Danish students, and counts for either 7,5 or 15 ETCS points. Others can also enroll in the course for a fee. The course description is not yet available on the CBS homepage, but I will link to here as soon as it is ready.

But before all that, I myself will hopefully learn many new and useful things about Ruby, Rails and all things geeky at RailsConf in Las Vegas. See you there? :)

Mary Poppendieck started her talk with a story about the dot-com bubble of the 1800’s: Plank roads. She ended the story by asking: Is agile development a plank road? Is agile development just another fad that gets everyone exited for a time, but doesn’t really solve anything in the long run? Poppendieck never really answered this question, but she went on to go through software development from the 1960’s up to our century, concluding that for each decade and each new software development paradigm, large software projects (over a year) was still late, over budget and full of bugs – even today this is the case. She did, however, point out that the opportunity to do small software projects (around 3 months) has increased greatly during these decades, and that large projects is falling out of favor.

Why is software development so hard to manage? Why have we passed through so many paradigms in such a short time span? Poppendieck had three theses on why this might be so:

Thesis 1: Silver Bullet Thinking
People has a tendency to believe that a certain process, method or tool is so great that it will magically solve the problems right away for anyone who use it. They tend to ignore the details of this silver bullet, and just try to apply it directly to their company or project while ignoring what the real world is like. When this happens, the “silver bullet” will just be a fad, and it won’t work.

Thesis 2: Different Contexts
Extremely different approaches to software development exists, from the very rigorous and structured to the very people-driven and agile, and both extremes and a lot between seem to work in different contexts. One approach might be good for one kind of software, but not necessarily for another kind of software. There is no single answer, no single method that works for all contexts.

Thesis 3: Essential Tension
There is an essential tension between getting started with development (coding) right away versus planning and spending time doing design and architecture. The first might get you fast results and early feedback, while the latter might save you from major refactoring and incoherent software. Spending time planning is good, but following a plan you created when you where most ignorant is not. Maybe there is a fundamental tension between these two poles, and maybe projects need to be in the middle, driven by essential tension.

Mary Poppendieck went on to describe how to fail with agile. She listed four mistakes that people are likely to do when trying to adapt agile processes, but she also listed four principles that should be followed to avoid these mistakes:

Mistake #1: Copy What Successful Companies are Doing
Poppendieck referred a lot to the book The Wisdom of Crowds, and quoted the notion of “the instinct to imitate” from book in connection with this mistake. If we let ourselves be influenced too much by existing practice, we will believe the same things are right and make the same mistakes. The solution is to maintain indepence and think for yourself. Don’t believe in all success stories, but look for the flaws and mistakes in them. Just because it is the textbook’s way of doing it, it doesn’t mean that way is good for you. You don’t get world class development by copying best practice, but by inventing it.

Mistake #2: Insist that Everyone Follow a Standard Process
Different people have different views of the world, and different processes will work best for them. For this reason, the agile process should draw on local knowledge, letting people fit the process to their needs locally and constantly improve this process. Again, there is no such thing as a “best practice” for everyone.

Mistake #3: Limit Agile Implementation to Software Development
Poppendieck quoted The Wisdom of Crowds for its notion of “the value of cognitive diversity”. Having a team with software developers only is not very diverse, so you need to gather a group of people from all parts of the organization – creative, testers, operations, etc. The solution is to widen the boundaries and accept that “IT” is part of “The Business” and not outside of it. The team should be responsible for business success, not just technical success, which has no value in itself.

Mistake #4: Avoid Tension
There is a natural tension, not just between programming and planning, but also between marketing people who wants the job finished right now, and developers who want to develop forever. Many other examples could be found. Tension, even conflicts, is what provides a balanced solution, and consensus decisions is nothing to strive for.

Mary Poppendieck concluded her talk by explaining, that if we want to make agile something more than a fad, we must recognize all three theses – stop silver bullet thinking, recognize different contexts and welcome essential tension – while avoiding the common mistakes.

Agilists believe that good practices and processes can improve consistency but that repeatability – in the statistically controlled feedback loop sense of the term – is a fantasy. Unfortunately, it is a fantazy that many corporate executives believe in, and that belief exacerbates the dysfunctionality between product development and management. Executive management has been told that they can have it all, and they want to believe it. They are then disappointed when plans don’t work out. Their solution: more processes and more standards. Are traditionalists willing to commit to this view of repeatability as a problem rather than a solution? Agilists believe that change must be adapted to, that it can’t be planned away. You can have flexibility, or consistency, or some blend of both. But expecting a process or methodology to provide ultimate flexibility and complete predictability at the same stretches the limits of credulity.

Jim Highsmith, Agile Software Development Ecosystems

Happy new year!

The gap between the agile practices and where you are now, feels so big, that it seems impossible to get there.

If you are working with some kind of software development, you have probably considered going agile. Maybe you are in a big, rigid, bureaucratic development methodology today, maybe you are just doing the good old code’n'fix, or perhaps you are working in some kind “agile-ish” way. If you are perfectly happy with how your development process is today, or if you are already doing the real agile thing with XP, Scrum or something else, then this post is not for you.

The nine steps are steps that even very big and heavy organizations can take take towards becoming agile. You don’t become agile overnight – it takes time; especially because agile development, as opposed to many other methodologies, is all about having the right values embedded in the company culture, and less about making documentation in a specific way or using the right tools.

The steps are ordered so they should get harder and harder to implement, but still possible in most organizations. The first four steps are mostly technical, while the other five steps are mostly about better project management. All of them are steps to become agile.

1. Daily builds

Also known as continuous integration, daily builds are all about integrating, linking and compiling the entire code base of the development project automatically. If a full build is very time consuming and resource demanding; this can be done overnight, but if its lighter; it can be done even more often, preferable every time new files are committed to the code base.

Developers develop and develop and develop and 6 months later they finally try to put everything together.

The advantages of daily builds are great for projects of a size where individual developers can’t build everything themselves at their leisure. It results in very early feedback to the developers, and the less time that has passed since the developer wrote code that broke the build, the less expensive is the error to fix, because the developer is still available to the project and has the code in fresh in mind. This is also why big bang integration is such a bad thing – if you only test everything together near the end of the project, you are asking for trouble.

2. Automated smoke tests

If you don’t do test-driven development, if you don’t do any serious unit testing; then automated smoke tests are a place to start. A smoke test can run automatically and will always return either passed or failed. Passed doesn’t mean that code is fantastic or has all the correct logic, it just signals that:

This software is not bad enough to reject

It’s not easy to understand how automated smoke are different from unit tests, but the difference is really just this: They are simpler. If you are already unit-testing with JUnit, NUnit or similar testing frameworks, you are already past automated smoke tests. If you are only testing from you GUI, smoke tests can be a place to start.

Smoke tests are the next early warning layer building upon the daily build. You software might build, but then a smoke test might fail, and you get rapid feedback once again with all its benefits. There is a rule of thumb that says that customers mainly use 20 % of the functionality in the software – your smoke tests should cover these 20 %, and nothing more, really.

3. Implementation by feature

While many large software companies organize and implement each layer in the typical 3-tier architecture by itself, implementation by feature (or by slice) will often be much more efficient. In the typical bottom-up development cycle, architects and developers try to predict every possible need the upper layers might have, which in the end most often results in some of the functionality not being used, and other functionality still lacking, since requirements are constantly changing.

By implementing by feature, that is; developing a slice of user functionality through all layers in the software, you get something done, finished and working, and you don’t write a lot of code you don’t need. If your company is organized around architectural departments, you can try to setting up cross-functional teams for implementing specific features.

4. Test earlier and more often

All serious software developers test their software to a certain extent, but too often, testing is something that doesn’t get started until just before implementation ends, because implementation just keeps going on. This doesn’t give testers a fair chance to test to the software thoroughly, and when they find bugs, the developers will have forgotten most of the rationale behind the buggy code.

While agile methods suggest pair programming as one way to produce code of much higher quality, Rothman suggest a much less time-consuming technique that can be used if pair programming is not a viable option; the develop-test technique:

1. Take two developers that works on similar code (i.e. they can understand each others code)
2. Let them work independently as usual for, say, two hours
3. Then let them hand off the code to each other for a review
4. The developers must each create unit tests for the other persons code, and comment on its general quality
5. Now the code is all unit tested, and each developer has gotten some constructive and early feedback

Again, much more subtle techniques of course exists, but this can be a place to start.

5. Timeboxing requirements

This is one of my favorites. A typical project might take 10 months, and out of that, developers will spend 5 months on requirements and 5 months of design, implementation, testing and all the other tasks that are part of development. It is wrong to use all that time on requirements.

Why? Because even when you spend that much time on the requirements, you will never be a 100 % right. Probably not even 50 % right. Even after 5 months. The solution? Timebox your requirements phase.

Timeboxing: You spend as much time as you have decided to spend on a particular activity, and by definition; you are done.

So make a decision. Set a deadline. If you allocate 3 weeks to gathering requirements, then you are done after these 3 weeks, and hopefully, this has forced you to focus on the important requirements. You can’t gather requirements by yourself either, you have to do it collaboratively with – preferably – representations from all stakeholders present. Perhaps in a cross-functional team.

6. Rank the requirements

It is much better to rank you requirements (1, 2, 3, 4, 5 …) than to just categorize them (high, medium, low). Why? Because when the requirements are just categorized, we often even doesn’t get through all the requirements in the “high” category, and we might just do few from the “medium” category, just because they are easier to implement.

Ranking requirements explains exactly the order the requirements has to be implemented.

A good technique for setting your requirements list in a prioritized order, is to use pairwise ranking. You simply sit down with the important stakeholders and compare the requirements in pairs – is this requirement more or less important than the other? Then, when you have the prioritized list, you could, for instance, freeze requirements 1-10 and implement these, allowing the rest of list to be rearranged and changed. If you go in that direction, you are in fact already in your way to use Scrum.

7. Rolling wave planning

I wish I had a crystal ball. I wish I could see out past two or three weeks. But I’m not very good at that.

With rolling wave planning, you put up major milestones as far as you have to, but you only do detailed planning for the next 3-6 weeks. Rothman recommends 4-5 weeks of detailed planning, since most people like to know what they are doing for the next month or so. The benefit is that you have much easier time when your environment and requirements suddenly change. Let me put in a few more quotes from Rothman’s presentation, since she really explains it much better than I:

You can’t adapt a change, if your plan is set in concrete, but you can adapt a change, if you plan for change

What if the client wants to see a plan? You can make up fiction! You can give the client anything they want – it’s not accurate!

8. Milestones based on deliveries – not functions

When you setup you milestones for the project, they should based on deliveries, that is; code that satisfies given requirements, instead of functions, that is; code that completes certain aspects of the software architecture. This way, you know exactly how far you are from customers point of view, since the customer doesn’t really care about your architecture and layers, but just wants to see his or hers requirements implemented.

The worst thing you can do is to do reporting based on how many percent done the project is. 90 % done is very bad place to be, and you can end up setting there for a long time. So look at deliveries instead. Say: We have delivered code that satisfies the demands for feature XX.

9. Cross-functional teams

Cross-functional teams has been recommended as a solution several times in this post. Functional teams has blind spots, i.e. architectural teams doesn’t think about how their code interfaces with other peoples code, not because they are bad developers – they just don’t think about it.

Cross-functional teams are very helpful at preventing other people from ignoring what’s really going on in the system. When you put people with different areas of expertise together, they are able to create a more holistic solution and do a full delivery.

While it’s probably unrealistic to try to convince the top management in your company that all the development teams should be reorganized to fit to agile development, it is not that hard to put cross-functional teams together across a locked organization structure.

Conclusion

Agile methods sounds great when you read about them – and they probably are great once they are fully implemented into an organization. But it’s not easy to get there. These nine steps are a place to start. I’ll try to convince the smart guys and girls at my company that we should become agile, and these steps can hopefully help me.

I will also be attending the next meeting in the Danish Agile User Group which takes place at November 27th. Join the group if you are in the Greater Copenhagen area and finds this stuff interesting :)

When Obie Fernandez was in Denmark to speak at JAOO, he also met up with a bunch of us from Copenhagen Ruby Brigade at a café in central Copenhagen. While most of the evening was spent completely relaxed just talking, Obie also gave his presentation from RailsConf Chicago, which I missed when I was there – glad I got the chance to hear it in my own hometown.

The presentation was mainly a collection of case stories about projects, where ThoughtWorks has successfully used Ruby on Rails and agile development, with a lot of good points mixed in. To be honest, I can’t recount too many of these points, since Jesper Rønn-Jensen kept buying rounds of drinks to everyone and it was very late in the evening, but I think most of what he said related to Rails can be found in his own summary.

I don’t think the actual presentation material can be found online, since it contains information about clients that’s not 100 % cleared, and it’s a pity, since a lot of what I got out the presentation was from seeing the images of real developers working on real agile development. When I read texts by Kent Beck, Jim Highsmith or some of the other agile gurus, I keep thinking that it sounds almost to good to be true. That no one could actually be doing this stuff for real, because it is just so wonderfully different from how we normally develop software. That’s why it’s nice to see images of people pair programming or having stand-up meetings, while stories on post-its hangs on the whiteboard in the background – people are doing this!

Now, two valuable things I do remember Obie saying, are that:

1. The only really good way to do pair programming, is have a setup with one computer, two mirrored screens and two sets of keyboards and mice that works simultaneously. This way none of the two developers has to sit and twist their neck to see the screen, they can select their own distance to each other and most importantly; they can switch between the coder and observer roles instantaneously. (This later made me think of how ironic the image on the front page of www.pairprogramming.com is.)
2. Traditional unit testing is not a very good fit for many Rails-applications, at least not if they are basically just CRUD-applications without any advanced business logic. This is due to the fact that ActiveRecord – which does the actual crudding – is already thoroughly tested, and that mainly controllers and user interface is added. For this reason, Obie recommended Selenium as the way to test your application, or – better yet – as a way to do test driven development. Selenium allows you test your user interface and application flow in an automated way through your browser, using JavaScript to control the tests.

Ole Jepsen, a Danish expert in agile development (which I don’t have a picture of, so I put myself in instead – just imagine that I’m talking notes while Ole is talking), gave a lesson at my Agile Methods class in a very agile way. After a short introduction of himself and his view on development, he asked us – the students – to come up with subjects about agile development we wanted him talk about. Then each of use got three votes, and the subjects where prioritized after number of votes. “These are our stories as you – the customers – has prioritized them, and I – the supplier – will try do as many of them in this time box as possible, starting out with the most important.” Ole said. Although I have pretty comprehensive notes, I will only dig into what I found was Ole’s most important points:

Pitfalls of agile development in practice
Just like waterfall methods doesn’t work like the text books say they will (at all!), agile methods of course also have their problems and pitfalls when you take them out of (say) Kent Beck’s comfortable descriptions and into the Real World^TM. Ole had an example from a big Danish telecommunication company, where a project which had failed several times, switched to use agile methods (“it has to hurt before they go agile!” was another of Ole’s points) – but not without problems:

• The customer (that is; the user of the system being developed) and the developers do not automatically communicate well, just because you put them in the same room. In this project, the customer and the lead architect held long meetings where customer explained the problem domain. Then the architect would spend two days writing Word documents and afterwards show them to the customer (which was in the same room), asking: “Is this right?”, and would get the answer: “No no, that’s completely wrong!”. In the end, the project manager had to ask the customer to take responsibility of really explaining the needs so the developers could understand it, and the developers, in turn, has to really listen.
• The project was planned in three main iterations, and near the end a fourth was added to catch up with postponed stories. After each iteration, a demonstration was done for the customer so they could see the progress. However, when the third iteration was nearing its end, the developers on the team persuaded the project manager to skip the demonstration, because work at that point was very hectic to finish up the project. They skipped the demonstration, but as a result; the customer started asking questions: Had they made progress? The point is; that in order work without a specific requirements specification, customer and supplier has to trust each other, and the customer has to be able to track the progress.

When go agile, and when to use waterfall
Ole explained that many people like to view the options for choosing a development method as two extremes:

• A large, predictable, complex project where requirements can be specified up front: Waterfall
• A smaller, dynamic (often web-) project: Agile

In Ole’s experience, though, there are no projects that won’t benefit from agile methods one way or another, while there are projects that will just never work with a waterfall model, because requirements are changing all the time. He also pointed out that if your project might fail, it should fail as early as possible.

Architecture in agile development projects
One of the big concerns of experienced developers considering going agile, is what happens to the architecture of the system being developed. Traditionally, one or more architects have laid the groundwork in designing the entire object model before the programmers where allowed to begin their part of the work. A traditional architect would consider it an embarrassment if his design has to be changed because of flaws or unforeseen things, but agile development does not allow much time for BDUF (Big Design Up Front).

The argument against BDUF is another acronym; YAGNI (You Ain’t Gonna Need It), which stresses the point that in trying to make a design that covers any conceivable aspect of the system, you are actually wasting a lot time designing – and often also implementing – stuff that will never be used. However, Ole explained, agile methods are in no opposed to architecture, it is just that the architect design, drawings and diagrams becomes less important, while the architects themselves becomes much more important.

Some agile practitioners are of the opinion that no dedicated architects are needed at all; all programmers are architects and visa versa. The purpose is that everyone “owns” the entire code base, so everyone is capable of making changes anywhere. Ole, however, does not like the idea of everyone making architecture decisions jointly – he prefers that a chief architect makes the final decisions and is responsible for keeping the design consistent and clear.

The above is just a fraction of all the exciting stuff I’m learning about agile development at the moment, and remember: You are not agile just because you code ad-hoc without planning your stuff in a code-and-fix way. Agile development requires both discipline and experience.

Anyway, I’m off to London for the next week and I’m not bringing my laptop, since I’m afraid of something happening to my precious, yet fragile Macbook Pro – so I’ll be offline for a week…