Tuesday, September 21, 2010

Talking Scrum, Walking Scrum, Hoping

I corresponded recently with my friend, Josh. He is a manager at a company "down South" and he ranted about a few frustrations he had with some (many??) of his company's managers and workers' about their ignorance of agile and Scrum. I can understand his feelings and why his patience is being tested. His email:

Hey Tom,

Just some thoughts. I am increasingly amazed at our people’s ignorance of agile and modern software development in general. This applies to our management and to our IT workers. We build software here for a living and we ought to be doing it the best that we can. Modern software development involves building and delivering quality software as quickly as possible using excellent practices and principles. To do this effectively and efficiently, we rely on the disciplined use of the agile framework of Scrum and proven engineering practices (at least we say we do.)

As you and I know, Scrum was created in 1993 by Jeff Sutherland and Ken Schwaber. The idea was to focus on the fast delivery of quality software that could be inspected for appropriateness by the customer at regular intervals and, if desired by the customer, shipped immediately. This was a radical departure from the “completely mix and bake” approaches that did not deliver the product until the end of a linear development process. This process involved identification and agreement of all requirements at the beginning, then development of a full design that was elaborately documented, and then a full development cycle that created the software in its entirety based upon the specifications identified in the requirements and design.

This was typically done through a unit and “white box” (code coverage) test level with intermittent incremental regression testing done. At some point, typically well into the development cycle, functional / system (i.e. black box) testing began. At this point, the defect ping pong game started. This caused disruption in the continuity of the programming because developers had to compete with 2 priorities: complete coding or fix defects. Typically, some number of developers were assigned to fixing the defects and this caused problems with coordination of the work and completion of both application functionality and resolution of defects. This describes the legacy of how we did work at our company up until I brought agile in a few years ago.

The software industry continued to practice this insanity and couldn’t understand why it could not deliver quality software at predicted (or demanded) dates. Same with our company. Frustrations increased and in the late 90s the web put even more pressure on development people. Failure to deliver software didn’t just mean that a date was missed and disappointment (anger??) resulted, but more and more revenue was lost and stature in the market place was diminished. Agile permitted for incremental delivery of the software based upon the premise that customers did not know exactly what they wanted or how much. By providing them with the opportunity to see and actually experience the software as it is being built, they can judge how much is enough.

At our company, people were fearful that this would open the door to “scope creep” but the considerations of time and money still come into play (unless there is a infinite amount of both.) So, the customer has to decide what is most critical in the application and whether investment in any functionality is worth the cost (a positive net present value.) With Scrum, the customer can decide when something is good enough, or even stop a project early if value is not being delivered (stopping the pushing good money after bad.) To me, that is a no brainer.

So, these various philosophical perspectives of agile were coalesced into the Agile Manifesto, which established that agile is not a process but a work philosophy and cultural guide. An organization cannot “do agile” but it can "be agile" and do its work in a manner that supports and is supported by the 4 values and 12 principles identified in the Manifesto. But, working in this way takes discipline, adjustment and acceptance of this philosophical foundation. You can’t just “talk agile” but you have to “walk agile” and we continue to have a helluva time with this.

The agile urban myths abound and it typically takes much more discipline and better engineering practices than the old linear methods. This puts pressure on the organization and its people. It requires change in thinking and challenges traditional ways and practices (e.g. using working software as the measure of progress, not the production of documentation and accepting change and modifying a plan rather than strictly adhering to it and making it difficult to change it through a rigorous and oppressive change control process.) Jeff Patton argues that agile is actually a culture not some kind of process in this blog post: Agile is More Culture than Process. Scott Ambler discussed the values and principles in detail in his essay Examining the Agile Manifesto.

I’ve harped enough, but the bottom line is that people need to learn about and understand agile. That means going out and googling it, read about it, and think critically about it. I firmly believe that agile isn’t going away here (despite the arguments that it isn't really here now); in fact, it is here to stay in my opinion. I do my best to dispel the myths to the folks I know such as Scrum projects don’t do documentation (all projects should documentation that is valuable to the team and to a servicing area, or required for compliance reasons – and that should be minimal.)

We typically don’t have coordinators, leads, etc. on Scrum projects because that adds to overhead and inhibits the notion of cross-functional, self-directed, self-organizing teams. We want to maximize the amount of “real work” that people do (e.g. coding, testing, etc.) IMHO, we have way too many of these coordinators, etc. and not enough carpenters (builders.) People who like carpentry (i.e. coding) and contribute to building products are told that they can’t be promoted to senior engineer unless they are willing to forego coding and get into coordination / lead positions. What nonsense!!

Our company would serve itself well to align with Deming when it comes to his 14 principles, especially as they relate to management (and agile supports these):

7. Adopt and institute leadership for the management of people, recognizing their different abilities, capabilities, and aspiration. The aim of leadership should be to help people…do a better job. Leadership of management is in need of overhaul, as well as leadership of workers.

8. Drive out fear and build trust so that everyone can work effectively.

11. Eliminate numerical goals, numerical quotas and management by objectives. Substitute leadership

Well, now you know how I really feel and think, Tom. Thanks for letting me vent and get this out. By the way, you might appreciate Dilbert yesterday: