Reliable Estimates Using Agile

Agile tools shine at managing projects and providing quality. But what is required to “manage” a project to success?

 What is the purpose of a project?

The purpose of a project is to provide a deliverable: a product or service. A deliverable is defined using specified requirements. Quality is meeting the specified requirements using measurements based on quantitative objective evidence.

To provide a deliverable, the following are required:

• Tools necessary to provide the deliverable such as buildings, wrenches, computers, specifications, education material, etc.
• Instructions provided by operation procedures, manuals, safety standards, specifications, requirements, etc.
• Qualifications of those who are providing the deliverable. Are the people qualified? Are they being trained so they have the necessary knowledge required to be “qualified”? Are they competent (qualified and have “common sense”)?
• Accountability through responsibility and authority are required of those who are providing the deliverable.

When an organization comes up short on any of these key attributes those involved will try to fill in the gaps:

• Lacking Tools - Those involved will attempt to use the tools they have to “do their best” to provide the deliverable.
• Lacking Qualifications - Those involved will attempt to work within the knowledge they already have to provide the deliverable.
• Lacking Instructions - Those involved will attempt to estimate and then assume what is required to provide the deliverable.
• Lacking Accountability - Without accountability, there is no incentive to provide the deliverable: everyone involved must be made responsible for meeting the deliverable.
• Lacking Authority - Without authority, those involved will not be able to remove hurdles that are stopping them from providing the deliverable.

Those involved then meet their own expectations, as opposed to the key stake holders expectations, and claim success against “overwhelming odds”! This means there is a gap between what is delivered and what was promised to be delivered: the specifications. What you have then is project failure.

How Does Agile Assure Success?

The following are just a few examples of how agile assures that a project is successful:

• The team is responsible for building the project. They have the authority to request that the Agile Master remove impediments. Examples include requesting any missing or required tools, instructions and/or missing qualifications.
• The Agile Master is responsible for assuring the agile rules, agreed upon by everyone, are followed. The Agile Master has the authority to act in all cases where the rules are not followed. For example, they can ask the CEO to leave a daily iteration meeting if the CEO isn’t following the rules. This is a lot of authority (of course, the CEO can always fire the Agile Master).
• The product owner is responsible for the specifications of the project. The have the authority to get any information required provide the specifications. This includes working with the key stake holders or key people who are knowledgeable about the business domain.
• For software, agile requires that a detailed definition of “done” is specified so everyone involved knows just what it means when the sentence “We are done” is uttered.
• The team commits to complete the deliverable in a given period of time (called an iteration) making the team accountable. However, this is based on estimations made by the team that must be accepted by the product owner (after all, they are the ones committing and being held accountable for the actual deliverable). They are also given a lot of authority to assure they can meet their commitments.

The key thing to note from these examples is that everyone has responsibility clearly defined by the agile rules and the authority to execute that responsibility. When software is the deliverable, agile defines in great detail the tools required (such as test driven development, regression testing, pair programming, agile estimation, iteration planning meetings, etc.).

What I’ve written above is based on the Kaizen (http://en.wikipedia.org/wiki/Kaizen) philosophy of continuous improvement. One of the reasons why agile works so well, and specifically agile management concepts, is that it is also based on the Kaizen philosophy. From here on out, I will focus on key aspects of agile and how they fit in with assuring a successful project: providing a quality product.

[ED: you may notice a lot of similarity between the East Agile Master and a SCRUM Master]

Some things on the Internet can disappear in a flash, leaving no trace of their ever having existed. But a lot of the web remains permanently archived. For better or worse, this can remain a part of history.

In 1996, Brewster Kahle created an archive of the web with part of the fortune he made from Alexa. See Archive.org. He had retired into the modest life of a "librarian" as he described himself then. Today, because of his work, and that of others, and similar projects, we can see what the web was like in the past.

As an example, you can see what our website (under our original Stanyan Group brand) looked like back in time.

2002

2001

1999

(N.B. The Stanyan Group was incorporated in California in July 1997).

One of the great aspects of Agile is that it can be continually reviewed and improved through retrospection. At East Agile, we apply Scrum, an Agile management process, but with a few changes. Agile itself is a collection of tools that are available to Agile users (Pair Programming, Standup meetings, Retrospective Meetings, Test Driven Development, etc.). XP (http://www.agilealliance.org/), Scrum (http://www.scrumalliance.org/ ) and Crystal (http://alistair.cockburn.us/Crystal+methodologies ) are examples of branded systems that draw from the set of Agile tools available. As such, there is nothing wrong, and it is quite Agile, to draw the best from these branded Agile systems (just as the founders of Agile did when they choose which tools were most Agile when engineering).

Scrum, as it stands, expects easy access to the Product Owner, all Stake Holders, the Scrum Master and the Team. Scrum also suggests that a minimum iteration should be around 2 weeks. Extreme programming feels iterations can be quicker. I’m not saying there is anything wrong with Scrum as it stands. We greatly appreciate the transparency and consistency that Scrum brings to the table. But our Product Owners are often thousands of miles away.

I am going to blog about agile tools used by Scrum and how we implement those same tools within East Agile. Basically, we have been able to make Scrum work when the Product Owner is thousands of miles away: without losing transparency. We are also able to use Scrum and still deliver on a daily basis. I am sure we aren’t the first to do this but this is our story.

I think it is important to remember the Agile Manifesto (http://agilemanifesto.org/) which says:

•  Individuals and interactions over processes and tools
•  Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

At East Agile, every decision we’ve made in implementing Agile has kept this manifesto in mind.

I think it is also important to give special mention to Ken Schwaber(http://www.controlchaos.com/) and Jeff Sutherland(http://scrum.jeffsutherland.com/) for bringing us Scrum.

[ED: See Kent Beck's seminal 1999 book: Extreme Programming Explained: Embrace Change. Addison-Wesley. It coins eXtreme Programming (XP). Follow @KentBeck on Twitter]

It is very important for us to not compromise on paired programming.

eXtreme Programming, or XP, (especially at East Agile) is an engineering process designed to create a consistent, reliable, scalable and low risk software development environment that reduces risks especially in already unstructured, uncertain and risky business ventures. Not doing full paired programming can introduce the potential to complete a project faster, but also introduces a very significant risk of catastrophic failure. We never choose to make the speed in exchange for substantially increased risk.

That is not to stay that we must pair program 100% of the time. There will be times when it makes sense for a team to split their focus or for individuals to work alone. Examples of this include (1) reading and researching, (2) chores such as system configuration, (3) simple bug fixes in deployed systems, or (4) one member of the pair is away for a brief unrelated purpose. The important difference is that the team comes back together and works as a pair on primary development. At this point their separate experience and learning becomes fused together into a common understanding related to the project at hand.

If developers work independently on separate parts of an application on a continuous basis, even if they happen to be sitting at the same desk when they do that, then very important problems from solo-programming start to emerge.  (1) To the degree that each thread of work is separate, it happens without continuous code review. (2) The same is true about collaboration. Even occasional collaboration on these matters becomes less effective and less insightful because the other person has not built up the mental map of the code base that comes from being involved in the full development process.

And (3) if a developer is sick, quits, or gets fired, there will emerge large gaps in understanding of the code base and large sets of knowledge that are missing from the remaining team. Turnover can simply not be avoided; it happens everywhere. Indeed, the original developer does not even have to leave; that person could be engaged in another more urgent project, or involved in another completely unrelated role in another part of the company. Furthermore, programmers often assert that they would find it easier (or prefer) to rewrite entire applications rather than reverse engineer or try to figure out code that they did not write themselves. Not working in a pair on all major aspects of a project introduces big blocks of risk (bad things that could happen unpredictably at any time, and often at the worst possible time) versus the alternative of potentially slower development that is nonetheless much more predictable and dependable.

As an example, consider that earlier this year we spent about a month developing an application for a client with a single pair of developers. During the course of that month, we had two developers leave the project for various reasons. In the end, four programmers worked on the project, and only one was around for the entire duration. However, development hardly slowed despite the disruptions. We completed the project on schedule. The client noticed no interruptions or problems. And when the client came back to us to do more work a couple months later, we still had someone around who had worked on the entire project. This would not have happened without paired programming.

I would assert that if people are not working productively in a paired programming environment, that the problem is not paired programming. (1) Some people do not work well with other people. They may need to find a company that either deliberately or unwittingly is willing to let them work alone creating applications that are poorly understood by others and that have a high risk of never being completed or of being abandoned because the original developer has moved on. I would also suggest that people who do not work well with others might also have a higher likelihood of moving on, even halfway through a project. (2) Sometimes the problem is that developers are not paired properly with people at their same level. A very fast developer will find it hard to work with a slow developer. And a junior developer will have a hard time adding value to a senior developer. Sometimes a person who is unfamiliar with the programming language, environment, or application will have a hard time making contributions (although this pairing can be an excellent way to bring an otherwise productive person up to speed). And (3) there will be exceptional people and circumstances where someone working alone gets more done just as reliably and maintainably as a pair. But there are fewer people who are this exceptional than there are people who think they are this exceptional. Firms and projects take a risk when they decide that someone does not need to pair program because they are exceptional in this regard.

myBrain 

http://apps.facebook.com/mybrain

singlepostcard 

http://apps.facebook.com/singlepostcard

socialscore 

matchthis 

We support the Facebook application development community by organizing presentations in Silicon Valley through our Facebook Application Development Meetup. The group consists of more than 1400 members and has met at least monthly since it was founded by Lawrence Sinclair, of East Agile, on July 2008.

Eric Hosick's first book should appear on Amazon in a few days. It is a careful distillation of the core object oriented programming concepts every programmer should know. Eric recently joined the East Agile development team, and I congratulate him on his contribution to the larger developer community.

See: Amazon.com Kindle edition

We are attending the Banking Vietnam conference in Hanoi.  One key lesson from this is that business analytics and profitability reporting are important missing parts of Vietnamese banking infrastructure. There is also a strong interest in risk management issues and helping Vietnam move from a cash economy to one based on electronic and formal payment mechanisms.

I was rather surprised today to read that data stored on "cloud" services might not have the same legal privacy protections in the US that data has when stored inside your home (or business). This came to my attention after a Wired Magazine article (April 20, 2010) about Google's recent openness about government worldwide requests for information.

A broad consortium of tech companies and privacy groups recently announced a push to modernize the nation’s privacy laws so that data stored by third parties, especially by so-called cloud computing services like Gmail, are treated just like data stored on citizens’ home computers. Currently, e-mails stored online lose much of their legal protection after 6 months, and the Justice Department recently tried to get at unopened mail online without having to get a proper search warrant.

The principle is that user data left online more than six months can be considered "abandoned" and so open to less restrictive government scrutiny. The primary, or at least initial, target seems to have been email. But the law could apply to phographs, usage and other behavioral data, friends, and status messages. As far as I can tell, this applies to just about any personal data stored online.

What to do about it?

One way to resolve this issue would be to insist on storing data online only in encrypted form, with all decryption done on the client side.  Currently, to ensure security, network connections between client and server are encrypted. But the data is then stored in clear, unencrypted form, exposing it to employees, theft, and government access at the host company premises.  This could help with some data, intended for the owners eyes only, such as email and personal documents. But this would not work easily for data that is intended to be shared with a limited audience. And this would not protect behavioral data, such as login times and places, and purchases.

As a stop gap measure, individuals and companies should consider deleting all data, including behavioral data as it grows older than 6 months.

One could also lobby to get the laws changed. But there are a lot of competing interests, and success is not inevitable, or likely to be rapid. However, an important solution should be legal, and some action in that direction is starting to take shape.

East Agile one of Palm's recommended webOS developers in 2009. During their validation process, we created an photograph album application that lets users drill down into the EXIF tag information that describes the characteristics of their photos. Palm offered to place this application on their store, but instead we opted to make it open source in order to help the developer community. The code has been available on github since January 2010.