In the beginning there was chaos. On the first day God initiated the project, on the second day defined requirements, then design and you know how the story ends. On the seventh day God looked at the beautiful waterfall He had created, and saw it was good. But paradise did not last long, and as technology started to accelerate and markets could not wait years to get technology solutions, Adam the developer decided to eat of the tree of wisdom, and Agile was born. Initially, it was perceived by the higher powers as “cowboy development” and they were expelled from the garden and confined to start-ups. But Agile managed to thrive and made its way back into paradise, only to find that the higher powers expect it to pass, like any other mortal, through the Gates of Hell.
When Dr. Robert Cooper, a Canadian professor at Mc Master University, came up with the Stage-Gate process thirty years ago, his focus was, and still is, innovation and product development, particularly in manufacturing. However, structured organizations like financial services, telecommunications, etc.; with significant IT portfolios saw the potential and adopted some form of a Stage-Gate process as the core of their portfolio governance. Today, around 80% of large organizations in North America use the Stage-Gate process. The success of this approach to govern IT project approvals was based on the perfect fit with the waterfall process which, at the time, was the only mode of delivery that existed, besides “cowboy development” that is still healthy and kicking.
As Agile has become mainstream in the past few years, structured organizations that use a Stage-Gate process are finding that, as initially defined, does not handle Agile very well. Once an idea is approved in Gate 1 (see Figure 1 below) and it is decided that delivery will follow an agile/iterative mode, things start to complicate. Stage 1 is not too “hellish” for agile projects, as it requires high level requirements and design, which are also found in the early stages of agile. It is in Stage 2 where things start to complicate, as the business case, in most structured organizations, should be based on a defined scope, well understood and defined requirements and design, and a detailed plan for execution. Agile projects simply do not have any of these: scope is flexible and they only have high level requirements and design, and a release plan with no detail on what will go in each iteration, not to mention tasks within each iteration.
Figure 1: Stage-Gate Product Innovation Process
Furthermore, the business case for Gate 3 requires an estimate of time and cost: a baseline which, in waterfall projects comes after months of planning. Here is where agile projects really break the rules in most organizations. In order to get to a point where an agile project can make a commitment of how many more iterations are needed to get to an initial release, the full team needs to be assigned, and this team starts executing actual work, development and testing, so they can assess their velocity and use it to develop the release plan. In projects where a vendor participates in the development of the solution, they also need to be involved and, of course, get paid. At this stage, “development” has not been approved yet, so the gates in paradise start to shake badly and turn into the gates of hell every time an agile project gets assessed.
The good news is that the Stage-Gate process has evolved to include agile projects. In recent years Dr. Cooper has modified the process to include spirals, a series of build-test-feedback-revise loops, as depicted in Figure 2. This Agile-Stage-Gate leverages the agile framework using prototypes or actual product to get customer feedback and evolve the solution in an iterative fashion.
Figure 2: Agile-Stage-Gate
Structured organizations are now adopting this new version of the Stage-Gate process, as a response to the need to speed, to shorten time to market. Agile does not necessarily mean faster, but it does deliver sooner, as viable product can be put into production in a few months, rather than a few years.
However, in order to restore paradise, it is not only Finance and IT Governance that need to evolve. Other functions like Procurement, Legal, Organizational Change Management, etc.; also need to adjust their processes; if they don’t they could become a roadblock for agile projects which, in most cases, are used to implement innovation that supports business strategy.
One of the statements in the “Agile Manifesto” (a document put together by the “gurus” of agile in 2002) is “collaboration more than contract negotiation”. Procurement and Legal departments in many structured organizations are simply not prepared to make these changes to their processes and, most important, their beliefs. I remember when I started negotiating the development of BRMTool, the Benefits Realization application for my company, P3M Solutions: the negotiations with the off-shore development company where stalling as our legal advisor was trying to use the same approach that worked for waterfall projects in this project that was, not only agile, but was going to venture into the use of an off-shore vendor, Peru Software Factory (PSWF), and use agile, a combination that was not frequent and even considered not-doable at that time. At the end, the president of PSWF and myself decided to push the lawyers out of the way and simply collaborate, focusing on defining how we would work together rather than defining detailed scope and change processes. This collaboration was so successful that, not only the product release plan met every milestone, but also that the President of PSWF became one of the partners at P3M Solutions. Collaboration really worked!
Finally, the main change structured organizations need to make to adopt agile approaches is a five letter word: trust, which happens to be the cornerstone of agile. Trust means delegation of authority to make decisions, from executive level to management level, to people that can be part of the team and sit with them day in day out. Trust means understanding that not all requirements will be met, but the product will deliver the capability the business needs to realize the expected benefits. The output of the project will not likely have all the “bells and whistles” initially thought for it, but will have the “fitness for use”, what the business needs, that “good enough” solution that gets faster to market.
Organizations that adopt agile approaches and Agile-Stage-Gate and also adjust their supporting functions to remove roadblocks will be in an advantageous position to compete, achieving faster “time to market” Instead of talking about innovation, organizations need to adjust their governance structure, processes and organizational culture in order to foster and not quench innovation. Organizations can decide to restore paradise, or wait until hell freezes over.
