By definition, Agile application projects don’t have all of the requirements fully specified or all of the features designed at the beginning of a project. Instead, they start with a product backlog of user stories that describe the required features and capabilities at a high level that need to be built. Ideally, the level of detail of those user stories will be driven down to a fleshed out specification during sprint planning for implementation during the sprint.
But what if the business’ goals (and therefore the feature requirements) for a project can’t be easily defined?
That is, the business stakeholders know they have a problem that needs to be solved but they’re not sure what success should look like. For example, say a company wants to build a system for creating reports and performing analysis over data that is sourced by organizations external to the company but the stakeholders aren’t sure of which agencies will end up providing data and when, what format that data will arrive in, or what reports will be needed once the new data arrives. In addition to the stated requirements of importing new data and creating new reports and analyses, there is an inferred requirement that, over time, they want to be able to easily add new data without the costly overhead of invasive database schema changes, and to build out new reports and analytical functionality without having to incur large development costs (and the resultant delivery latency) from extensive changes to ETL, application, and report code. In cases like this, it is useful to label the requirements as intents to distinguish them from the as of yet undiscovered real requirements.
This article outlines an approach to leveraging a prototype-driven process to serve both the users’ need to clarify these intents until they become real requirements, and meet the development team’s need to deliver a Business Intelligence solution in a timely manner. The value of building a prototype is to help clarify both the stated and unstated requirements around a particular unmet business need. Building a throwaway prototype can be expensive, however, both in terms of time and resources. In most people’s minds, the downside to building prototypes is that actual construction of the “real” solution has to wait until the prototype has been completed (and requirements defined) which results in pushing the start, and thus the delivery, of the “official” application development project back a long time. But what if, rather than being throwaway, the prototype could be developed in such a way that it can be deployed to production after users have evaluated it, and further, that prototyping can continue without having to move out of discovery mode and into the more costly production-style application change management mode? This style of application development can and has been used successfully.
But how does the development team get started? To be successful, a blend of Agile processes and engineering disciplines is needed. A prototype-driven process will only work if the development team is not encumbered with a software development process designed to manage the factory-like implementation of an application where requirements are well-defined. This is because there is inherent tension between the factory-like processes, which have been defined to control quality and ensure predictability vis-a-vis costs and time, and the process of prototyping which is intended to quickly identify and refine requirements that are not well-defined on an as-you-go basis. Some methodology is required, however, to ensure that the prototype will be production quality when the time comes to deploy it.
Even though Scrum is considered to be an Agile development process, an even more Agile process is required. Instead of Scrum, a version of Kanban suited for software development is recommended. In a nutshell, this variant of Kanban embodies three principles:
- Use a visual process management system that tells the team what to produce, when to produce it, and how much to produce.
- Manage change using a continual feedback loop both within the team and between the team, the product owner and stakeholders.
- Once the prototype-driven development process has started, the development team will continuously make small improvements to the practices and processes.
For more information on Kanban for software development, see these articles:
What Is Kanban
There are other methodologies that can be adapted for highly Agile prototype-driven
development as well but this is the one that the author has successfully used.
But, while the right process is necessary, it is not sufficient. In addition to a well-defined but very lightweight process, the right resources are required. Or, stated another way, the right team capabilities are required:
- Thinking abstractly: While technical skills are important, the Solution Architect (or Technical Lead or whoever will provide technical and architectural guidance to the team) must be able to navigate the ‘seas of ambiguity’. They must be able to think in the abstract in order to be able to extrapolate the characteristics of the platform and application features required from the stakeholders’ stated intents. In fact, the entire team must be comfortable adapting to shifting requirements.
- Dealing with change: As the data and functional requirements are fleshed out over time, many small (and potentially some large) changes in direction will inevitably occur. The Business Analyst must be adept at briefly and quickly distilling requirements from the user feedback that will be continually generated by the prototype. Even the Product Owner must be both the user representative and responsive Change Manager since he or she will need to continually manage the expectations of stakeholders. Stakeholders and users won’t expect the speed and frequency at which changes may be presented to them for review and feedback given the delivery cadence they’ve come to expect from IT.
- Creative project management: Lastly, given the light-weight nature of the methodology, the Project Manager will need to avoid imposing any undue process burden on the team. They must both assume the role of Kanban Master and bridge the gap between extreme agility and IT management’s desire to track the progress of the project. Different ways to measure the progress of the project will need to be identified. The Project Manager will need to work closely with the architect to initially define and continually refine the expectation of what can and will be delivered at the end of the project which, while extremely Agile, will still necessarily be both time- and budget-bound.
- Being Agile but rigorous: To make up for what is missing in the way of requirements documentation and the artifacts that are normally produced by a formal development process, all of which are required to be able to transition the prototype to production application, the development team must adopt rigorous engineering practices. These practices should include coding conventions (e.g. naming standards, error handling, design patterns, testable code, automated code documentation generation, etc.) and processes (e.g. some form of test-driven development, “buddy builds”, code reviews, static security testing if needed, etc.). And, to help make up for the documentation deficit that inevitably accompanies extremely Agile development, it is recommended that a technical writer be assigned to the project to flesh out the technical and operational documentation that will be required to transition the prototype to production application.
One final ingredient is required to enable prototype-driven development. Building out an IT-supported prototype environment which is both accessible to the developers for deploying new code and available to users for testing can take time to design and implement but is critical to the success of the prototype-driven project. It is this environment which will enable the high velocity “try-test-adjust”, cycle between the development team and the users that will enable the project to be successful. And because of the starkly different nature of this type of development methodology with its speed and agility, evolving processes and project management and measurement, and atypical environment, IT management will need to adjust its practices to support and manage it. The risks of adopting this type of development methodology should be understood of course but the rewards can be great in terms of expanding IT’s delivery capabilities and enhancing its relationships with its users.
Part 1 of this article briefly touched on the processes and resources required for successful prototype-driven development. More planning and analysis than could be included in this article will be required on the part of IT organizations before they can successfully embark on a prototype-driven development journey. However, this methodology can and has been successfully used to develop great applications that exceeded users’ expectations.
Part 2 of this article covers how to continue prototype-driven development after the first version of the application has been released to production with all of the accompanying overhead of maintenance and operational management.