Software Development Methodologies

Hofstadter's Law: Any computing project will take twice as long as you think it will -- even when you take into account Hofstadter's Law.

There are frequent newspaper articles and new reports of government or large corporate projects running late or massively over budget showing that there is some truth to Hofstadter’s Law.

Why is this case? Software development is highly complex undertaking requiring skilled people to design, implement the application and to manage the process. It is also a relatively new field of engineering. In the early days of programming, programs were small and could be written in a few days by a single person. As computers become more powerful, the size and complexity of software grew proportionally. More and more people were required to develop and manage the applications and thus the first generation of development methodologies were born. A software development methodology is a process that is used to organize, control and plan a software construction project. Ideas were taken from other fields of engineering and melded together. These proved to be flawed and there was little interest in improving them.

In recent years more and more emphasis has been placed on improving the process of development. Here at Light Speed we have invested heavily in our staff, the technology we use to build software and in enhancing our development processes.

Before describing our processes and their strengths it is useful to know the weaknesses of the most commonly used methodology.

The waterfall model on first appraisal seems quite logical and sensible but it is deeply flawed. Waterfall traditionally has five phases:

•    Requirements – gathering the needs of the users
•    Design – deciding on the structure if the application and its subsystems
•    Implementation – writing the code
•    Verification – testing the code for bugs and ensuring requirements are met
•    Maintenance - fixing errors after release and modifying the application to changing needs.

By looking at the stages it is clear that there is a lengthy period of time from gathering requirements to the end user testing the application ensuring that what is built is suitable. A leading cause of failed projects is that user requirements were not clearly understood and an application which did not meet their needs was delivered.

Verification is done only after design and implementation. Unfortunately if a previous phase takes longer than scheduled, checking the codebase for errors is often rushed and incomplete, resulting in defective code being delivered.

Gathering requirements is a difficult task. A software developer is an expert in his own field and the users of software are experts in their own. Each domain has its own jargon and implied knowledge. It is challenging for software developers and designers to gain a good understanding the users domain especially at the very start of the project. Requirements are likely to change. If the project has a lengthy duration, the requirements at the start can be quite different to those at the end of the project since the business world is in a state of accelerating change. With the waterfall model it is difficult and expensive to update the code as these needs change. A change in requirements in the testing phase is far more costly to implement than a requirements change in the design phase.   

Project Challenged Factors    % Of Responses
Lack of user input    12.80%
Incomplete requirements and specifications    12.30%
Changing requirements and specification    11.80%

Figure 1.2.2 Leading causes of projects being delivered late and over budget.
(Adapted from Standish, 1995)

At Light Speed we understand that businesses need to be able to respond quickly to change, to take advantage of new opportunities as they arise and we effectively minimise the impact of these changes on the development of our client’s applications. We strongly encourage our clients to be involved in the whole development process. When a client becomes a participating member of the development team it is much easier for developers to check that requirements are being met correctly and that our vision of the finished application matches that of the client.

As each project is unique there is no methodology that is ideal for all types of projects therefore we adapt our processes to the needs of each individual project. I will briefly describe are most frequently used approach.

We do initial requirements gathering but we understand that these are likely to change and plan accordingly. The fine details of requirements are deferred until they are about to be implemented. We also do an initial design to guide the development process but at a much less detailed level than is seen in the Waterfall model. Requirements are prioritised by the client with assistance from us so the most important needs are completed first.  Requirements most likely to change are completed as late as possible to minimise rework.

We typically use an evolutionary and iterative approach to design and development meaning that the application will be built incrementally. A project can consist of many iterations. Often each lasts a few weeks only. At the start of each iteration, the requirements to be implemented are selected and fleshed out with the client. Then the detailed design work is carried out. After this, the requirements are implemented and tested in detail. An iteration is not considered complete until it has met all of the client’s acceptance criteria.

Testing occurs throughout implementation and not just at the end. This permits the client to evaluate that the requirements are being met during development and not at the end when it is too late. The impact of changes to requirements is greatly reduced since detailed requirements gathering and design is done just before the requirement is to be implemented.

Many of the problems that face software development projects can be mitigated by using the correct process while choosing the incorrect process can easily doom any project. At Light Speed we take great pride in the effectiveness of our own methodologies. We review the performance of our processes during and at the end of each project to learn lessons and to improve for future projects.