Why do software projects overrun?
I was talking to my sister the other day about some work she had done on her house. The builder said it would take 2 weeks and it ended up taking 5. She asked why this was and I said that it was the same reason that a software project (read web site for example) always takes longer than expected.
A customer approaches you and asks how long to build their site. You ask a few ball questions about the design, content and functionality make a load of assumptions and come up with a ballpark figure. About 4 weeks. Someone wrote (can't remember who, sorry) that engineers only have 3 measures of time, short, medium and long. Short for me would always be a day or two, medium would always be a month and long would be 6 months. Everything I estimate is one of those three units. That's stupid but that's what engineers do.
So when I say 4 weeks, what I am actually saying is, "4 weeks assuming that you know exactly what you want, that you don't change your mind or want something different once you've seen the prototype. I assume that the framework will behave as expected and I won't find any weird bugs. Whoever works on it will know exactly how to do everything and won't need any help from anyone else. They will not be sick or on holiday and will never be called in to help on another project. None of my equipment will fail temporarily or permanently, I will not lose any code and I won't hurt my hand playing tennis causing my typing to slow down. The design will be finished and usable and will not require any clarification or rework, it will also include relevant information for the mobile version of your site, which you probably assume will just work. All those things being correct, the web site will take about 4 weeks"
The customer hears, "It will take 4 weeks, even if the apocalypse starts, the office burns down or I suddenly realise that what I asked you to do is not strictly correct and needs to be modified. There is no way it will take even a minute longer than 4 weeks and I will plan its deployment for the day following the 4 week plan".
What happens? We all know what happens. Most customers don't actually know what they want, they mostly don't understand the web domain and sometimes don't even understand their own business! Perhaps they only have one or two people who are design authorities and they take too long to reply to queries, each one holding up the process. Like most customers, they know roughly what design they want but they don't know exactly what they want until the first design is basically finished and they start to change colours here and there, want to move things around like it was MS Publisher rather than the web. They, of course, don't even think about the fact that the site needs to be different on mobile and even if their side is perfect (it never is), you have your own battles to fight. Holiday, sickness, skill shortage, equipment problems, strange software bugs that only occur in one browser during a full moon on a Tuesday and everything else.
4 Weeks EASILY becomes 10 weeks and the customer gets funny, especially if they have to pay for 10 weeks instead of 4. Most software houses are wary of quoting fixed prices because there are so many variables so it becomes a lose-lose. Customers don't want to think they are paying for incompetence and software houses don't want to lose money to customers who can't specify what they want properly.
For some reason, we don't seem able or willing to fix this. It seems to happen time and time again, right up to the higher echelons of government who are possibly the worst of all at paying out stupid money for normal quality work.
We should take some advice from the building industry. They are certainly not perfect but a good builder and reasonable customer can get a decent quality building without losing too much sleep. Even an inexperienced customer who has money and no sense can usually be reasoned with because a building is slightly easier for most people to understand than a web site.
So you want a new house. Do you find a builder and ask for a price and schedule them in? No way! In the UK, you need planning permission, which takes time and requires drawings. The drawings don't have to be full architectural drawings but they need to be of a suitable quality to be submitted for planning consideration. The planning department can give general advice and might even have policy but they do not make the final decision. They might for instance recommend making the building 1 metre further back from the road because, "that's the norm". You would be advised to take their advice and modify your drawings because although the planning committee could overrule the advice from the department, they are more likely to take their side. This stage can be very long and drawn out, especially if you are doing something in an unusual area of the country or your proposed building is non-standard. Importantly, you spend time and money before you even know whether you will be able to build the building at all! This can cost thousands depending on who you employ to help you but what you (hopefully) end up with is the knowledge that your design is approved and will work and it is now in a position to be either further detail designed by the architect/engineer or otherwise priced for construction.
We rarely do this as a separate job in software and it is a shame. This is probably mostly due to the unrealistic speed that customers expect their web sites to be built in - and there are people out there who will imply this is possible. Perhaps one of your friends (who is not picky) got something on wix.com and tells everyone it was quick, easy and cheap so your customer expects the same for their fully bespoke site. Why not have two distinct work packages? Don't be pushing your customer to start paying you for build but cost out the design process separately, let them know that after the design work is complete, they can design not to proceed now or ever and it isn't costing any more than if it was all lumped together but it is more clear what is happening. If it slows down because the customer is not replying to queries, it is obvious what the delay is and the customer won't assume that you are just working on some other part of their site while you are waiting. What you end up with is an approved design that the customer is happy with and the designer is basically verifying as correct and acceptable for web - it could even include some simple web mockups to show how it works and what it looks like on mobile.
After that, the customer can then either get their own people to code it up (if they have any) or even get a number of quotations from developers to implement the site based on the fact that it is designed and complete and won't be changed. Developers need to invoice weekly for their work to ensure they are never out of pocket and if the customer changes their mind on something, it becomes a variation. If it needs re-designing, that's fine, it goes back to the designers who might OK it or tell the customer it will be very difficult. When it eventually comes back, the developers need to talk to the customer about how much of what has already been done can be kept and how much needs breaking down - this should be an easy conversation to have in the same way as a builder who has to knock a wall down after the customer changes their mind can explain why the wall needs knocking down.
One of the reasons why this also makes things easier is that the build phase is much shorter when you have a signed-off design. The implication is that assuming the developers have carefully checked that the design is complete, they should not have to ask the customer anything, they can just build it and release it for testing.
I know the theory sounds easy and I know that the reality is not that easy but having something is better than nothing and just because it isn't perfect doesn't mean that it isn't loads better than what we currently have. The thing I struggle with the most is that we call ourselves engineers but most of us can't engineer our way out of the minefield that is customer management. We need to be good at spotting weaknesses and conceiving solutions. They won't all work as expected but progress is littered with failure.
A customer approaches you and asks how long to build their site. You ask a few ball questions about the design, content and functionality make a load of assumptions and come up with a ballpark figure. About 4 weeks. Someone wrote (can't remember who, sorry) that engineers only have 3 measures of time, short, medium and long. Short for me would always be a day or two, medium would always be a month and long would be 6 months. Everything I estimate is one of those three units. That's stupid but that's what engineers do.
So when I say 4 weeks, what I am actually saying is, "4 weeks assuming that you know exactly what you want, that you don't change your mind or want something different once you've seen the prototype. I assume that the framework will behave as expected and I won't find any weird bugs. Whoever works on it will know exactly how to do everything and won't need any help from anyone else. They will not be sick or on holiday and will never be called in to help on another project. None of my equipment will fail temporarily or permanently, I will not lose any code and I won't hurt my hand playing tennis causing my typing to slow down. The design will be finished and usable and will not require any clarification or rework, it will also include relevant information for the mobile version of your site, which you probably assume will just work. All those things being correct, the web site will take about 4 weeks"
The customer hears, "It will take 4 weeks, even if the apocalypse starts, the office burns down or I suddenly realise that what I asked you to do is not strictly correct and needs to be modified. There is no way it will take even a minute longer than 4 weeks and I will plan its deployment for the day following the 4 week plan".
What happens? We all know what happens. Most customers don't actually know what they want, they mostly don't understand the web domain and sometimes don't even understand their own business! Perhaps they only have one or two people who are design authorities and they take too long to reply to queries, each one holding up the process. Like most customers, they know roughly what design they want but they don't know exactly what they want until the first design is basically finished and they start to change colours here and there, want to move things around like it was MS Publisher rather than the web. They, of course, don't even think about the fact that the site needs to be different on mobile and even if their side is perfect (it never is), you have your own battles to fight. Holiday, sickness, skill shortage, equipment problems, strange software bugs that only occur in one browser during a full moon on a Tuesday and everything else.
4 Weeks EASILY becomes 10 weeks and the customer gets funny, especially if they have to pay for 10 weeks instead of 4. Most software houses are wary of quoting fixed prices because there are so many variables so it becomes a lose-lose. Customers don't want to think they are paying for incompetence and software houses don't want to lose money to customers who can't specify what they want properly.
For some reason, we don't seem able or willing to fix this. It seems to happen time and time again, right up to the higher echelons of government who are possibly the worst of all at paying out stupid money for normal quality work.
We should take some advice from the building industry. They are certainly not perfect but a good builder and reasonable customer can get a decent quality building without losing too much sleep. Even an inexperienced customer who has money and no sense can usually be reasoned with because a building is slightly easier for most people to understand than a web site.
So you want a new house. Do you find a builder and ask for a price and schedule them in? No way! In the UK, you need planning permission, which takes time and requires drawings. The drawings don't have to be full architectural drawings but they need to be of a suitable quality to be submitted for planning consideration. The planning department can give general advice and might even have policy but they do not make the final decision. They might for instance recommend making the building 1 metre further back from the road because, "that's the norm". You would be advised to take their advice and modify your drawings because although the planning committee could overrule the advice from the department, they are more likely to take their side. This stage can be very long and drawn out, especially if you are doing something in an unusual area of the country or your proposed building is non-standard. Importantly, you spend time and money before you even know whether you will be able to build the building at all! This can cost thousands depending on who you employ to help you but what you (hopefully) end up with is the knowledge that your design is approved and will work and it is now in a position to be either further detail designed by the architect/engineer or otherwise priced for construction.
We rarely do this as a separate job in software and it is a shame. This is probably mostly due to the unrealistic speed that customers expect their web sites to be built in - and there are people out there who will imply this is possible. Perhaps one of your friends (who is not picky) got something on wix.com and tells everyone it was quick, easy and cheap so your customer expects the same for their fully bespoke site. Why not have two distinct work packages? Don't be pushing your customer to start paying you for build but cost out the design process separately, let them know that after the design work is complete, they can design not to proceed now or ever and it isn't costing any more than if it was all lumped together but it is more clear what is happening. If it slows down because the customer is not replying to queries, it is obvious what the delay is and the customer won't assume that you are just working on some other part of their site while you are waiting. What you end up with is an approved design that the customer is happy with and the designer is basically verifying as correct and acceptable for web - it could even include some simple web mockups to show how it works and what it looks like on mobile.
After that, the customer can then either get their own people to code it up (if they have any) or even get a number of quotations from developers to implement the site based on the fact that it is designed and complete and won't be changed. Developers need to invoice weekly for their work to ensure they are never out of pocket and if the customer changes their mind on something, it becomes a variation. If it needs re-designing, that's fine, it goes back to the designers who might OK it or tell the customer it will be very difficult. When it eventually comes back, the developers need to talk to the customer about how much of what has already been done can be kept and how much needs breaking down - this should be an easy conversation to have in the same way as a builder who has to knock a wall down after the customer changes their mind can explain why the wall needs knocking down.
One of the reasons why this also makes things easier is that the build phase is much shorter when you have a signed-off design. The implication is that assuming the developers have carefully checked that the design is complete, they should not have to ask the customer anything, they can just build it and release it for testing.
I know the theory sounds easy and I know that the reality is not that easy but having something is better than nothing and just because it isn't perfect doesn't mean that it isn't loads better than what we currently have. The thing I struggle with the most is that we call ourselves engineers but most of us can't engineer our way out of the minefield that is customer management. We need to be good at spotting weaknesses and conceiving solutions. They won't all work as expected but progress is littered with failure.