How long have you got?

Software sounds easy but it's hard. Basic surgery sounds easy but our experience and the amount of training that Doctors have to receive has convinced us that it cannot be that easy. For software, we are starting to leave the earliest days - in the same way that medicine used to be practiced by butchers, quacks and loosely experienced people (chemists, physicists and mentalists!).

Software is hard because it is not a solution, it is a framework to produce a solution!

The first two reasons that software is hard are very simple. We do not understand the problem well enough and we do not know the optimal solution to the problem.

Understanding the problem

For example, we might be "building an application to help us track orders", which sounds simple - about as simple as saying, "I want to build a house in the Victorian style"! Great but not enough detail to even think about starting. When you say, "order" what is an order? How many are processed per day worse case? How many people use this system? How important is this to your business in terms of acceptable downtime etc. The people who should be able to answer these (generally, the Customer) might not be savvy enough to articulate or think about things in a broad enough way to give really distilled valuable information to write good requirements. They might disagree with you just because they are the Customer. They might say something is OK but later realise it is not OK. So many problems and so many challenges, even at this stage.

Requirements Capture is crucial but often rushed or not done at all, leading to problems later but also, many Customers are loathed to pay for this part of the work even though it is no different in intent than having an architect draw plans for your house. Many Customers will expect an up-front price, which is just as hard in software as it is in house building. I can give you a ballpark price but until you pay me to draw out what you want, I can't give you anything more specific than +200%/-50%

Evaluating the solution

The second problem is, of course, related to the first. What is the optimum solution? Perhaps, more generally, what is an optimal solution. It doesn't have to be the best if it is good enough. Clearly, if you don't understand the problem, you cannot determine a good solution but even if you do understand the problem, how can you decide what is a good solution? It's like somebody buying a car, a Ferrari and a Mini are both cars and can both get from one place to another but their strengths and weaknesses are very different and we cannot always clearly understand whether the customer is after cost, flexibility, scale etc. They often don't know themselves or even, "I want it to be fast, scalable, flexible and very cheap"! We would laugh a house-builder out of the room if they asked for that but yet we are not good at educating customers about how things work.

Choosing the framework

Software frameworks are numerous, dissimilar and also have their pros and cons. Some cost money but are arguably better, others are open source and liable to the whim of the core team when they get updated. Some support certain patterns well, others support different architectures, platforms and extensibility. Some are designed to make a narrow range of functionality very easily, others are more designed to do anything, as long as you code it.

A framework will often be chosen due to the experience of the development team and this makes sense in the same way a hospital will use procedures that make sense for the experiences of their staff even if there is something that is slightly better out there. There are almost no cases where something is so much more amazing than any of the other top 20 frameworks that you should choose it over what you already know.

But even here you are torn between choosing something that is largely aimed at a specific market like Wordpress or Moodle when you are worried that the customer will then expect much more functionality that simply cannot be added on later. A bit like buying a small concrete house with 2 bedrooms and then deciding you want it majorly remodelled and the garden extended later - not easy and very expensive. On the other hand, if you choose something more generic like Rails, Drupal or .Net, you are in danger of writing a lot more scaffolding up-front to enable changes later to be made more easily. It should be something that is communicated to the Customer but again, we aren't always on the same page here and the cost and time difference, at this stage, is rarely obvious enough to make the decision easy.

Security

Security is not easy. It is another multi-faceted concept and combines a number of controls across different layers of the development stack from hardware up to the application but also includes management, process and risk assessment, something that Customers might expect to be "Fort Knox level" but without paying for it or allowing the time it takes to execute and test the controls.

Also, security, like other parts of software development are devils for the "unknown unknowns". If I know what a pen-test is but can't do it, I can decide whether to pay someone else to do it or to decide it isn't worth it. What about if I have never heard of padding oracle attacks or timing problems in SSL? Then I won't be asking the right questions to add the relevant controls. Of course, I can (and possibly should) pay a Specialist to do these jobs but it can be hard to know who knows what they are doing. There are plenty of Builders who build things that fall down, how do I know this security specialist is genuine?

Metrics and Logging

Building a system that you can monitor is essential. What are you supposed to do if the customer complains something is not working or something has been hacked if you don't have good logging in place to see these things? Many people get hacked and don't even know but logging well is hard. Logging can generate huge amounts of data so how do we set a baseline to tell good from bad? One event in a million might be important but how can we see that? How do we know if it is a glitch or an attack? If it was an attack, could we do anything about it anyway?

Like security, monitoring is for different reasons and will be driven by different teams. Operations, security, development and marketing will all want to know certain things about the system and the work involved can be significant. Not just logging but visualising the data that comes out and being alerted to unusual activity.

Performance and Measurement

Performance is hard. How quickly should something run when the system is not under load? How much should it slow down when load increases? How many users can it support? Are there elements of the system that are unnecessarily slow or is it all a bit slow? How do we do things well during development so we don't need to try to rework things later? How do we avoid assumptions about performance? How do we measure performance regression? Do we?

Performance can become a problem quickly if you are fortunate enough to become popular but the last thing you need is for a million people to visit your site and your web server fall over under the load. You should always design a scalable system unless there is a reason not to (like the number of users is a known/limited small number).

When your architecture is more than just an app and a database, measuring performance is one thing but what do you do with that information? Databases might be slower than Redis but can we utilise redis to reduce database use? If not, what else could we do? Can we just throw more hardware at it? If we used all the hardware our cloud provider had, would it scale to the whole world or would the bottleneck move? At what point will this happen? How hard will it be to deal with it now?

Abstraction

Abstraction is hard. There are plenty of OO experts who will tell you that you can abstract everything but do you even need to do that? It is definitely easier sometimes just to have an if/else instead of abstracting to another object and method but how can I measure the balance of abstraction being extendable and it just making everything harder to read and understand!

There is no answer to this question, you won't even agree between developers!

Translation

Lots of apps only come in one language but moving to multiple ones is hard. How does your framework do translation and more importantly, how easy is it to change translations? What if you have a marketing person who wants to tweak everything all the time? Do the translations need re-doing? What happens if you have a very high volume of text on your site? Can you automate the translation more effectively?

The Solution

OK, so Software is hard. There are other elements to software that are also hard but what is the solution if this is just a reality?

There are different measures you can take to reduce the burden of these steps and at least leave you with a minimal piece of hard work instead of everything being hard!

Please, please, please have a process, even if it starts as 10 bullet points on a checklist. At each stage you find you are experiencing the same problem, consider how your checklist can be expanded on, new questions added, new checklists for developer work to ensure they "thought about translations" or "considered input validation". You will find loads online but do something that is relevant for you and that will be enforced at your organisation.

Set a culture of working towards zero bugs instead of an acceptance that "stuff just has bugs in it". Any time a bug is seen, ask a very simple pair of questions, "why did that bug get injected?" and "what could we do next time to avoid it?" Most of the time there is a simple answer. Things are too repetitive or fragile or complex or "I just forgot". Most of these can and should be addressed. All bugs are not worth the same amount. Just because something is "just one bug" doesn't mean you shouldn't spend 2 weeks fixing something or writing something because it will fix not only the bug you found but also the ones you didn't and the others that would have come later because of your design.

Automate things! Back in the day, there was very little automation and therefore development was very limited and very slow! Now, we have all kinds of automation tools and loads of them are free so there is no excuse not to automate CI builds, deployments and other kinds of checks. You should ideally have a "click and go" performance test environment as well.

Have somebody who is really good at abstract thinking to own the interface with your customers. They should ask the same questions as developers: "why is this confusion there?" "why is there a difference between the expected cost and the actual cost?" "What is a good way to document the requirements so everyone understands?"

Use services and solutions that already exist. If you are considering translations, for example, spend a day on the web looking for what is for sale, and what other people use, you will mostly find the answers out there. If you need to buy something, insist on a free trial or money-back guarantee if you find that the product isn't a good fit for some reason.

In a nutshell, if you consider your software team as a car that needs constant care, updates and tweaks then you should be able to work most of this out and try and "make sure it doesn't happen again", then maybe each project won't be so hard and you can come in on-time and to-budget and make your customers repeat customers. Maybe!