How much will it cost to do ___ software project? That’s a reasonable question you are probably asked all the time if you are a software leader.
If you are a division executive or otherwise responsible for a P&L, you’ve confronted all the costs associated with custom software. If not, there are five hidden costs of custom software that you should understand.
It’s straightforward to estimate the costs of building a new piece of custom software if you treat it as a pure build project. We can see building projects all around us. The houses/apartments we live in, the cars we drive. We also understand the building software is not like building a car or anything else mass produced.
Of any vehicle produced, software is more like a fire engine. Fire engines are built one at a time by dedicated teams, and fire departments pay the cost of having them built, but they also shoulder ongoing costs of operating and maintaining them. Here are the five hidden costs of custom software.
Planning
The minute you hire software developers, project managers or any consultants on a project, the cost calculator starts. You can either do the planning yourself or hire others to do it. And it doesn’t quite work out to think you are going to do the planning yourself because you will never specify everything up front. Our industry spent about a decade attempting to do just that, and it didn’t work out. There are several main phases or activities within planning. These activities are things that must occur before coding can begin when starting a new project.
- Project vision: What does the overall destination look like? What’s the expected return from any investment?
- Architectural concept: What will the shape of the software generally look like? Will it have desktop, web, mobile, cloud components? Or will it run embedded in a military vehicle?
- Release definition: How many releases do you think make sense? Do we do it all in one release before we show anything to potential customers? Or do we have several releases where we incrementally broaden the audience and user base?
- DevOps process definition: What practices will we employ? Will we use a single programmer or many in various locations? How will we integrate code and deploy releases? How will we provision and manage environments along the way?
The Clear Measure model for DevOps-Centered Software Engineering calls out some layers of DevOps capability around the automated DevOps pipeline. Each requires a strategy, which is essentially planning. But once execution begins, planning continues throughout the projects. Each individual software feature needs some element of planning so that developers know what code to write. And any experienced manager knows the folly of having developers fill in these gaps while their hands are on the keyboard. Even the best self-organizing Agile teams organize these ongoing planning activities separate from coding. For each feature, these planning elements must be done before code is written:
- Feature concept and prioritization: Why would a potential software user care about this feature? Why do they need it? What problem does it solve? What should it take priority over? Who needs it?
- Feature analysis & definition: What does the user see and do while taking advantage of this feature? What type of data is involved with this feature? Inputs? Outputs? Screen mockups? This activity also accounts for 20% of all software defects according to research[1].
- Technical design: Is this feature buildable as a unit, or should it be broken up into smaller work items? What new technologies or architectural adjustments are needed for this feature to come online? What design patterns should be used to implement this feature?
- Test definition: How will we test this feature once it is built? What steps will one take? What unique test scenarios are there for this feature? Is there one way to use it or ten unique ways? (this is the best format for Scrum acceptance criteria).
Non-code defects
We can all relate to code that doesn’t function properly. We can count those defects, and we can track the time it takes to fix those defects. But good software teams still pass along 15% of defects on to users in production. And only 35% of total software defects come from problems in the code. This research, compiled by Capers Jones[2], shows that a full 65% of defects come from other sources such as requirements, design, documentation, and bad hotfixes. As outlined in the above model, it’s important to create a complete workflow for the software project so that every needed activity can be tracked as a discreet status of a software feature that is being planned, built, and verified. This is manifested in an Agile process as swim lanes, as many as 10 swim lanes in common scenarios. When a card (work item, user story) moves backward on the board, that represents a defect that has just been found. There is a problem that forces the card to move backward on the board. If the card had been advanced, that problem would have moved along with it. Using the analytical features of your software tracking tool, you can query for events when cards move backward. By doing this, you can quantify the number of problems (defects) that are found that never manifest themselves as a formal defect/bug.
Ambiguity
Developers are creative by nature. They enjoy building software and solving problems. If a software feature request comes to them, they will work on it and make it work. They will confront any ambiguity and work through it, sometimes for far too long. As a software leader, you can detect when a developer is struggling through ambiguity by measure changes in the frequency of version control commits or in the speed of cards moving across you board, whether it be physical or digital. In the planning activities outlined above, ambiguity will be minimized; however, if these planning activities are not explicitly performed, work items will come to developers with varying levels of detail and definition. When detail or definition is lacking, developers will and must resort to performing the pre-planning so that they can get on with the coding. Developers often begin coding on the parts they believe they understand, and then they consider the completed work as a constraint when making further decisions about how the ambiguity should be resolved. To combat this, create process steps ahead of coding so that ambiguity doesn’t reach the coding stage. This will prevent rework and wrong decisions. Not only does ambiguity cause slow productivity, but it also creates rework, thereby adding to the costs of software projects.
Maintenance
10-15% of building costs is normal for the first two years after a software product is release. This also applies to significant new software capabilities in existing products. Once software has been used for a couple of years, the bugs and problems will have been worked out, and the maintenance costs will fade substantially if significant changes are not made. User support costs will be ongoing as this is dependent on the nature of the user population. You may wonder where this maintenance costs come from. Research doesn’t give us that answer in detail, but a multi-thousand population of software projects yields this range. We can hope to be an outlier, but it probably wouldn’t be prudent to budget with that expectation. Once new software capabilities have been released, we know that there is an average of 15% defects that made it through the process somehow. These need to be found and fixed. These could be environment configuration problems, software tuning, performance adjustments, etc.
Undeclared architecture
In our DevOps model above, architectural blueprints are called out as a needed capability of a software team. If we don’t have a picture of what we are building, everyone on the team will have a different picture that they have built up in their own mind. Without a common and specific shared vision, the team will be aiming at a different target. We hope that the differences are small rather than large, but unless we as leaders give them the concrete vision, we can’t be sure how different. At the beginning of the project, our architectural blueprints will represent the hypothesis of the shape of the software that will meet the business objectives. Every release of the software, we will be making architectural changes, discovering new architectural elements, and will be revising the blueprints. It is important that we revise and store numbered versions of the architectural blueprints so that everyone have access to the current version. Philippe Kruchten has written about “4+1 View Model of Software Architecture”[3], and it is a very good model for emergent and iterative software architecture. In this model, we define the following layers: physical, logical, process, and development. We also anchor the blueprints using the list of functional scenarios that the software supports at the current revision of the blueprints. These scenarios are high level software capabilities, probably akin to Epics or Features, depending on your software process.
If you haven’t declared the architecture of the software by drawing these layers, it will diverge over time. Having a declared architecture will keep your team going in the same direction, implementing the chosen design patterns, leveraging the selected libraries, and your team will be able to maintain the logical structure of the code base rather than allowing various parts to be haphazardly linked inappropriately, causing the eventual spiral into the famous “spaghetti bowl”. Interestingly, legacy software systems are often described in this manner, where developers can’t sort out what the code is supposed to be doing. Old, aging software systems that have up-to-date architectural blueprints don’t tend to devolve into the tangled mess. They merely suffer from out of date technology rather than unchangeability.
A word on technical debt
I did not choose to list technical debt as hidden cost of custom software because it is much too vague. Any latent problem with the software could be labeled as technical debt. In fact, the NDepend product has an automated technical debt calculator through static code analysis. While this term has gained some traction and common usage among software developers, I don’t believe it does any good to use it in order to educate business executives or members of the board of directors. It is a term for factors in in the software that cause managing it to be more expensive than it should. But it is a catch-all term. I believe that to understand and then resolve these hidden costs, we must be more specific about them, confront them, and enact solutions for them.
Jones, C. (n.d.). Retrieved from SOFTWARE DEFECT ORIGINS AND REMOVAL METHODS: https://www.ifpug.org/Documents/Jones-SoftwareDefectOriginsAndRemovalMethodsDraft5.pdf
Kruchten, P. (n.d.). Retrieved from Architectural Blueprints. View: https://www.cs.ubc.ca/~gregor/teaching/papers/4+1view-architecture.pdf
[1] (Jones, n.d.)
[2] (Jones, n.d.)
[3] (Kruchten, n.d.)
