Understanding Technical Debt for Software Teams

Overview of Technical Debt

“Technical debt is a metaphor commonly used by software professionals in reference to short-term compromises made during the design, development, testing and deployment processes”.

In order to stay competitive, many organizations opt for software development methodologies like Agile, to accelerate the overall development processes. Cramped up release schedules often force teams to skip the standard best practices, resulting in the accumulation of technical debt. Technical debt is given less priority during the rapid release cycles and is addressed post the production release.

Organizations often push large, complex changes to speed up the release process. Short-term compromises are acceptable to a certain extent, however, long-term debt can damage an organization’s IT infrastructure and reputation. Sometimes, it comes with a heavy penalty of re-engineering and post-release fixes. These damages can be in the form of high costs for:

• Remediating pending technical debt
• Customer dissatisfaction due to scalability and performance issues
• Increased hiring and training
• Increased modernization time

Cost of refactoring, re-engineering, re-base, and re-platform could be much higher than the original cost during the initial development. These compromises should be thoroughly analyzed and approved by IT stakeholders and CXOs. This involves looking at the future tradeoffs, risk appetite (risk capacity), and cost. Organizations also need to evaluate the pros and cons of taking technical debt decisions.

Taking on technical debt can be both tricky and risky for organizations. Hence organizations must factor in associated risks and operational cost. One of the consequences of Technical debt is the implied cost of reworking on applications and their architecture. Hence, organizations should choose easy development paths and limited solutions to shorten the production time.

If the technical debt are not addressed over time, the accrued interest makes it more difficult to implement changes, resulting in business and technical challenges.

A Scandinavian study reveals that developers on an average waste 23% of their time due to technical debt.

As if that wasn’t alarming enough, Stripe published data which shows that software developers on an average spend 42% of their workweek dealing with technical debt and bad code.

Major Drivers of Technical Debt

• Need for faster solution design processes
• Faster development of source code
• Quick releases
• Cut throat business competition to release new and unique features early in market.

Impact of accumulating Technical Debt

• It results in daily operational costs to accommodate remediation.
• A longer development cycle leads to slower application releases.
• It incurs long term financial loss due to technical debt accumulation.
• It may result in compliance issues and lack of proper standards.
• Code quality and design gets compromised.
• More time is spent on debugging rather than development.
• It may result in failures that can put an organization’s reputation at risk.
• It can be a cause of security breaches and hefty fines.
• It can potentially lead to loss of agility and lower productivity due to outages.

Types of Technical Debt

• Design/Architecture Debt: It represents a design work with backlogs which may include lack of design thinking processes, UI bugs and other design flaws that were neglected. Most organizations do follow standard design practices like 'The Open Group Architecture Framework(TOGAF)' due to the agile way of designing. Tools and techniques like the ADM and TOGAF implementation governance, provide the required format and standard of solution design.
• Code Debt: It is the most common debt, which is skipped due to speedy agile delivery, complexity, or lack of subject knowledge. In some cases new features are added in the latest version, which the dev team may not be aware of. This might result in the dev team working on the same feature again, resulting in unnecessary cost and time investment. Sometimes, the development team doesn’t follow standard best practices for coding and may use quick workarounds. Also, they might not refactor the code because of time-bound release cycles.
• NFR/Infrastructure Debt: Introduced during designing and implementing Non Functional Requirements (NFR) such as:
  • Inaccurate scalability configuration may crash applications on high load.
  • Improper availability planning leads to outage issues when any data center is down.
  • Inaccurate caching, logging leads to slower application performance.
  • Repetitive code of error/exception handling may create refactoring and performance issues.
  • Additional auditing and tracing may lead to performance issues and occupy unnecessary database storage.
  • Ignorance of security may lead to serious data breaches and financial loss.
  • Improper observability and monitoring may not give alerts on time for any major issues in application and infrastructure.
• Testing Debt: The pressure of quick agile releases may force organizations to miss out on most of the manual and automated testing scenarios. Frequent unit and detailed end-to-end integration testing can detect major production issues. Sometimes, these detailed testings are skipped during the development phase which leads to major production bugs.
• Process Debt: It is introduced when a few less important business and technical process flow steps are skipped. In agile development, there are many processes that are followed like sprint planning, Kanban, Scrum, retro meetings, and some other project management processes such as Capability Maturity Model(CMM) and Project Management Institute(PMI), etc. Sometimes these processes are not followed religiously due to time constraints, which may have severe impact later.
• Defect Debt: It is introduced when minor technical bugs are skipped during the testing phase like frontend UI cosmetic bugs, etc. These low severity bugs are deferred to the following releases, which may later have an impact in the form of production bugs. These production bugs spoil an organization’s reputation and profit margin.
• Documentation Debt: It is introduced when some of the less important technical contents in the document are skipped. Improper documentation always creates an issue for customers and developers to understand and operate after the release. The engineering team may not properly document the release and feature details due to quick release schedules. As a result, users find it difficult to test and use new features.
• Known or Deliberate Debt: Known or deliberate debt is injected on purpose to accelerate releases. This acceleration is achieved by workarounds or alternate methods or technologies that use simple algorithms. For example, sometimes the dev team does not evaluate and consider better algorithms to avoid cyclomatic code complexity in the source code. As a result, it reduces the performance of the code.
• Unknown Outdated/Accidental Debt: It is introduced unknowingly by developers/designers and other stakeholders. It is sometimes introduced by regression of other related code changes, independent applications and libraries. For example, if all applications use the same error handling library code and if there is a regression issue in that error handling library, it may impact all dependent applications.
• Bit Rot Technical Debt: According to Wired, it involves, “a component or system slowly devolving into unnecessary complexity through lots of incremental changes, often exacerbated when worked upon by several people who might not fully understand the original design.” In practice, many old and new engineers work on the same module code without knowing the background details of the code. New engineers may rewrite or redesign code without understanding the initial design and background. It may cause complications like regression issues, etc. This happens over time, and it should be avoided.

Causes of Technical Debt

• Business competition: Competitive business markets may force organizations to roll out frequent feature releases to outperform their competitors and keep the customers interested.
• Time constraints due to agile releases: With tighter deadlines, the development team doesn’t have enough time to follow all coding/design standards such as language-specific coding standards, TOGAF enterprise design, suitable design patterns, review, testing/validation, and other best development practices.
• Save short-term cost: Some organizations want to develop and release features faster to save additional development costs on coding and design effort. They may prefer employing a small development team for faster releases with minimal short-term cost. These organizations may also hire junior or unskilled developers for more profit margin.
• Lack of knowledge and training: The development team may change very frequently during exit, internal movement, and new hiring. Faster release cycles may result in undertrained resources due to lack of functional or technical training and little to no knowledge transfers about product and design.
• Improper project planning: Tighter release schedules may result in improper project planning which plays a major role in introducing technical debt. For example, skipping important meetings with all business stakeholders or project planning phases.
• Complex technical design and technical solution: The development teams prefer simple technical design and solution over a complex one, because they don’t want to spend more time and effort understanding complex algorithms and technical solutions. Complex solutions take more time to understand and implement. They also need more POC evaluation and effort.
• Poor development practices: Most development teams prefer shortcuts by following poor development practices. Due to aggressive release timelines and lack of knowledge, dev teams don’t follow standard coding and design practices.
• Insufficient testing: It is a major contributor to technical debt. Regular unit and integration testing for even a small code change is very important. Testing and validation are the only mechanisms to identify bugs and shortfalls in software applications. These tests also find technical and functional bugs. Insufficient testing can lead to the introduction of technical debt.
• Delayed refactoring: Tight deadlines may force development teams in giving less priority to refactoring code in early stages. Hence they defer and delay code refactoring to prioritize quick releases.
• Constant change: ‘Change is the only constant.’ Software applications evolve and adopt new designs and technologies over time. It’s hard to cope with these constant changes in parallel. It takes time to revisit the source code and then use the latest design and technologies.
• Outdated technology: Most traditional organizations use outdated technologies. They make late decisions to upgrade or defer modernization with modern technologies. They miss a lot of new modern features which are considered to be technical debt. This debt can be mitigated only by shifting to modern technologies.
• No involvement and mentoring by senior developer and architect: It’s very common to have less or no involvement of senior developers and architects during design and development. Senior mentors play an important role in guiding the development team to avoid technical debt. Those senior developers/architects might have a better understanding and experience of working on the same project or software applications.

Identifying and Analyzing Technical Debt

• User feedback: User feedback/reviews are very important in identifying technical debt and mitigating it. Organizations should listen and act on users’ feedback for improvement and handling bugs. These feedbacks and bugs are considered to be technical debt.
• Analyze bad code smell: Use manual and automated code review to understand bad code smell like memory leakage of JVM applications. There are many code analyzers or tools like SonarQube, PMD, FindBug, Checkstyle, etc, that can help. They could be integrated with automated build and deployment of CI/CD pipelines for every release.
• Monitoring and observability tools: Application Performance Monitoring (APM) tools are the best tools to continuously monitor software applications, for example, VMware Wavefront/Tanzu observability, Dynatrace, DataDog, etc. They have special algorithms to check the performance of applications and underlying infrastructure. They also analyze application logs and generate failure reasons reports. These reports are a great source of identifying technical debt.
• Manual and automated code review: Continuous, manual and automated code review processes definitely help to identify technical debt using static and automated code analyzers.
• Operational profit and loss analysis: This analysis is done by business and senior CxO people. They analyze operational cost (Opex) and loss reports. These reports give a fair idea of improvement and address important technical debt quickly. Addressing this technical debt is very important for any organization because it impacts their business revenue.
• Performance metrics: Application Performance Monitoring (APM) and load testing tools also generate performance reports of software applications that are on high load. This is the best way to identify and mitigate technical debt due to NFR configurations like the performance of application and infrastructure, read caching availability, scalability, etc.
• Understand long-term or short-term requirement: Organizations identify technical debt by understanding long-term and short-term technical requirements. Accordingly, they prioritize, plan and remediate. These requirements are prioritized based on business criticality and urgency.
• Review with latest industry-standard best practices: Some technical debt can be analyzed by comparing it with the latest industry design and software development standards such as Agile, TDD, BDD, Scrum, Kanban, Cloud native, microservices, micro frontends, TOGAF.
• Code refactoring tools and techniques: There are modern tools available which are capable of analyzing legacy monolithic apps and provide suggestions or refactor partially to modern cloud native microservices design. They also provide tools to migrate on-prem VM (Virtual Machine) to cloud VM with easy lift and shift rebase.
• Security analysis: Some security related technical debt is identified during the security analysis phase. There are some security analysis tools available like CheckMarx, SonarQube which generate security reports for applications. There are other infrastructure security tools like Vmware Carbon black endpoint in security, RSA, Aquasec, Claire aqua security etc.

Best Practices to Avoid Technical Debt

To reduce technical debt, it’s essential to analyze and measure it. You can calculate technical debt by using remediation and development costs as parameters. These are a few techniques to avoid technical debt:

• Remediate application technical debt by using feedbacks.
• Religiously follow consistent code review practices. Have multiple rounds of manual code and design reviews by senior developers and architects.
• Perform automated testing after every build and release.
• Monitor and analyze reports based on observability and monitoring tools.
• Analyze and evaluate the performance and business impact of any new code and design change before implementation.
• Follow standard coding best practices.
• Follow the manual and automated static code review for any release.
• Use incident management and issue tracker to report and track bugs.
• Always review and validate solution architecture before implementation.
• Follow static and dynamic code analysis using code analyzer tools like Somarqube, PMD, FindBug, etc.
• Follow agile iterative development approach and regularly do retrospective meetings. Also, measure technical debt in each iteration.
• Use project management tools like Jira, Trello, etc.
• Do code refactoring of legacy code. Always revisit code and modularize common code components.
• Strictly follow test-driven development (TDD) and Behavioral Driven Development (BDD) approach for every module of code.
• Follow continuous build, integration, test, and validate the approach on all releases.
• Last but not the least, technical debt should be documented, measured, and prioritized.

Estimating Technical Debt Cost

It’s very important to measure technical debt cost as it helps stakeholders and senior management to analyze and prioritize remediation costs. This should be a measurable number to make business decisions. It also helps to track the technical debt remediation status. There are so many measurable variables for calculating technical debt.

There are various tools available like SonarQube to check code quality, code complexities, lines of code, etc.

We can calculate technical debt as a ratio of the cost to fix a software system (Remediation Cost) to the cost of developing it (Development Cost). This ratio is called the Technical Debt Ratio(TDR):

Technical Debt Ratio (TDR) = (Remediation Cost / Development Cost) x 100%

Good TDR should be <=5%. High TDR shows bad code quality, which involves more remediation costs.

Optionally, remediation cost (RC) and Development cost (DC) could be also replaced by hours, which will help to calculate remediation time in terms of total efforts in hours.

Key Takeaways

These are some key points about technical debt cost:

• The average organization wastes 25% to 45% of its development cost.
• Hiring and training new engineers involve additional costs and an increase in coordination costs.
• Operational overhead cost by spending 15 to 20% on unplanned work.
• Impacts organizations’ revenue for additional and unplanned work.
• Waste of time in analyzing improvement of source code and design.
• Lower productivity rate around 60 to 70%.
• Cost of project management and modern tooling.

Conclusion

Technical Debt can impact different factors like overall operations cost, velocity, quality of the product and can easily end up impacting teams’ productivity and morale. Hence avoiding technical debt or addressing it at right intervals during the development process is the best way forward. We hope this blog helps you have a better understanding of Technical debt and the best practices for remediation.

Squadcast is an incident management tool that’s purpose-built for SRE. Your team can get rid of unwanted alerts, receive relevant notifications, work in collaboration using the virtual incident war rooms, and use automated tools like runbooks to eliminate toil. Start now for Free.