Measuring software development performance – Part IV : Debt minimization

August 16, 2012 1 Comment

In our previous articles about software development team performance, we talked about alignment productivity, and quality.

Today, let us talk about debt minimization.

Debt minimization capability

IT debt is a notion that is getting more and more popular at the analyst level. Gartner estimates IT debt to grow as large as 1 trillion $ by 2015.

Up until recently, IT departments have presented the cost of developing applications without really explaining or measuring the induced cost for the future.

But as soon as you develop and deploy an application, you generate recurring maintenance costs that will last as long as the application is used. This cost will disappear only when this application is replaced.

Experience shows that applications always last longer than anticipated for several reasons. Even when you think of an application as a temporary solution, there will be business changes or budget restrictions that may impact the timing of the next version; there might be issues, as with all software projects, causing late delivery; or even, possibly, there may be technical disruptions that result in the failure of subsequent application launch.

So delivering an application that will run with the minimal maintenance effort is actually one of the most important elements to consider when measuring performance of a software development team.

About five years ago, in a software vendor meeting, Microsoft had shared this slide. It was the evolution of developer teams for the Windows group split by roles. The blue bar indicates developers dedicated to maintenance, the orange bar indicates the ones dedicated to compatibility and the yellow indicates the one focused on innovation. This illustrates the evolution challenge cost (especially for an ISV) when a piece of software is used and successful. A lot of your development bandwidth is necessary to maintain the legacy.

image

The Windows slide also illustrates a very important point. Application debt is usually tricky to measure because most of the associated hidden costs are not included the pure corrective maintenance cost. Of course, when an application is really buggy, people notice they have a big issue and usually take some actions.

But in most cases, a significant part of the real debt is hidden in a bad design that translates into over-sized evolution maintenance costs, the issue becoming bigger and bigger over time. The slide above does not really indicate if the innovation part produces as much value as it did in the past. However, one can get some insights thinking that the effort was mostly part of some pre-Vista work.

Issues are difficult to avoid, because people usually only realize there is a problem after it is too late. It is a bit like the “leaning tower of Pisa”. And the real solution in these cases is to rebuild, which organizations think they cannot afford.

So they will spend much more money in maintaining a badly-designed system but – because it is usually less visible than the tower of Pisa in software – this fact will be hidden in operating costs and associated with the cost of “unreasonable evolutions” asked by those “over-demanding business users”.

If your evolution costs look like the non-industrialized curve below, it is time to think about a real modernization of your application:

image

In our view, there are two different kinds of systems:

  1. The ones which are very stable functionally. With these systems, even if an individual evolution might be too costly, it is probably good to maintain it for long as possible, even when the technology is very old. People often launch project when the technology is no longer supported by vendors. In our view, this is not a so important argument, when your system has run for decades. There is little risk that it stops with such a history behind.
  2. The ones which are still alive because the business requires relatively quick adjustments to the application. With these systems, if you have run into a “heavy debt” situation, it is always better for the long term to re-design and solve the issue. The tricky thing is that you probably need to do this “by pieces”, to secure the success of the project; “big-bang” projects often lead to major failures. Throwing away old pieces of code, reducing the size of the code base and aligning it to more recent technology will save a lot of money rather quickly. But you need to do it the right way, with the proper approach, the right people and efficient tooling.

The leverage effect of application debt

As your application grows, the legacy you have developed weighs on maintenance, and over-complexity may severely impact the cost of evolution.

Let us take a typical scenario for an application that has a 10 year lifecycle. The following table describes a “nominal scenario”, with an initial development of 100 and evolutions over the following years. Maintenance is calculated as 15% of the cumulated past workload.

Nominal scenario

Cost line

Year 1

Year 2

Year 3

Year 4

Year 5

Year 6

Year 7

Year 8

Year 9

Year 10

Total

Initial Development

100

                  

100

Evolution

  

30

15

50

25

10

40

25

16

5

216

Maintenance

  

15

20

22

29

33

35

41

44

47

284

Yearly cost

100

45

35

72

54

43

75

66

60

52

600

This scenario gives an overall cost of 6 times the initial cost, which is consistent with many projects that live normally and have certain levels of evolution over time to align with the business.

Now, let us imagine a team that is not fully optimal but adds a bit of extra complexity with a classical factor of 20% a year, which is not outrageous. Still, we estimate that the complexity multiplies over the year and increases the cost of evolutions by this cumulative approach. We still calculate maintenance by a 15% of the past workload.

Yearly over-complexity deviation factor: 20%

Cost line

Year 1

Year 2

Year 3

Year 4

Year 5

Year 6

Year 7

Year 8

Year 9

Year 10

Total

Complexity deviation

1,2

1,2

1,2

1,2

1,2

1,2

1,2

1,2

1,2

1,2

  

Initial Development

120

0

0

0

0

0

0

0

0

0

120

Evolution

0

43

26

104

62

30

143

107

83

31

629

Maintenance cost

0

18

24

28

44

53

58

79

95

108

508
 

120

61

50

132

106

83

201

187

178

139

1257

The cumulated cost of the project is doubled in the end, and interestingly, the cost of the evolution in year 10 is 6 times what it would be without the burden of this overly complex legacy!

Another observation that might be of interest is to understand what happens when projects start the wrong way. With a starting complexity deviation factor of 30% at the beginning, even if you react and behave perfectly over the following years by progressively lowering the factor from 1,3 to 1, you will never decrease your yearly maintenance cost enough to align with the previous team.

Yearly over-complexity deviation factor: starting 30% but going down to zero

Cost line

Year 1

Year 2

Year 3

Year 4

Year 5

Year 6

Year 7

Year 8

Year 9

Year 10

Total

Complexity deviation

1,3

1,3

1,3

1,2

1,2

1,2

1,1

1,1

1

1

  

Initial Development

130

0

0

0

0

0

0

0

0

0

130

Evolution

0

51

33

132

79

38

167

115

73

23

711

Maintenance cost

0

20

27

32

52

64

69

94

112

123

592
 

130

70

60

164

131

102

236

209

185

146

1433

Projects that start bad never end well, this is also a field observation that one can make. When it is really bad, it may be better to just start a new project. Here is a graphical version of the application cumulated costs of the three above scenarios.

image

Although a bit simplistic, we believe this model is consistent with some of our observations in the field where complexity translates into cost increases over time.

As we explained in the article about productivity, it is also very important to be able to estimate the real size and complexity of your project. A good estimation is also a way to ensure developing only the essential elements, because every single line of code that is not strictly necessary will be very costly over the whole application life cycle.

From our observations in the field, we are always surprised to find people are not familiar with simple metrics such as:

  • Number of business entities or tables,
  • Number of screens / pages,
  • Number of reports,
  • Number of lines of code.

For sure, one needs these factual elements (and even better, other details such as business rules, data properties, user interface fields, etc…) to be able to evaluate costs, both for producing software but also for maintaining it and implementing changes.

If the only criterion used to measure the impact of a feature request is workload, as defined by your developers, you may be in trouble. Firstly, you have no way of challenging this estimate, and second, there is a high degree of variability among developer skills.

As always, feel free to comment and share your own experiences.

Daniel COHEN-ZARDI
SoftFluent CEO

Filed under Software Development Tagged with , , , , , , , ,

Measuring software development performance – Part III: Quality

August 2, 2012 2 Comments

In our previous articles about software development team performance, we talked about alignment and productivity.

Today, let us talk about quality.

Quality

Quality is a critical pillar of software development and measuring the performance of your team against this criterion is fundamental.

Quality consists of 5 dimensions: reliability, performance, scalability, supportability, and ergonomics/ease of use.

image

Quality is obviously a topic "per se", with its own best practices and organization challenges. We developed this topic in an article the "Software testing best practices". It became the most visited post on our blog.

So the idea today is not to address the entire broad topic of quality management; instead we will focus on how to measure the performance of your team based on the five dimensions of quality as it relates to software development.

Reliability

Measuring reliability of an application can be pretty straightforward, even if comparisons are not always easy because this information is most of the time kept confidential inside companies.

Basically the best indicator is the # of open bugs and their evolution over time. Of course, the less bugs the better. But be careful, a lack of bugs may indicate insufficient testing or that the application is not being used (except, of course, when you reach this state over the long term). And in general few bugs means that there are few evolutions.

What is really important to monitor over time when delivering software is:

· The trend for the # of bugs: it should increase at the time you deliver a new version of your software and decrease regularly from there; then,

· The split in importance of bugs: critical, major or minor,

· The consistency of the bug / line of code ratio when delivering new software.

An important indicator that we will talk about later on is the time needed to correct a bug. If a lot of time is needed to fix a significant number of bugs, it may be a sign of a bad design and those situations are likely to have an impact on reliability.

The consequence of a lack of reliability can be disastrous, especially for software publishers. Unreliability can dramatically increase the support cost of an application. There are some real stories of vendors that were killed or lost significant business by a lack of reliability of a specific version of their software.

Performance

We are all software users. Who of us has never grunted against a slow application or an unresponsive web site?

It is obvious when we are on the user side, but some developers tend to minimize the importance of implementing a responsive user interface. In the end, whatever the technical reason behind it or the work that is done by the software, users won’t use a solution that is too slow.

Consequently, when coding, you need to choose technical options that provide a good performance experience for your users. Of course, this does not mean doing anything only for the sake of performance, or everything would be written in assembly language, but it is important to always optimize the components that have a strong impact on performance: remote connections, data access, manipulation of the appropriate reduced set of data, usage of loops (especially imbricate loops).

If some data processing requires a lot of time, you need to think about asynchronous mechanisms, splitting data or using other design options to avoid a bad performance experience for your users.

The consequence of a lack of performance will be user non-acceptance of the application, whatever the proper implementation of features you provide. If it is an internal application, lack of performance may cause conflict in a company. If this is a product you sell… then you won’t sell it!

Scalability

People tend to confuse performance with scalability. These are two different aspects of quality, and sometimes they require contradictory design options.

Scalability is the ability for your application to grow without limit, either in the number of users it supports or in the volume of data it can manage. For example, Access is a very well performing DBMS, but it is not scalable; it doesn’t work well with many users.

With the emergence of the Internet, SaaS and cloud applications, scalability has become often more important than performance. And it is most of the time a technically challenging issue if you have to deal with large number of users sharing the same data.

With the diminishing cost of hardware, multiplying the numbers of machines is clearly less costly than over-optimizing performance with developer work time. But simply adding machines does not work; you need to design your application in a way you can split the work between those machines and still guarantee consistency.

Measuring the scalability of your applications is a rather costly process. It requires setting up complete bench platforms dedicated to the process of evaluating your application under a heavy load of users/data.

It also requires the appropriate tools and skills to correctly interpret the measures and be able to assess the elements you need depending on the growth of your user base. This process is known as "capacity planning".

A lack of scalability can cause a major business disruption. If the application becomes stuck, then it cannot welcome new users. And there is usually no short term solution if the application has not been designed with scalability in mind.

Supportability

Supportability is the ability of an application to be correctly operated in production.

One key element is the proper management of exceptions and errors, as mentioned in our article "Code instrumentation best practices".

A production-aware application should also be able to deal with non-forecasted events such as network failures, bandwidth drops or hardware issues.

To measure results on this dimension of quality, we recommend tracking the average time to correct a bug, which involves:

· finding what happened,

· finding the piece of code concerned,

· being able to get the data used in the scenario that caused the error,

· identifying the non-forecasted scenario,

· proposing a valid correction for the code.

The speed at which you can reproduce issues is also a key indicator for an application’s supportability.

Thinking about disaster recovery and how you can restore the application in case of major events is also part of this dimension.

The consequence of having an application low in supportability is the risk of downtime. Beyond the induced costs and the image problem, a discontinuity of service can impact business revenue when the system is down.

Ergonomics and ease of use

Ergonomics and ease of use is also a major pillar of quality.

Measuring this dimension and making comparisons is not so easy though. People tend to either underestimate the importance of ergonomics, or – on the other hand – expect every application to have the same ease of use as major consumer web sites, which are developed with tens of millions of dollars.

Some applications bear some intrinsic complexity of their business which might require very specific knowledge making it a specific a challenge to keep usage simple.

The best indicator of ergonomics is the ability to use and learn an application quickly. If possible, we recommend measuring this for new users. How much time does it require to get them on board using the application? If a long time is needed, how much is intrinsic to the complexity of the application and how much is related to the ergonomics and design?

It is better to measure this on new users than on users that might have been used to other systems. Their judgment may be biased by habit and some reluctance to change.

Increased ease of use will lower training costs, improve productivity of your users and ease the acceptance of the applications.

As with performance, a lack of ergonomics may result in user non-acceptance of the application and extra hidden costs.

In conclusion, when measuring the quality of your software development, don’t forget to assess all 5 dimensions. They all contribute to make your software successful.

As always, feel free to comment and share your own experiences.

Daniel COHEN-ZARDI
SoftFluent CEO

Filed under Software Development Tagged with , , , , , , , ,

Measuring software development performance – Part II: Productivity

July 13, 2012 4 Comments

In our last article about software development team performance, we developed the alignment pillar.

Today, let us talk about productivity.

Productivity

Productivity is obviously a critical performance indicator in every business. But as software developers are costly and scarce resources, it is all the more important in that area.

Still, and although developing software is not really a new discipline, there is no universally recognized methodology to measure software development team productivity. The first thing to ask is: "Why is it so?"

First, by definition, each piece of software built is unique. You don’t need to build two times the exact same piece of software as you can reproduce it indefinitely by copying it. This makes it really impossible to make a formal and rigorous comparison between two pieces of software.

Second, the up-to-date technology is something that changes at a quite rapid pace. So each time a methodology relative to a certain wave of technology is reliable enough, it is generally already obsolete.

Third, there is a huge area for creativity in finding the different solutions to a unique problem. So measuring the result in terms of "lines of code", for example, is measuring the size of the solution, not the problem. And among the solutions to a problem, the lightest solution is actually the most valuable one, not only because it might cost less to produce (sometimes not by the way, it can take longer to write a synthetic text than a long one), but mostly because it costs less to maintain.

This said, should we give up with the idea of measuring software development productivity?

This is not our opinion at SoftFluent.

Let us start by looking at different industry approaches to address the topic.

Agile methodologies

Undoubtedly, this is the major methodology trend at the moment in software development. Still, the focus of agile methodologies is mostly about securing an efficient collaborative process between product management and developers. It addresses pretty well the alignment challenge we talked about in our previous post.

Talking about productivity, the standard practice is to evaluate development task using complexity levels and "points". Depending on the level of complexity, a non-linear formula gives the development time.

In many agile methodologies, the evaluation uses Fibonacci series to find development time depending on complexity. Fibonacci series is built in a way each number of the series is the sum of the two previous numbers, which gives us this kind of table.

 

Complexity level Estimated development time ratio
1 1
2 2
3 3
4 5
5 8
6 13
7 21
8 34

When managing the backlog of features, these are evaluated in complexity which translates into points. The classical measure used in agile methodologies then focuses on velocity which is the number of points developed per iteration, as per the schema below.

This works pretty well to evaluate the relative efficiency of a given team over time. But the main weakness is that it only gives a relative measure. Points are defined by the development team and cannot be compared to anyone else on the market. So there is no way to benchmark your development team using this approach only.

Function points analysis

Function points analysis is an evaluation method that addresses this challenge by defining in an "absolute" way what needs to be produced in terms of complexity.

FunctionPointsAnalysisCategories

Using categories such as:

  • External Inquiries
  • External Interfaces
  • Internal Logical Files
  • External Outputs
  • External Inputs

One needs to describe in a detailed way the different functionality of the software in each area.

There are then complexity matrixes where one can position the different functionality and calculate the adjusted function points of their project.

For people who want to know more, we recommend this document from Software Metrics, a company specialized in that area.

Working with traditional waterfall methodologies with a significant design phase can work pretty well and should be comparable from project to project.

The main drawbacks of this method are the following:

  • it requires a significant skill investment to really master the approach and get the expected results and measure,
  • it requires a quite rigorous and complete process in describing the software, an effort that we see less and less on the field as people have shifted to agile methodologies.

Our own approach for business software

In our view, we think that getting some absolute measure of the software that is produced by your team for a given budget is really important.

And as we understand the challenges mentioned above, we adjust our own simplified methodology to measure the size of a software development project, specializing in the area of "business applications".

When we talk of business applications, we consider all kind of applications that usually manipulate structured data, internal applications focused on a specific business or vertical, customer-relationship management software, complex e-commerce or Extranet web sites, and etc.

In our experience, the size and complexity of a project is a function of a quite limited number of parameters, that compares from a project to another, whatever the sector or domain area.

Each application usually has a bit of specificity in some way (workflow complexity, external connection challenges, dynamic user interfaces, customized advanced business calculations or multi-tenancy for SaaS applications), but we observed that the impact it has in terms of development cost is relatively constant from a project to another.

If you combine several complexity factors, you may also find ratios that help you deal with this complexity and you should be able to evaluate a project based on the following numbers:

  • number of entities
  • number of simple business rules
  • number of advanced business rules (cross-entity for example)
  • number of user interface elements (screens, web pages)
  • number of reports
  • number of external interfaces
  • number of batch calculations or processes
  • number of technology variations for components (rich-client, web page, mobile, database providers, cloud systems)

In projects using CodeFluent Entities, we often even found out that a project (including all project management dimensions) should never cost more than:

  • USD 5,000 per entity if it is an enterprise application,
  • USD 8,000 per entity if this is a packaged product that needs to be deployed at various customers (as it is generally more complex than a single-deployment application).

This gives a very simple way to detect issues when auditing real-world projects with a bit of history and legacy. We also compare this information with the total number of lines of code for the application. Our experience gives us relatively constant ratios correlating the number of entities to the numbers of lines of code in languages such as C# or VB.NET.

We also found out that CodeFluent Entities reduces the number of manually written lines of code by a factor of 2, with a direct link to the future maintenance effort.

And of course, the good news when using our tool, is that you get at least metrics about all the parameters mentioned above, whereas we still see many "blind" customers on the field, that are unable to know these key metrics.

What about your own experience?

We are eager to hear from your experiences, as this is a very hot topic.

Daniel COHEN-ZARDI
SoftFluent CEO

Filed under Software Development Tagged with , , , , , , , ,

Follow

Get every new post delivered to your Inbox.

Join 103 other followers