Scalability versus performance

June 10, 2013 Leave a comment

People tend to confuse performance with scalability. These are two different aspects of the capabilities and quality of a software application that sometimes require contradictory design options.

Performance is an obvious need from a user prospective. Who of us has never grunted against a slow application or an unresponsive web site?

Consequently, when coding, you need to choose technical options that provide a good performance experience for your users. Of course, this does not mean doing anything only for the sake of performance, or everything would be written in assembly language, but it is important to always optimize the components that have a strong impact on performance: remote connections, data access, manipulation of the appropriate reduced set of data, usage of loops (especially imbricate loops).

In our CodeFluent Entities product offering for example, we think we made the appropriate balance for giving the right level of abstraction through a model while getting better control on performance than with an ORM approach.

Beyond using the right tools, some design choices can be critical. If some data processing requires a lot of time, you need to think about asynchronous mechanisms, splitting data or using other design options to avoid a bad performance experience for your users.

Users won’t accept an application that lacks performance.

However, this is not the only point. Now that Internet and SaaS applications are becoming the rule, applications need to manage high volumes of data or users and scalability is getting as important as performance.

Scalability is the ability for your application to grow without limit, either in the number of users it supports or in the volume of data it can manage. For example, Access is a very well performing DBMS, but it is not scalable; it doesn’t work well with multiples users, as it has not been designed for that goal.

image

Scalability is most of the time a technically challenging issue if you have to deal with large number of users sharing the same data. With the diminishing cost of hardware, multiplying the numbers of machines is clearly less costly than over-optimizing performance with developer work time.

But simply adding machines does not work; you need to design your application in a way you can split the work between those machines and still guarantee consistency. Avoiding the consumption of the whole memory is also a challenge you should pay attention too.

In our CodeFluent Entities product offering for example, we implement BLOB storage in a way that is automatically scalable using streaming. When developers use the classical byte[] type, they might not see the dangerous issue before the application goes into production with several users connected to the server.

Measuring the scalability of your applications is a rather costly process. It requires setting up complete bench platforms dedicated to the process of evaluating your application under a heavy load of users/data. It also requires the appropriate tools and skills to correctly interpret the measures and be able to assess the elements you need depending on the growth of your user base. This process is known as "capacity planning".

A lack of scalability can cause a major business disruption. If the application becomes stuck, then it cannot welcome new users. And there is usually no short term solution if the application has not been designed with scalability in mind.

This is why this topic should be anticipated and planned carefully.

Daniel COHEN-ZARDI

Filed under Software Development Tagged with , , , , , ,

Software testing best practices

April 26, 2013 3 Comments

Often under-estimated and not much considered by developers, testing is however one of the key elements of software engineering, and a fundamental pillar without which no software can be put on the market with appropriate quality. The goal of this article is to evangelize the major dimensions of software testing and share our experience of best practices as we perceive them at SoftFluent.

An often undersized investment in quality

Software testing is a broad area that allows the control of software quality. This quality is made of several dimensions that can be grouped within the three following categories: features, engineering and adjustability.

Still investment in testing is often undersized from software vendors or IT teams. Similar to what happens with security, the feeling of spending non-productive time is stronger than the power of best practices that recommend forecasting at least one tester for two developers. Field reality can easily be two times less.

In some geographies such as France, the phenomenon is probably aggravated by the important weight of service companies and custom developments. Their relative importance is stronger than anywhere else, partly for cultural reasons, and consequently, the culture of packaged software is weaker. Service companies have a lower sensitivity than software vendors to quality, because the impact of a correction is often limited. On the contrary, a software vendor must invest upfront on product quality; otherwise the deployment of its product can quickly become very costly.

At companies such as Microsoft, whose software products are deployed by tenths of millions, some development teams can have up to two testers for one developer!

The different test levels

First, let us note that software testing is accomplished at various levels. Potentially, one can distinguish four levels:

  • Unit tests are accomplished by the developer to check the good execution of functions that he is responsible for. These functions are often tested independently, with limited set of data. It is a good practice to early define representative test samples, as the developer will be able to develop unit tests based on these samples which will improve quality.
  • Integration tests are usually accomplished by a dedicated tester, within the development team or inside a quality team which works closely with the development team. The particularity of these tests is that they target the testing of a full set of features that often relies on leveraging several components a full process beyond unitary functions. One usually talks about "white box" testing, as it is possible to have a look at relationship between components to check their efficient cooperation. Having test data early in the cycle is also an important element of efficiency.
  • System tests or qualification consist in playing complete scenarios, representing software use cases, without considering implementation or underneath components. These tests follow a "black box" logic and It is highly recommended that the team realizing these tests be distinct from development team. This team can even be placed "off-shore" with the need of requiring detailed formalism to fully describe usage scenarios. During system evolution, these scenarios will be replayed to validate non-regression.
  • Acceptance tests are lead with pilot users to validate software adequacy to the business and easy adoption by those who will use it regularly. Once again, involving users early in the cycle to use part of the software will be a good manner to get aware of user acceptance criteria, and, if necessary, adjust some elements such as ergonomics.

Our belief in unit, integration and system testing is that it is always a good idea to automate these tests. A manual test is an expense. A test which is consolidated in a replayable script is an investment. Automating unit testing and some integration tests allow extraction of some of these tests to be replayed directly in the automated build of the system.

Additionally, test artifacts such as unit test code, configuration files or data set are part of the software documentation necessary for its maintenance and serve as examples for using software public interfaces (APIs). This is particularly important for systems that will expose features as services, or get integrated with other systems.

Different test natures

It is also worth noting the different natures of testing:

  • Functional tests validate the right behavior of the software as per the service delivered. These tests are often described in detail in scenarios in which most of the investment is made, consistently with the effort made in specifications formalizing the feature. Our recommendation on this topic is consistent with points mentioned above. Writing scenarios early is a very good practice as it validates the functional analysis, while giving material to the developer for its own unit tests. But putting the exclusivity of the effort on these tests is a common mistake.
  • Platform compatibility
    tests include as a minimum setup tests on targeted environments (operating system, database…) or even more advanced tests for certifying a specific platform, especially for ISVs. These tests allow checking the right application of best practices such as the use of virtual directories, execution without administrator rights, uninstall procedures, resource localization, etc. Developer desktops should not be confused with the target execution environment.
  • Robustness tests are targeted to testing the software behavior in suboptimal conditions. These tests are scarce systematical, although they are not so complex to implement: unplugging a cable, stopping a machine… It is important to know how the software will behave within these circumstances and define a recovery scenario. Not testing these cases saves a few hours in the short term but preserves a risk whose financial consequences are often impossible to evaluate. It is quite obvious that the user will sooner or later be exposed to a network breakdown, so it is better to anticipate the software behavior and document it.
  • Performance tests are often delivered at the end of the project. Factors influencing performance are numerous and it is risky to conclude early on partial tests without having representative volume of data and exact execution conditions. However, developers can usually test behavior of some unitary services quite early in the cycle when these elements bear a performance risk. Moreover, experience demonstrates that the result of performance tests must be precisely documented to formalize measuring conditions. Conclusions must be made based on relevant performance counters such as CPU consumption, memory usage, requests per second, Input/Output flow…
  • Scalability tests should be distinguished from performance tests. An application can perform well but be limited in its ability to sustain an increasing number of users (or any other parameter). A software system, especially if it is supposed to be used through Internet, should be designed to scale up almost without limit by sampling adding machines. Tests must be made for this, with tools allowing simulating numerous simultaneous users.
  • Usability tests are quite never lead as they require involvement of final users. However, they are a significant factor for real and perceived quality, improving comfort of software usage. These tests include both "Look" (visual appearance) and "Feel" (ease of use).

Favor early detection of bugs

The reader will have understood, through the different comments made in previous paragraphs, that at SoftFluent we are closer to agile approaches than pure waterfall methodology, and we favor everything that allows short cycles, for testing as well as for functional specifications by the way. The reason is above all financial.

The cost of a bug is very different depending on the stage at which it is introduced and at which it is detected. As an example, if a design error, detected in the specification stage cost 1 to correct, the same error detected at qualification stage will cost 10, and potentially up to 100 if the software is already deployed. The following table estimates ratios for different kind of bugs depending on these stages.

Introduction/Detection Specifications Architecture Development Qualification Post-Deployment
Specifications

1

3

5

10

10-100
Architecture

-

1

10

15

25-100
Development

-

-

1

10

25

This is why we try to validate most elements upfront, before having a significant development effort, costly by nature. Furthermore, bugs are easier to understand and correct on a limited perimeter than on a complete application which obviously bear some complexity.

Evolution towards agile methodology

The positioning of tests across time is quite different using a traditional V Cycle methodology or using an agile approach:

From a human prospective, agile approaches require a permanent collaboration between developers and functional experts, collaboration which is very often the secret of successful software. In traditional methods, an advanced formalism is often a way to be theoretically more confident, but it is also a way for developers to protect themselves behind written documents to avoid any modifications, even if these are necessary to fulfill business needs. The phenomenon also happens with testers, although the tester should be the developer’s best friend, as it helps him find his bugs!

As a conclusion, let us remember that software testing is an important job, that comprises multiple aspects, but is not enough recognized and valued, although it plays a critical role in software quality. Everyone being able to evaluate its daily experience with computing should easily admit that progress is still to be made in this area! So let the developers and IT people give to software testing the place it deserves.

10 pieces of advice

  1. Test is a real job, having test roles distinct from developers is an absolute necessity
  2. Test should not be underestimated, about 25 to 30% of a development project
  3. Test data should be produced early in the cycle to detect design errors and facilitate the work of developers
  4. An application should always be designed to be testable, and a good split in components is a very important factor
  5. One should invest in all dimensions of testing, including acceptance and usability tests
  6. It is necessary to define operating conditions for the system to test configurations, scalability or robustness in suboptimal conditions
  7. It is always a good idea to invest in automating the tests, especially for non-regression
  8. Tests are developments that should be versioned
  9. As for development, a partial work which is well targeted will cover most of the important elements following an 80/20 rule. It is better to start this way and improve through an iterative model than being utopist and believe in total completeness of tests
  10. A model-first and strongly typed static approach facilitates testing, whereas a fully dynamic approach based on metadata, although very flexible for the developer, will postpone the discovery of key issues to very late moment in the cycle.

As usual, feel free to add your own comments

Daniel COHEN-ZARDI

Filed under Software Development Tagged with , , , , , , , ,

Benefits of model-first development

April 8, 2013 Leave a comment

CodeFluent Entities White Paper – Version 4.0 – April 2013

WhitePaper

 

The objective of this white paper is to describe the software development challenge and clarify its root causes.

The first half of the document explains the market challenge and why this is a tough business issue. This part is widely applicable by anyone interested in software development and is not dependent on our offering.

In the second half of the document we explain how SoftFluent addresses the challenge through itsCodeFluent Entities model-first software factory and associated methodology.
Read CodeFluent Entities WhitePaper

Learn more about CodeFluent Entities

Filed under CodeFluent Entities, News, Software Development Tagged with , , , , , , ,

Code Generation: good or evil?

March 7, 2013 Leave a comment

How many times have I heard endless debates about the benefits or risks of generating code. Developers sincerely argue about the virtues or dangers of it with a level of passion that sometimes approach the religion war levels.

The real question

After having spent 20 years in this industry, I can strongly affirm that this question is asked the wrong way. The question is not about “should you” or “shouldn’t you” generate code? The question is when and how?

Whether you know it or not, as a developer, you certainly use code generation mechanisms at different level of the tools that you use.

Here are some examples in the .NET world:

  • ASP.NET is itself a code generator. Intermediary classes are generated and compiled on the fly.
  • The Windows Forms designer is a code generator. It will create C# or VB.NET classes from the graphical form editor.
  • The XAML designer is a code generator. It will create hidden .G files from the graphical XAML editor.
  • The XmlSerializer class, massively used for XML serialization, is an implicit code generator. It will create code and compile it to improve serialization performance.
  • Some Visual Studio files like .RESX (resources) files and .SETTINGS (app settings) files are used by the IDE to generate utility classes.
  • Entity Framework .EDMX (Entity Data Model) files are used to generated Entity Framework ObjectContext classes.
    Reality is that code generation is not an issue as long as:
  • It is seamless (you don’t even know you use it),
  • Or you trust the generator (you don’t need to check what is generated).

    Benefits of code generation

To understand when you should use it, let us start by focusing on the benefits it can bring you. Properly used, code generation allows you to automate programming tasks that would be done by hand otherwise. It is not so different that any kind of automation of repetitive tasks.

So when is programming becoming repetitive? Mostly when you need to follow a structure and implement patterns in the same way for a significant number of elements.

Typically, when you implement a business application, you will have several business entities and you will need to structure your application in layers. Doing that, you will need to implement data access and manipulation methods, business object model structure, service layers and even user interface elements or reports that will reproduce the same implementation patterns and principles.

Coding this by hand does not only take more time, it is error-prone, and you have a higher risk of heterogeneous quality and a higher maintenance debt.

image

Looking at the schema above, one can easily understand that it is lighter to maintain the source elements of the generation process than maintaining the manually-written equivalent.

If this is so obvious, why do some smart developers challenge this?

When code generation does not work

Well, the real issue is that generating proper code with the sufficient level of flexibility and evolution capability is a very tricky process, which is most of the time underestimated by developers, even the smartest ones. So developers have often had bad experiences that somehow made them think that code generation itself is a bad idea.

There are mainly two different failure scenarios:

One is the developer that has tried to develop its own generator. It usually starts naturally as the developer notices he is doing a repetitive task, so he tries to automate it which sounds pretty smart at first glance. But when complexity grows, adjusting the generator to richer needs becomes more time-consuming than doing the code by hand.

Would we add the “code generator” maintenance to the cost schema, it would certainly become higher than the manual code over the long term, at least if you want to deal with a significant feature set and support technologies as they become available.

Then the developer usually get trapped by the alternative of investing more time into the generator or considering what he has done as a one-shot process and try to maintain the generated code directly. Bad luck! Both are dead ends.

The investment into a powerful and flexible generator grows exponentially and there is no chance to keep the pace of technology innovation without this becoming the developer unique goal. Chances are that he will then forget about the business application itself.

Maintaining the generated code by hand is also a bad idea. A generated code base is always larger than the manually created one, so it is counter-productive to use a generator if you need to maintain the generated code.

The second failure scenario is having used a limited code generator. So what is a limited code generator?

There are different ways a code generator can show limitations. The classical one is the lack of openness of many code generators. Many of them deal with the complexity challenge by reducing your options as a developer. Doing this too strongly can limit your possibilities as a developer. So extensibility and customization capabilities should be checked in detail when evaluating such a tool.

On the other side of the spectrum, there is the opposite approach that only relies on “Templates”. Template-based code generation is a very simple and powerful mechanism for mixing code and content. It is typically what was used for “Active Server Pages”, using the <% … %>, syntax when Microsoft proposed its first dynamic approach for the web.

It is very seducing at first glance, as it is a very open approach because you can always evolve your templates to add features. However, though useful in some scenarios, it does pose a complexity issue when applied to rich applications. Those code generators intrinsically bear low value as it transfers the complexity to templates. When the code between the brackets is weighing more than the content elements, it is a sign that you have gone too far.

A serious “Business Object Model” layer for professional applications cannot be generated without templates becoming highly complex. At best, those templates are maintainable by only one superman developer who designed them. At worse, they cannot evolve at all.

This is exactly the same reason that made Microsoft drop ASP, that would work for simple web sites that were slightly dynamic, and develop the ASP.NET platform, a new approach that was necessary to provide a platform for richer web-based applications with the proper level of maintainability.

Attributes of code generation that works

So it there are so many ways to fail with code generation, how I can be successful with it?

In our view, the key attributes of successful code generation for business applications are the following:

  • The code generator is provided by a team responsible for it, external to the teams that develop business applications,
  • At best, the code generator is maintained by an external company that makes its business out of it,
  • The generator is reliable enough so you uniquely concentrate your maintenance effort on the source elements for generation,
  • The generated code keeps human-readable and simple enough not to introduce a higher level of complexity than manual code,
  • Code generation of central or back-end parts (database or business model) allows dealing with complexity through a powerful modeling of concepts,
  • Template-based code generation is used only on front-end components, such as automatable user interfaces and reports, which implement more visual and content elements than business logic.

As always, feel free to comment with your own experience,

Daniel COHEN-ZARDI

Filed under Software Development Tagged with , , , , , ,

Social Email Login

February 20, 2013 1 Comment

Your users are tired of having to register on your site because it takes time, and it’s another username/password to remember. If you have tried to offer the social login feature on your site and realized that it was not that simple to build, then this Social Email Login utility is for you!

Social Email Login is a library built in ASP.NET 4.0 that will let your users log in using their favorite social network. This library was made because OAuth libraries available on the net today are usually too complicated, and often depends on other libraries. Sometimes you just need a simple login to identify the user, and nothing more…

The main social networks are available out of the box, such as Facebook, Google, Microsoft Live, Yahoo and Twitter. The goal of this project is to have a simple and flexible tool that retrieves the email address of the user who logged in using one of the available social network, and then, use that email to integrate easily with the ASP.NET Membership provider.

This is a simple tool because it only tries to retrieve the email address and nothing else.
This is a flexible tool because it lets you add service providers very easily.

The available authentication protocols are OAuth 1.0, OAuth 2.0 and OpenID.

Dependencies:
Social Email Login has only one dependency, which is the CodeFluent RuntimeClient, a free library, available as a Nuget (http://nuget.org/packages/CodeFluentRuntimeClient/). This library is used mainly for two of its features:

  1. to manipulate the different parts of a url
  2. to work with JSON (de)serialization

Many json utilities exist on the net today, but most of them are over complicated, and too big. So we decided to use the CodeFluent RuntimeClient library, because it does what we need and works for any type of ASP.NET application. Because the source code is on CodePlex, you can obviously change this library to another one if you like.

Nuget:
https://nuget.org/packages/SocialEmailLogin/

CodePlex:
http://socialemaillogin.codeplex.com

Website Demo:
http://www.softfluent.com/downloads/socialemaillogin.demo.zip

HELP
To run the demo, you need to edit the web.config:

  • select your SQL database
  • enter both the consumerKey and consumerSecret keys for each service provider you wish to use. You will need to create an app for each service to retrieve your consumerKey and consumerSecret keys.

Filed under News, Software Development Tagged with ,

Generating JSON web services from an existing database with CodeFluent Entities

February 11, 2013 Leave a comment

This article posted on CodeProject will show you how to generate a JSON base web service layer from an existing database using CodeFluent Entities. We will also generate a web client back office following an “Import wizard”.

A common scenario

 

Let us say that we are facing the following scenario:

  • We have a database that we want to expose via a JSON based web service layer, providing CRUD (Create, Read, Update and Delete) operations.
  • We also need to build a back office in order to manage and administrate the data coming from our database.
  • We may need, on a future, to access in a different way our database, for example from a Smart Client or expose a SOAP based web services layer (there are always new ideas).
  • We need to deploy this system as soon as possible.

Let us start, what we need to do is:

  • Build a data access layer capable to load data, create new data, update and delete existing data (and make sure it works).
  • Manage validation data (and make sure it works).
  • Build a JSON based web service layer:
    • Build every needed service contract and operations.
    • Configure our service contracts to support JSON.
    • Host our services.
    • Make sure it works
  • Build a web based client (and make sure it works).
  • Lay the foundations so any possible evolution and additional architecture can be supported including mobile access through different smartphone devices.
  • And everything I have missed.

Or…. We can use CodeFluent Entities to do the plumbing and being sure that it works.

In the starter wizard, we can see some of the possible built-in architectures that can be generated by CodeFluent Entities, and of course you can imagine your own architecture by creating a custom CodeFluent Entities project with your relevant set of producers.

clip_image002

The scenario we mention here is developed "step by step" in the full article on CodeProject

Filed under CodeFluent Entities, News, Software Development Tagged with , , , , ,

Benefits of Model-First Software Development

February 6, 2013 Leave a comment

CodeFluent Entities White Paper

whitepaper-image

The objective of this white paper is to describe the software development challenge and clarify its root causes.

The first half of the document explains the market challenge and why this is a tough business issue. This part is widely applicable by anyone interested in software development and is not dependent on our offering.

In the second half of the document we explain how SoftFluent addresses the challenge through itsCodeFluent Entities model-first software factory and associated methodology.
Read CodeFluent Entities WhitePaper

Learn more about CodeFluent Entities

 

Filed under CodeFluent Entities, News, Software Development Tagged with , , , , ,

The mythical "Indian man-month"

January 7, 2013 1 Comment

Introduction

For long, it has been established that measuring software development result cannot be reduced to a time-spent unit such as man-month. As early as in 1975, The mythical man-month became a famous book that explained this in different ways, including the mention that adding resources to a late software project could only delay it more. The book was republished in 1995, a confirmation of its accuracy, but also a proof that the necessity of -explaining this to non-software people remained fully necessary 20 years after. I have no doubt it will be necessary to keep repeating the message in 2015, as software production remains a widely misunderstood discipline among decision-makers.

image

A lot of people keep comparing this to the physical industry where the same component needs to be produced thousands or millions of times with the exact same process. In software, a source code is produced only once as it is immaterial and can be copied as needed at almost no cost. And still, even when comparing with industry, one should think whether a team of 10 guys with shovels would dig a hole faster and at less cost than a team of 3 guys with bulldozers. So it is not very hard to understand that each business evolves in method and tooling. Software is probably even more concerned than many other businesses, considering the technology evolution pace.

Emergence of off-shore

Still, since the emergence of the "off-shore" model in the late 1990s, the focus on hourly cost of developers has peaked without many voices in the software industry to challenge these new software production models.

The reasoning was as simple as the following most of the cost of software development is the salary of developers, as it is a time-consuming activity, which is true. So if one goes to countries where the time is less costly, it will cost less in the end. This fully neglects the importance of the methodology, in particular the interaction process with users or product management, the skills and the tooling but who cares?

At that time, the development of cost-killing methodologies, combined with the power given to purchasing departments and their basic comparison methodologies and the absence of a relevant metric to measure software production (beyond the man-month) made this reasoning the general trend in the industry, leading to what I would now call "The mythical Indian man-month syndrome", with Indian developers being of course cheaper than their western countries equivalent with higher salaries.

Despite the awareness of numerous field failures (one can read this 2004 article as an example and notice the cautiousness of the title), the trend has continued to date. To some extent, skills and part of the production methodology has of course improved in the off-shore countries, but the most important point about interaction with business users remains.

20% savings "when it works"

As the president of the R&D focus group of the AFDEL (French software vendor association), we had a session of experience sharing about off-shore. Not only were mentioned many failures, but the fact that struck me the most is that the successful projects talked about 20% savings in the end, when including all the hidden costs that were needed to make it work. And it was also mentioned that it required about 2 years of ramp-up to make it successful.

Interestingly, I found this interesting CIO magazine article that confirms this observation made as early as in 2003! This article has the merit of listing any of the costs, as well as explaining this "20% savings in the most favorable case" reality.

Understanding the harsh reality about off-shore should be no surprise, as I personally love one comment on this article:

It is very difficult for business users and IT workers residing in the same locale to work out software requirements and successfully execute a project. In fact, some statistics say that around 80% of IT projects fail to meet their goals. Now imagine moving the technical team several thousand miles away, put them on a work schedule that is completely opposite that of their end-users, give them a different native language, and give them a completely different set of cultural mores and norms. With as much sarcasm as I can muster, I must say that none of this seems intuitively likely to increase the odds of success, but it does have the "advantage" of being really cheap.

Paying a low price for something that does not fulfill your need is for sure a wrong spending!

Probably more than 100% extra costs in other cases

Now let us come back to field experience, off-shore "per se" is not the solution to the numerous challenges faced in software development. This is why we observe many failures on the field where we estimates costs being around almost the double as they should be, not only in the short term, but over the long run.

It is not the matter about whether developers are good or not in some geographies (although there are geographies with more or less skills) but mostly the nature of software development that makes it challenging to work distantly from users.

Trendy agile methodologies are smart enough to put an emphasis on the proximity of product management role. We have developed the importance of alignment in our series of post about "Measuring software development performance", that developed 5 key critical dimensions:

  1. Alignment
  2. Productivity
  3. Quality
  4. Debt minimization
  5. Predictability

If the team is large, you might have a chance to run for the 80% side of the spectrum with the "scaling effect", but even so, it is quite challenging to reach it. And we have seen many customers moving back from off-shore models and now hiring local resources.

It is also worth mentioning that many failures on the field are just taboo and then perfectly explained by technical teams, taking advantage of the difficulty for decision-makers to really measure success. It is quite common in this industry and is of course exacerbated with the complexity of a distant team.

Beyond the costs

Beyond the costs, going too far with off-shore often causes:

  • Skills challenges, as you may not find the best technical solutions,
  • Loss of control, as you may depend on external partners that do not have the same interest as yours,
  • Loss of agility, as you may require months to change even simple business requirements,
  • Increased risks of various nature because of the distance, the language, the culture and sometimes even the legal or intellectual property risks.

In the end, by externalizing too much of your development, you will lose the skills to even evaluate whether you are doing well or not. One could also mention the citizenship dimension but this is not even my point here.

This is why, even if off-shore is often a failure, few people publicly admit it, leading to a significant market hypocrisy, which really contrasts with discussions experts have together when they share their experience in a face-to-face mode.

Seriously, as an informed decision-maker, would your risk losing control for an uncertain potential 20% savings?

I personally would not we have never even thought of externalizing the R&D of our software, as an example.

As we have written in a previous post, it is only with the contribution of software engineers that you can make a real long-term difference on the market, as those guys are the ones able to make some critical choices. Some choices can impact the software development cost with another order of magnitude while delivering the same value!

So make sure you keep at least some local skills, or you will expose yourself to losing control at some time in the future.

Daniel COHEN-ZARDI

SoftFluent CEO

Filed under News, Software Development Tagged with , , , , , , ,

Motivating developers

December 4, 2012 1 Comment

Mazlow’s pyramid applied to developers

Approaching the mid 40s, I have now managed a quite significant number of developers, directly or indirectly, by also observing many development organizations as an external consultant.

Every manager in the development business will also tell you that you don’t manage developers like you manage other people. There are very good articles about it on the net, and I started this article by making a simple compilation of those developer motivators, also to make sure I do not forget a key one.

At the same time, every HR consultant will also explain you that managing people is a discipline you can learn, and there are many theories and best practices about it. Then, those theories should apply to developers, as they are also human beings. So how can these two professionals be right at the same time?

The answer I came up with is that, using the same theory, it could be interesting to just consider that needs and motivators are expressed a different way for developers. This gave me the idea to apply the good old Mazlow’s pyramid to developers… by tweaking it a little bit to transpose the needs at a slightly higher abstraction level.

As a reminder, here is what the Mazlow’s pyramid is all about. It explains that motivation is a kind of hierarchy, and you can’t jump at the higher level if the needs below are not fulfilled. Please HR professionals, don’t blame me if this is too simplistic or if you are a fan of another theory, this one has the merit to be quite easily understandable.

Mazlow

Putting my compilation of all the key motivation elements I selected as important for the developers, here is how I would transpose it for developers.

Physiological needs

1. Give developers the right tools to do their job

Being equipped with the proper level of hardware and tooling is as critical as breathing for a developer. If you don’t give them the right tools to do their job, not only will you lose precious hours of productivity, but don’t expect to go anywhere higher on the pyramid. Giving undersized hardware to developers or putting them in noisy open spaces is also a sign that you don’t value their job.

2. Refrain from excessive process constraint

Development is a job that requires some level of creativity (which is sometimes also an issue when not used at the right place but this is another story I will develop in another post). Being creative is not compatible with heavy process constraints. You cannot find a solution to a problem by just planning every second of your time or being disturbed by an excessive number of meetings.

3. Reduce their inherited technical constraints as much as possible

Systems live longer as expected. As a consequence, there are often some level of technical constraints in your project: an old database schema that cannot be changed, a poorly-designed legacy system that adds some rigidity or a customer that imposes usage of some specific technology. The more of this you have, the less comfortable the developer will be. Reduce those constraints as much as you can. One key to this is to really makes this a goal, even if you take a multi-step approach to make it compatible with your business. Keeping legacy constraints other time is very costly in terms of maintenance anyway.

Safety needs

4. Ensure the work they do will be used

It might seem obvious, but it is not. Since there is a high level of project failures in the software industry, developers that have already faced a situation where all their code was dropped might not feel safe as some issues arise on the project. And do you know of any projects where no issues popped up? The best way to do this is of course to repeatedly deliver your projects by correctly managing them. Implementing continuous delivery through agile methods is also a guarantee not to have a multi-month tunnel effect that can lead to a large drop of code base, which is not only a financial loss but a huge motivation killer.

5. Compensate them at the appropriate level

Compensation is clearly not the ultimate goal of real developers. Still, it is of course an unavoidable need as for any human being on earth. You can work much better when your personal requirements are fulfilled with some increasing level of comfort at home. Making sure there is some regular progress in compensation as the skills and the achievements grow is also very important. I personally position it at "Safety" level of the Mazlow pyramids for developers.

6. Provide reasonable career path visibility

Equally, developers will feel safe if they see their job as sustainable, and even better, as providing a path where they can climb some steps. They are not striving for career in the sense of managing more and more people, but they for sure value the titles that recognize their knowledge and expertise, as well as their ability to design complex systems or to mentor others. This is why all successful software companies have titles such as "distinguished engineers" or "technical fellows".

Belonging needs

7. Share the big picture

Once the above needs are fulfilled, you might be able to climb at the "belonging" level of the pyramid by creating a team effect. One key element of creating the belonging feeling is by sharing the big picture of the project. Don’t expect too much from developers if you just give them the tasks one after the other without giving any meaning to what they do. It is very bad for motivating teams, not to mention the fact that you will probably lose opportunities to improve the overall design.

8. Manage them through field people

This is a very important one. A developer will not find legitimate to be managed by someone who never was a developer himself and cannot fully understand their job. Considering the nature of software development, I think it normal. It is quite impossible to understand the issues developer faces if you have not done the job before. A good development manager does not need to have developed for 20 years, but he (or she) should at least have developed for some years to make the right decisions when they become necessary.

9. Favor technical discussions between peers

The more your developers discuss about technical issues and solutions, the better. Not only it will make the team more efficient and skilled over time, but it will also contribute to this belonging feeling that increases developer satisfaction and motivation.

Esteem needs

10. Make sure they always learn new things

This one is not always easy to achieve at every moment, depending on your business constraints. Fortunately – for this point because it can also be a real thorn in the side for your budget – technology evolves quite fast. So if you proactively work on removing the legacy constraints and provide enough guidance to developers, there is certainly room for learning new things on all projects that I have seen.

11. Recognize good work

It might seem obvious but people often forget about it. Everybody needs to be recognized. Depending on the nature of the people, you can do it publicly, or more intimately, but tell the developers you are happy when they achieved a good job or when they worked hard to meet a milestone. Simple recognition gifts or even just a few words like "Good job !" can make a difference. And there are moments that clearly need to be celebrated collectively.

12. Encourage external contribution to communities

Beyond the technical exchanges inside the group or the company, you might encourage your people to be active in the community. For younger folks, it might mean more listening than contributing, but it is always a good idea to do so, as it also gives them ‘icons’. For more senior people, if you have very good ones, give them time to contribute and get recognized in the community, especially if they have skills to write about technology or to speak at conferences. It will certainly have some positive impact on your business anyway.

Self-actualization needs

13. Make them accountable from A to Z

Reaching the self-actualization level of the pyramid should be a goal for development people. It is unlikely to get reached in the early days of your career, but having the big picture in mind is a good motivation. One sign that you are getting there is that you start to be fully accountable of what you deliver. This might be high pressure but it is also the counterpart of getting maximum freedom as per next point.

14. Put them in control to use their creativity

Developers are at the highest degree of satisfaction when they can control every decision on the project. Being able to make all architecture and tooling choices is very satisfactory. It goes with the accountability of point 1, which can be dangerous if you don’t have all the skills, but this is what all senior developers should strive for. Once you get there, you can express all your creativity to find the most relevant solutions to your business scenarios. As a manager, you have to portray this target to developers to motivate them and keep your best people at this level on freedom.

15. Contribute to something big (or cool)

The ultimate level is reached when, additionally, what you have built is "big", or possibly more important for developers, "cool". Cool is hard to define as it can take several form depending on your profile and sensitivity, but it may as well be the resolution of a tricky issue as the contribution to a project that has worldwide visibility or usage.

Listing all those key elements to motivate developers, here is the developer version of Mazlow’s pyramid I was able to draw.

MazlowDevelopers

As usual, feel free to comment.

Daniel COHEN-ZARDI

Filed under Software Development Tagged with , , , , , ,

Measuring software development performance – Part V: Predictability

October 31, 2012 Leave a comment

Through our previous articles we elaborated about the first 4 performances axes of development teams:

    The fifth important element to take into account in order to evaluate the team development performance is the predictability of the results.

According to various studies also referred by the Afdel in the White Paper about driven innovation: 58% of purchased software are never put into production, only 13% of IT projects are delivered on time and 30% are purely and simply stopped. Thus, it is clear that the issue remains on this axis to drive and predict software projects.

Obviously, it is very difficult to compare the predictability level of a development team performing management applications in a relatively stable market functionally speaking and a team of research and development of an innovative software company positioned on a technological field close to fundamental research.

Be that as it may, the purpose of any development team is to achieve results, that they are either proof of concepts, software to be marketed or applications for internal use. In the business applications world, taking into account the current state of the art, it seems to us particularly feasible to obtain consistent results in delays and costs that can be evaluated using relatively simple and limited macroscopic criteria.

For software companies, it is important to be able to formalize a roadmap as well as releasing regular versions of it. Although, it is known that these roadmaps are hardly followed, big gaps can lead to disaster since they are often linked to financial forecasts. Below is an example of a traditional roadmap about Microsoft Visual Studio for the past years:RVB de base

The trend towards the "Software as a Service", where competition is aggressive, generally requires more regular release like "Web" companies do (whose business model is based on the Internet such as business to consumer web sites). It is then essential to integrate continuous improvements, which is actually creating a roadmap. In the latter case, the roadmap is simply built the other way, with fixed dates and a variation of what can "fit" into a given release. Generally it is aligned on a season or semester. Here is an example for the Microsoft Dynamics CRM’s offer:

Livret-blanc-visuel1_v1

Note that software companies often need to make new products to differentiate themselves from competition and maintain their market position. We will highlight this fact afterwards when talking about the extra dimension of creativity, especially for teams of a software companies with a Research and Development department.

The issue is quite similar for large companies because they must be able to orchestrate the release of business applications meanwhile preparing deployment and training as well as users support. Ideally, development department of large companies should behave as small software companies. Actually, as organizations, this point is becoming a market trend.

From our experience, many failures are due to a mismatch between the expected result and what is really produced. This discrepancy is often due to a rupture between the design and development stage. The off-shore has contributed to some disasters in that area.

Trying to imitate models from traditional industry, some have forgotten that – contrary to the ‘physical matter’ industry – the implementation phase of a software is never the reproduction process performed identically. Each development is unique by nature, since it is intangible and can then be replicated at zero cost.

This reasoning transposed into the world of software development has now shown its limits hence a movement of reflux of these models at low cost begins to emerge… with problems related to the lack of resources that the model has created.

It is also common, especially with technological breakthroughs, to find teams who lose control of timing and R&D projects that take several months or years of delay, with budgets growing in the same proportions leading to massive deficits. This phenomenon is very marked in software companies, as large waves of investment are sometimes more than a decade away with a major cultural leap that is not assessed by the management or by teams.

It is therefore essential to have regular milestones including deliveries of versions and new features to avoid the tunneling effect of some major projects. This effect can create huge gaps in the product roadmap.

This is why agile methods are also interesting to prevent from the tunneling effect by maintaining alignment between the holders of the "product" vision and the production teams. Besides, some indicators of predictability are also included in these methods as a percentage of each achieved iterations compared to what is expected at the beginning of the iteration.

RVB de base

Even though this aspect is clearly useful and interesting, it is important to have the right debate here. Predictability to reach is beyond the scope of the measured work at each iteration – usually 2 to 3 weeks – and it must be achieved at a more macroscopic level as mentioned before with the roadmap.

To conclude, note that the issue of predictability is stronger than it seems, because beyond the direct financial consequences of a delay, when the loss of trust is established, the team dynamics rapidly becomes a vicious circle. Schedules are taking delay, functional managers "load the boat" because they know they will have to wait too long for postponed features, and the project becomes more exposed to the risk of a major failure.

Filed under News, Software Development Tagged with , , , , , ,

← Older posts

Follow

Get every new post delivered to your Inbox.

Join 103 other followers