Release management: SoftFluent’s experience – 12/12

Common mistakes as observed on the field

This series of posts aimed at sharing our experience. We detailed our own approach and how we deal with the different dimensions internally as an ISV.

But as we also have field experience in advising software companies, I thought it would be worth to also gather-up a list of common mistakes which we observed and that can severely impact the ability to master the product release management process.

  1. Linking product version to a particular customer: In the field, we see many companies claiming to be ISVs but in fact linking strongly product versions to one or a few customers. In our experience, it is very hard to keep a real product-oriented approach when you have a too strong customer connection. It often ends up in having a custom development that is too specific for the customer context.
  2. Setting too high expectations for version 1: Some ISVs also set very high expectations from the beginning, or for their new offering if they start a brand new product. Trying to address all needs in a very complete way in a first version is doomed to failure most of the time.
  3. Starting without a minimal vision of sellable product scope: On the opposite side of the spectrum, we also see teams that start without having a real vision of what minimal scope can be achieved and sold as a product. Pushed by the trendy messages about agile methodology, some people think you don’t need to know very well where you are heading to start your project. In most cases, this does not work well. In our views, agile methodologies work well when used as a delivery method inside a more macroscopic global planning process at vision level.
  4. Neglecting documentation: Because it is not very fun to write documentation, especially for developer profiles, this area is not enough covered with the appropriate quality level. As an ISV, you should never develop a feature that you are not ready to fully document. Isn’t it obvious written that way?
  5. Underestimating tests: Test and QA is a critical area for software success. It encompasses many dimensions (see our previous post about software testing best practices) and should be allocated the right level of resources and attention.
  6. Considering coding as pure low-added value production: We see some ISVs trying to apply the recipes of industry with a simple scheme saying that all the value is in the design, and that code can be outsourced to low-level service companies, those companies being local, near-shore or off-shore. It can be true for some very basic management software that is broad in coverage but very simple in the features. Still, most of the time, you would be able to achieve much more efficient results with 1/10th of the investment with software engineers of the right level, and above all, the proper level of experience.
  7. Subcontracting what is not internally mastered: I am a big believer of specialization and subcontracting. This can prove to be very useful when used properly. But there is a golden rule that I learned over time: you cannot subcontract what you don’t understand well. You need to have the ability to pilot and control the work of your subcontractors, and you cannot do this if you lose the basic knowledge of key elements. This applies strongly in all software dimensions that you might want to subcontract.
  8. Going for a technological leap without measuring the methodology impacts: I have audited tenths of projects that have fallen into this trap. Despite the platform vendor pitch that everything is getting simpler with new technologies such as .NET or Java, this is simply not true. These platforms give new technical opportunities and connection capabilities, but they are more complex and bigger than ever. An ISV switching from a traditional CASE Tool based approach (such as Progress, PowerBuilder or NSDK to name a few) is more than likely to fail his new projects if he does not measure the impact of technology change. The methodology needs to be adjusted as there are many implementation possibilities in those open platforms. Finding the right balance on many topics is more than tricky.

As a conclusion, we do hope that the experience we shared might be useful to some people in the field. These are only our views, we might have missed some points and we are always eager to learn more, so we welcome any feedback on those elements.

Daniel COHEN-ZARDI

CEO, SoftFluent

Release management: SoftFluent’s experience – 11/12

Tools summary

Considering the numerous elements detailed in these discussions, we mentioned different tools that are useful to our Research and Development department.

 

Area

Tools

Roadmap Planning

Internal tool: Excel lists of modules and features

Product Design

Microsoft Visual Studio Team Foundation Server 2010 & especially WorkItems (tasks & bugs)

Development

Microsoft Visual Studio 2008/2010, JetBrain Resharper, Redgate .NET Reflector

Embedded in our product: Actipro SyntaxEditor

Documentation

Innovasys HelpStudio & DocumentX

Internal Tool for schema « DocMaker »

Testing

Rich internal tooling for UI automation

Build management

TFS Build, MS Build, Internal tooling, WIX, CodeFluent Templating Engine

Release/version management

SoftFluent.Licensing, Direct integration with our web site

Licensing & activation

SoftFluent.Licensing, internal technology for activation and obfuscation

Support

WordPress blog, YAF forums, Customer-organized dedicated mailbox

Release management: SoftFluent’s experience – 10/12

Product support

Our product support team leverages two key tools to provide you with top level information. First, the team regularly selects product features and posts an example and an explanation to draw attention to this feature on our product blog. By subscribing, you will learn a new tip almost every day, directly from SoftFluent team members.

image

Beyond this one-way information process, the support team closely monitors our forums and make sure all questions get answered in a timely manner.

If this looks like a bug, we start by trying to reproduce the issue, then if confirmed, we then escalate it to R&D. If the question is more open, or if the behavior is by design and you need to address it differently, our team may propose solutions or directions to look at. If necessary, you can also ask for paid consulting hours to help you implement a complete solution.

In the end, we get constant positive feedback about our reactivity and we work hard to maintain this level as we know this is a key factor to customer success with CodeFluent Entities.

image

Release management: SoftFluent’s experience – 9/12

Version & release management

SoftFluent implements its own license protection mechanisms, which include client and server-side tools, as well as activation and obfuscation mechanisms.

Once a build is produced, and if it passes our quality assurance tests, we make it public. This is what you can see in the client-side tool that you get when you download CodeFluent Entities. Builds with the green flags are considered valid and fully tested.

image

On our side, we have a licensing application where we can manage the builds released on the web, as well as the different information related to licensing.

This is where we can upgrade your free personal license to a commercial one for example, and modify the appropriate metadata that goes with the license. This is also the tool we use to manually activate licenses if you use the email-based option.

image

Another interesting point of our release management process is the “Product update blog”. As soon as the developer has finished a work item that we define as ‘public’, he documents the modification with a user-friendly comment, whether it is a new feature or a bug correction. This comment will directly feed the “Product update” RSS, giving you all the information about what is in a new build.

image

Release management: SoftFluent’s experience – 8/12

Build management

The build management process is also a key pillar of successful product release management.

We can launch the build almost any time, which means we do the check-in/check-out in a way that avoids breaking build. We don’t do continuous build, as we don’t really see the benefits of this in our case. The fact is we launch a build once we have implemented a consistent set of features and at least once every 6 weeks.

The build process, triggered by Team Foundation Server Build takes care of the following:

· Builds all binaries

· Obfuscate all binaries that are marked as obfuscated

· Signs all binaries using our company’s certificate (Authenticode)

· Builds final installation files (.MSI) and also signs them using Authenticode. We use WIX (
http://wix.sourceforge.net/
) to build the .MSI files. WIX files are in fact dynamically generated to automatically include numerous files we need to ship with the product, which can vary per build.

The whole process also handles Debug & Release variations as well as 32-bit and 64-bit platforms targets (see CodeFluent Entities & Bitness for a prospective in our context).

The build process relies on Team Foundation Server Build as the screenshot below illustrates:

image

As soon as we have delivered a new build, we also integrate platform tests and play the setup programs on various combinations of environments.

We don’t use all the possible variations but we mix them in ways that we are sure to test:

· Different OS versions including 32-bit and 64-bit,

· Different language versions,

· Different database versions,

· Different Visual Studio versions (we support 2008, 2010 and 11 developer preview as of today).

Release management: SoftFluent’s experience – 7/12

Documentation

Our documentation process is tightly coupled to our development process.

Immediately after having completed a feature, we do the relevant part of the documentation. We use Innovasys HelpStudio for this and we generate the “Reference documentation” on documented assemblies using DocumentX. However, we have written a custom tool (named “CodeFluent DocMaker”) to create the documentation files concerning all XML schemas used by the product.

All this works well once you get the right development discipline in putting the comments where appropriate in the code files

The documentation is generated as:

· a .CHM file,

· an HXS file,

· a 800+ pages PDF booklet,

· and also published directly on our web site.

image

We decided to make our product documentation public for two reasons:

• We want all developers to see the extended possibilities of our product,

• We get the benefit of Google indexing it which means CodeFluent Entities developers can easily find answers to their CodeFluent Entities topics by doing Google Searches.

Release management: SoftFluent’s experience – 6/12

Test & QA

We have a broad product which means a large complexity to test all the different combinations of options. Because of this complexity, we quickly realized that relying on “classical” testers – even off-shore testers to reduce cost – would be prohibitive and would require significant time to assess quality at each new release.

This is why we decided to invest a lot in automating most of our tests. We did this quite early on for CodeFluent Entities Core Edition (this edition is the XML-based standalone one without the graphical Modeler integrated to Visual Studio) and this lead us to having tools that compare the files generated by different builds.

At each build, we generate numerous models, the sample models delivered with the product, models that were designed on purpose to use all the product options and maximize the coverage of our tests, and real customer project models to limit our regression risk to the maximum.

We then browse the results of these automated tests, and the created test report highlights the differences (see below). Sometimes those differences are normal (a change that was decided), but if not, we can quickly identify the bug and make a correction before releasing the build.

image

Since the moment we developed the Modeler, we also needed to test the user interface. This is even trickier as classical UI test tools are not very resistant to the real changes that occur during the development process. If a control is moved to a different position on the screen, it might be necessary to rewrite the scenario, etc…

So we built our own test framework to introduce the concepts specific to our product and expose an API to create test scenarios that can be programmed and replayed on the UI.

image

Actually, we rely on the two pieces in “blue” that are provided by Microsoft (and are available out-of-the-box on every Windows installation), but we developed the framework following the described layers above to:

· Handle typical scenario concepts, independent from any product: session, log, trace, screenshots, mouse, keyboard, etc. (SoftFluent Automation)

· Encapsulate the main user interface elements of Visual Studio like the Solution Explorer, the Property Grid, New Project Dialog, Main Menu, etc. (Visual Studio Automation)

· Encapsulate the main user interface elements of the CodeFluent Entities Modeler: Ribbon, Menu, and all specific dialog boxes (CodeFluent Entities Automation)

Our end-user test scenarios then leverage those layers to simulate actions on the user interface and we can play complex scenarios for hours on different machines to maximize our test coverage.

Our tooling records everything that happens and takes a snapshot of the user screen whenever we ask the scenario to do so, or when an error occur (see below).

image

This is a very powerful approach as you don’t have to redo all the work at each build. Of course, you test also the scenarios in the process, so errors are not always a product issue but it helps a lot in finding issues quickly. Without huge human bandwidth, we can detect most of the regressions that may happen.

Usability tests is also an area that we are thinking about to improve ergonomics and make sure developers get the best experience with our product at first try.

Release management: SoftFluent’s experience – 5/12

Development

Our development process is iterative. It is not a pure academic agile methodology but we apply some of the principles that we like in those agile approaches. We organize ourselves to deliver a new public build every 4 to 6 weeks.

We manage to do so by using rough estimates and by grouping features and bug corrections that relates to similar topics. Depending on how fast we implement, we sometimes take the opportunity to squeeze minor features that were planned later.

We do code reviews, not only in peer mode, but also with a direct and non-complacent eye from our CTO. This ensures that we do not only get the expected results but that coding rules are well respected, especially for maintenance and evolution capability purposes.

We also use tooling and generate part of the code where appropriate. In some areas, CodeFluent Entities is built using CodeFluent Entities, which is a classical consequence of going quite far in computer-aided software engineering.

With all the platforms and target architectures that we support, the product now includes 48 different projects and about 600 000 lines of hand written code (without counting neither blank nor generated ones).

image

Release management: SoftFluent’s experience – 4/12

Product design

Inside a specific module, product design is the process that will detail the features and the behavior the product should have for the different scenarios.

Basically, we follow a quite simple approach on the dimension. We write short specifications that mostly describe the typical customer scenarios that we want to support. We don’t go very far into detailing the user interface as we have a developer-oriented product, but we do it when it is relevant to align the whole team on key decisions. We did this for our starter wizard for instance.

In fact, when it comes to specifications, we are true believers that almost everyone should be able to read them without requiring cryptic formalism or a bachelor degree in UML modeling. I summed up our philosophy on this topic on a former post: Formalizing specifications.

image

We create test cases, and, because of our offering, this often means mini-applications that are used as samples to illustrate the functionality. At some stage, we usually discuss the supported scenarios as we always have 1,000 different ideas of possible features which unfortunately we cannot all implement.

For bugs that are detected by our support team (or by customers who contact support), they are first reproduced by support to be fully qualified and sent to R&D, with the exact conditions for the bug to occur. We then plan the correction and integrate the use case in our test scenarios.

In the end, the feature implementation and the correction requests end up as work items in Team Foundation Server and are distributed across the R&D team.

Release management: SoftFluent’s experience – 3/12

Roadmap planning

The planning of our roadmap is mostly driven by our roadmap committee. Every month, we hold a 2 hours meeting where we:

1. Review modules that are in progress at R&D

2. Dig into some feature sets when necessary to fine-tune the limits of a specific module version,

3. Discuss upcoming modules and update the priority list if needed

4. Discuss important customer-oriented requests when there are, and when those are well aligned to our roadmap.

On this last point, let us take some examples about what are valid customer requests in our views:

· In the past, adding Oracle Database support to our CodeFluent Entities product by developing a new Oracle Database producer was aligned with a customer request. We knew we wanted to do it, but we waited until a customer had a real world project to implement it. This way, we made sure the developed module went quickly into production and gave us real-world feedback right away. This was all the more important that we are rather a “Microsoft shop”, and getting the feedback of a customer on Oracle was very important.

· Currently, we are prioritizing Visual Basic support for the middle tier generated by CodeFluent Entities (a.k.a. “Business Object Model”) based on consistent customer feedback. This is not due to one customer in particular, but instead we are getting requests for this feature from multiple customers in the community. Likewise, we are in the process of planning MySQL and PostgreSQL producers.

· In the future, we might consider generating Java code, as this would open a larger market, especially in geographies where Microsoft is poorly recognized.

· On the contrary, if a customer asks us to implement a specific feature in the “business object model”, like for example doing caching a different way, we won’t put this in the roadmap planning right away. Instead, we will advise the customer on how to do a custom extension himself, or we can do it for him as a consulting request.

In the end, we formalize our roadmap using two tools:

· Simple Excel lists of modules and features, synchronized on a central database,

· A visual slide that sums up modules that were delivered and those to come. It might seem unusual to detail what was done in the past and the timing, but we think it important as the value of our offering is about digesting Microsoft technology at a relevant pace. And the slide says more than words could about this (see below).

image

Follow

Get every new post delivered to your Inbox.

Join 103 other followers