The pains of upgrade and how to alleviate them

User customizations! The same thing that many times makes the product successful, turns into a nightmare for its developers. Anyone who faced the challenges of deploying an upgrade over an existing installation knows what I'm talking about. If only things could be as simple as scraping the old installation and unpacking a new set of files! Alas! The cunning user has made changes in our precious files! He loves our product because of that! Everything is stored in readable, textual files! He reverse-engineered them and discovered that he could turn our great product into a fantastic one by tweaking these two little parameters! And now we're overriding everything! Oh, no!

My experience taught me that although the problem happens again and again, there is a set of rules which if followed judiciously, helps if not to avoid the problem altogether, then to minimize it at least. I'll state the rules and explain them with a set of examples.

Rule #1

Draw a clear line between the things you allow your users to customize and the things you don't. For instance, if the configuration is stored in files, put the customizable files in one dedicated folder. Name the folder appropriately. When the structure is organized this way, even if it is not documented, it conveys a message to your user - this is the area where you can make customizations and only here!

Rule #2

Separate the factory (out-of-the-box) settings from the user settings. For example, if you have a configuration file with default settings you allow your user to change, try to split it in two - "your" file that contains the default (hidden deep inside your files structure) and "user" file, which contains comments guiding him what and how he can customize.

Rule #3

Do not expose the factory defaults in a form that provokes changing them. For instance, instead of placing them in a file, place them in a resource packed inside a JAR and read it as an InputStream.

Sometimes the reality is tough and you find yourself being led by your users. I mean that sometimes a majority of them starts customizing things you didn't intend them to customize but you have to support it because you value your users. Even in this case, try to fix the situation post-factum by organizing things differently and implementing a clever upgrade tool that would transform the existing situation to a new format.

Why is it important to follow these rules? The first is rather simple - it is unpleasant to find yourself in a situation where you have overridden user changes in a certain place just because you didn't have a clue anyone besides yourself would want to change it. As I have just said, it can still happen even if you follow the rule #1. But in this case you'd at least have an excuse - this specific file does not reside in the "customizable" area, we didn't intend to support customizations done there. Again, if the user has a justified case, it's a subject for negotiations and support, but deep inside the user will know you're right.

The second rule is less obvious and one has to make a mental effort to follow it. Many times the format of a configuration file is predefined by some third-party tool or library we use and it's very easy to miss the point of decision. It wants a settings file - let's create this file and use it. The problem is that you'll never see any problem with it until the next release where you come with an upgrade. Then several bad things can happen.

• You may upgrade the third-party library and see that it needs an additional parameter. If you override the configuration file, you override your user's possible modifications and make him angry.
• You may reach the conclusion that you want to change the default value of one of the parameters. If you change it blindly, you may break the behavior for those users who customized it. If you don't change, you save them but the rest of the users don't get the benefit of the improved default.
• The third-party library provider may decide to change the format of the configuration file. It rarely happens (everyone is in the same game) but if it does, you're in a deep sh*t.

However, if we consider these scenarios in advance, we may save a lot of headache in the future. This is the time to be creative. If the format supports the "include" directive, use it to split the file into the "factory" and the "user" portion (placing them in different directories as per Rule #1). If not, perhaps you may simulate it yourself by first parsing the two portions and combining them into a single temporary file to be fed to your third-party. The format is not friendly for a split? Perhaps it's worth to invent your own and again, convert it on the fly to something the third-party expects. If the third-party supports configuration by API this might be the best option. The same rule holds if you store the configuration in the database - do not mix the defaults and the user settings in one table because of the same problems outlined above. In short - think forward and try to save yourself from a complicated upgrade in the future. 

The meaning of the last rule is - if you don't want your user to customize something, don't provoke him. For instance, if you have (for any reason) a file that lists the names of your classes implementing a certain interface, do not store it in the filesystem. Turn it into a resource and hide it inside a JAR file. In general, avoid files because they are much harder to keep track of, to redistribute, to overwrite etc. When something is a resource, you can be 100% sure it is available, up-to-date and matches the corresponding code. So resources should always be your first choice. Resort to files only when it is absolutely inevitable.

To summarize - I believe that the absolute majority of upgrade problems can be prevented if the features are written with the upgrade in mind. The complexity of upgrade is not inherent, it is mainly caused by our own design mistakes. Design for upgrade!