Here are the lessons I learned in the process:
Do not overwrite existing files using <File> elements.
When your software gets uninstalled, it will remove your file, but the old file won't be restored. The solution that first popped up in my mind, namely to backup old files, isn't as easy as it seems. The problem is that the backup copy will be left behind after the software gets uninstalled.
<CopyFile> is not your friend.
...at least as far as backing up existing files goes. I tried to back up the old file by copying it into my application's folder using CopyFile, and failed miserably. Maybe I missed something, but it seems CopyFile is designed to make copies of the files you install, not of existing files.
<CustomAction> is way too complex for its own good (and yours).
CustomAction can be used in many different ways, using slightly different syntaxes. I attempted to use it to backup files, and finally succeeded. It wasn't easy. I tried to use it to run a command (COPY), but it failed. Same problem with other commands such as REN and DEL. No success running batch files either. I tried running a jscript, and that failed too. Since the creator pf WiX advises against using scripts in custom actions, I did not insist. I ended up using "xcopy.exe", which worked (with a catch: the target must be an existing directory, otherwise xcopy demands to interact with the user, and we don't want that).
I found ONE use of CustomAction, and I am not planning on trying out other uses. At least not for a while.
<RemoveFile> is your friend.
Unlike CopyFile, it's easy to use to clear up stuff in your application's folder. I used it to remove the backup after restoring it during uninstallation.
In case you wonder how to handle back-ups in WiX, that's the way I did it:
<Property Id='XCOPY'>xcopy.exe</Property>
...
<CustomAction Id='Backup'
Property='XCOPY'
ExeCommand='"[ExistingFolder]PlugIn.dll" "[OldFiles]" /Y'
Result='check'
Execute='deferred' />
<CustomAction Id='Install'
Property='XCOPY'
ExeCommand='"[NewFiles]PlugIn.dll" "[ExistingFolder]" /Y'
Result='check'
Execute='deferred' />
<CustomAction Id='Restore'
Property='XCOPY'
ExeCommand='"[OldFiles]PlugIn.dll" "[ExistingFolder]" /Y'
Result='ignore'
Execute='deferred' />
<InstallExecuteSequence>
<Custom Action='BackupPlugIn' Before='InstallPlugIn'>Not Installed</Custom>
<Custom Action='InstallPlugIn' After='InstallFiles'>Not Installed</Custom>
<Custom Action='RestorePlugIn' Before='RemoveFiles'>Installed</Custom>
</InstallExecuteSequence>
...
<Directory Id='NewFiles' Name='New Files'>
<Component Id='NewFiles' Guid='...'>
<File Id='NewFile' Name='Plugin.dll' Source='build/PlugIn.dll' />
</Component>
</Directory>
<Directory Id='OldFiles' Name='Old Files'>
<Component Id='OldFiles' Guid='...'>
<CreateFolder />
<RemoveFile Id='CleanUpBackup' Name='PlugIn.dll' On='uninstall' />
</Component>
</Directory>
Note how I avoided overwriting files in the "Component" elements. I do that using CustomAction instead, both when installing my plug-in and when restoring the old one.
Something that's missing are rollback actions. There should be one for each deferred action, namely an action removing the xcopy-ed file. I don't know of any standard program to remove files on Windows, and I'd rather not write one myself. Where is "rm"?
UPDATE:
WiX experts are probably shaking their heads when they read this code:
<Custom Action='RestorePlugIn' Before='RemoveFiles'>Installed</Custom>
</InstallExecuteSequence>
While it does do what I intended when the user removes the software, it is also executed when users perform other actions than install or uninstall, such as repair an installation. As a result, an attempt to repair an installation will restore the old plug-in, which of course is more likely to destroy the installed software...
