Like many developers, I have both VS2003 and VS2005 installed on my machine. I had also placed a shortcut to the Visual Studio command prompt in my programs menu. A couple of days ago, I had the opportunity to write my first Windows service application. I got everything set the way I wanted, and it was time to install. So, I opened up the VS command prompt, and typed in the magic chant "installutil servicename.exe". And fairly promptly got the error "Exception occurred while initializing the installation: System.BadImageFormatException: the format of the file 'servicename.exe' is invalid...".
If you're encountering the same error, make sure that you're using the correct installer. For me, it turned out that I was using the VS2003 installer. As soon as I switched to the installer from VS2005 (using the correct command prompt), everything worked just fine. It took me a while to track this down - perhaps this will save you some time.