I have always marked my .NET assemblies as visible to COM with [assembly: ComVisible(true)], thinking that I never know when someone might need to call them from COM. I also started using FxCop and started seeing this warning from code analysis:
CA1017 : Microsoft.Design : Because 'MyLibrary.dll' exposes externally visible types, mark it with ComVisible(false) at the assembly level and then mark all types within the assembly that should be exposed to COM clients with ComVisible(true)
Is there some reason why you would not simply want all of your public types exposed to COM? I'm guessing that there is, but I cannot imagine what this reason is. If anything, it seems markedly inconvenient.