Spinning horizonless compact objects may be unstable against an "ergoregion instability." We investigate this mechanism for electromagnetic perturbations of ultracompact Kerr-like objects with a reflecting surface, extending previous (numerical and analytical) work limited to the scalar case. We derive an analytical result for the frequency and the instability timescale of unstable modes which is valid at small frequencies. We argue that our analysis can be directly extended to gravitational perturbations of exotic compact objects in the black-hole limit. The instability for electromagnetic and gravitational perturbations is generically stronger than in the scalar case, and it requires larger absorption to be quenched. We argue that exotic compact objects with spin chi less than or similar to 0.7 (chi less than or similar to 0.9) should have an absorption coefficient of at least 0.3% (6%) to remain linearly stable, and that an absorption coefficient of at least similar to 60% would quench the instability for any spin. We also show that-in the static limit-the scalar, electromagnetic, and gravitatonal perturbations of the Kerr metric are related to one another through Darboux transformations. Finally, correcting previous results, we give the transformations that bring the Teukolsky equation in a form described by a real potential also in the gravitational case.