Ultracompact objects are self-gravitating systems with a light ring. It was recently suggested that fluctuations in the background of these objects are extremely long lived and might turn unstable at the nonlinear level, if the object is not endowed with a horizon. If correct, this result has important consequences: objects with a light ring are black holes. In other words, the nonlinear instability of ultracompact stars would provide a strong argument in favor of the "black hole hypothesis," once electromagnetic or gravitational-wave observations confirm the existence of light rings. Here we explore in some depth the mode structure of ultracompact stars, in particular constant-density stars and gravastars. We show that the existence of very long-lived modes-localized near a second, stable null geodesic-is a generic feature of gravitational perturbations of such configurations. Already at the linear level, such modes become unstable if the object rotates sufficiently fast to develop an ergoregion. Finally, we conjecture that the long-lived modes become unstable under fragmentation via a Dyson-Chandrasekhar-Fermi mechanism at the nonlinear level. Depending on the structure of the star, it is also possible that nonlinearities lead to the formation of small black holes close to the stable light ring. Our results suggest that the mere observation of a light ring is a strong evidence for the existence of black holes.