We explore the use of the mass spectrum of neutron stars and black holes in gravitational-wave compact binary sources as a cosmological probe. These standard siren sources provide direct measurements of luminosity distance. In addition, features in the mass distribution, such as mass gaps or peaks, will redshift and thus provide independent constraints on their redshift distribution. We argue that the entire mass spectrum should be utilized to provide cosmological constraints. For example, we find that the mass spectrum of LIGO-Virgo-KAGRA events introduces at least five independent mass "features": the upper and lower edges of the pair instability supernova (PISN) gap, the upper and lower edges of the neutron star-black hole gap, and the minimum neutron star mass. We find that although the PISN gap dominates the cosmological inference with current detectors (second generation, 2G), as shown in previous work, it is the lower mass gap that will provide the most powerful constraints in the era of Cosmic Explorer and Einstein Telescope (third generation, 3G). By using the full mass distribution, we demonstrate that degeneracies between mass evolution and cosmological evolution can be broken, unless an astrophysical conspiracy shifts all features of the full mass distribution simultaneously following the (nontrivial) Hubble diagram evolution. We find that this self-calibrating "spectral siren" method has the potential to provide precision constraints of both cosmology and the evolution of the mass distribution, with 2G achieving better than 10% precision on H(z) at z less than or similar to 1 within a year and 3G reaching less than or similar to 1% at z greater than or similar to 2 within one month.