We estimate the potential of present and future interferometric gravitational-wave detectors to test the Kerr nature of black holes through "gravitational spectroscopy," i.e., the measurement of multiple quasinormal mode frequencies from the remnant of a black hole merger. Using population synthesis models of the formation and evolution of stellar-mass black hole binaries, we find that Voyager-class interferometers will be necessary to perform these tests. Gravitational spectroscopy in the local Universe may become routine with the Einstein Telescope, but a 40-km facility like Cosmic Explorer is necessary to go beyond z similar to 3. In contrast, detectors like eLISA (evolved Laser Interferometer Space Antenna) should carry out a few-or even hundreds-of these tests every year, depending on uncertainties in massive black hole formation models. Many space-based spectroscopical measurements will occur at high redshift, testing the strong gravity dynamics of Kerr black holes in domains where cosmological corrections to general relativity (if they occur in nature) must be significant.