The upcoming detection of gravitational waves by terrestrial interferometers will usher in the era of gravitational-wave astronomy. This will be particularly true when space-based detectors will come of age and measure the mass and spin of massive black holes with exquisite precision and up to very high redshifts, thus allowing for better understanding of the symbiotic evolution of black holes with galaxies, and for high-precision tests of General Relativity in strong-field, highly dynamical regimes. Such ambitious goals require that astrophysical environmental pollution of gravitational-wave signals be constrained to negligible levels, so that neither detection nor estimation of the source parameters are significantly affected. Here, we consider the main sources for space-based detectors the inspiral, merger and ringdown of massive black-hole binaries and extreme mass-ratio inspirals - and account for various effects on their gravitational waveforms, including electromagnetic fields, cosmological evolution, accretion disks, dark matter, "firewalls" and possible deviations from General Relativity. We discover that the black-hole quasinormal modes are sharply different in the presence of matter, but the ringdown signal observed by interferometers is typically unaffected. The effect of accretion disks and dark matter depends critically OH their geometry and density profile, but is negligible for most sources, except for few special extreme mass-ratio inspirals. Electromagnetic fields and cosmological effects are always negligible. We finally explore the implications of our findings for proposed tests of General Relativity with gravitational waves, and conclude that environmental effects will not prevent the development of precision gravitational-wave astronomy.