Black holes are unique among astrophysical sources: they are the simplest macroscopic objects in the Universe, and they are extraordinary in terms of their ability to convert energy into electromagnetic and gravitational radiation. Our capacity to probe their nature is limited by the sensitivity of our detectors. The LIGO/Virgo interferometers are the gravitational-wave equivalent of Galileo's telescope. The first few detections represent the beginning of a long journey of exploration. At the current pace of technological progress, it is reasonable to expect that the gravitational-wave detectors available in the 2035-2050s will be formidable tools to explore these fascinating objects in the cosmos, and space-based detectors with peak sensitivities in the mHz band represent one class of such tools. These detectors have a staggering discovery potential, and they will address fundamental open questions in physics and astronomy. Are astrophysical black holes adequately described by general relativity? Do we have empirical evidence for event horizons? Can black holes provide a glimpse into quantum gravity, or reveal a classical breakdown of Einstein's gravity? How and when did black holes form, and how do they grow? Are there new long-range interactions or fields in our Universe, potentially related to dark matter and dark energy or a more fundamental description of gravitation? Precision tests of black hole spacetimes with mHz-band gravitational-wave detectors will probe general relativity and fundamental physics in previously inaccessible regimes, and allow us to address some of these fundamental issues in our current understanding of nature.