The tidal deformability of compact objects by a companion has a detectable imprint in the gravitational waves emitted by a binary system. This effect is governed by the so-called tidal Love numbers. For a particular theory of gravity, these depend solely on the object internal structure and they vanish for black holes in general relativity. A measurement compatible with nonzero tidal Love numbers could thus provide evidence of new physics in the strong-field regime. However, in realistic astrophysical scenarios, compact objects are surrounded by a nonvacuum environment. In this work, we study the tidal deformability of configurations of black holes immersed in matter, focusing on two analytical models representing an anisotropic fluid and a thin-shell of matter around a black hole. We then apply our results to the astrophysically relevant case of a black hole surrounded by an accretion disk, in the parameter region of interest of the upcoming LISA mission. Our results indicate that there are challenges to overcome concerning tests of strong-field gravity using tidal Love numbers.