Most tumor-targeted drug delivery systems must overcome a large variety of physiological barriers before reaching the tumor site and diffuse through the tight network of tumor cells. Many studies focus on optimizing the first part, the accumulation of drug carriers at the tumor site, ignoring the penetration efficiency, i.e., a measure of the ability of a drug delivery system to overcome tumor surface adherence and uptake. We used three-dimensional (3D) tumor spheroids in combination with light-sheet fluorescence microscopy in a head-to-head comparison of a variety of commonly used lipid-based nanoparticles, including liposomes, PEGylated liposomes, lipoplexes, and reconstituted high-density lipoproteins (rHDL). Whilst PEGylation of liposomes only had minor effects on the penetration efficiency, we show that lipoplexes are mainly associated with the periphery of tumor spheroids, possibly due to their positive surface charge, leading to fusion with the cells at the spheroid surface or aggregation. Surprisingly, the rHDL showed significantly higher penetration efficiency and high accumulation inside the spheroid. While these findings indeed could be relevant when designing novel drug delivery systems based on lipid-based nanoparticles, we stress that the used platform and the detailed image analysis are a versatile tool for in vitro studies of the penetration efficiency of nanoparticles in tumors.