The realization of a highly efficient optical spot-size converter for the end-face coupling of single photons from GaAs-based nanophotonic waveguides with embedded quantum dots is reported. The converter is realized using an inverted taper and an epoxy polymer overlay providing a 1.3 mu m output mode field diameter. The collection of single photons from a quantum dot into a lensed fiber with a rate of 5.84 +/- 0.01 MHz is demonstrated and a chip-to-fiber coupling efficiency of is approximate to 48% estimated. The stability and compatibility with cryogenic temperatures make the epoxy waveguides a promising material to realize efficient and scalable interconnects between heterogeneous quantum photonic integrated circuits.