Andreev bound states with opposite phase-inversion asymmetries are observed in local tunneling spectra at the two ends of a superconductor-semiconductor-superconductor planar Josephson junction in the presence of a perpendicular magnetic field, while the nonlocal spectra remain phase symmetric. Spectral signatures agree with a theoretical model, yielding a physical picture in which phase textures in superconducting leads localize and control the position of Andreev bound states in the junction, demonstrating a simple means of controlling the position and size of Andreev states within a planar junction.