We investigate experimentally the effects of light assisted collisions on the coherence between momentum states in Bose-Einstein condensates. The onset of superradiant Rayleigh scattering serves as a sensitive monitor for matter-wave coherence. A subtle interplay of binary and collective effects leads to a profound asymmetry between the two sides of the atomic resonance and provides far bigger coherence loss rates for a condensate bathed in blue detuned light than previously estimated. We present a simplified quantitative model containing the essential physics to explain our experimental data and point at a new experimental route to study strongly coupled light matter systems