We present a fully passive method for implementing a quantum phase gate between two photons traveling in a one-dimensional waveguide. The gate is based on chirally coupled emitters in a three level V configuration, which only interact through the photon field without any external control fields. We describe the (non)linear scattering of the emerging polariton states and show that for near resonant photons the scattering dynamics directly implements a perfect control phase gate between the incoming photons in the limit of many emitters. For a finite number of emitters we show that the dominant error mechanism can be suppressed by a simple frequency filter at the cost of a minor reduction in the success probability. We verify the results via comparison with exact scattering matrix theory and show that the fidelity can reach values F similar to 99% with a gate success probability of > 99% for as few as eight emitters.