While the ABJ Chern-Simons-matter theory and its string theory dual manifestly lack parity invariance, no sign of parity violation has so far been observed on the weak coupling spin chain side. In particular, the planar two-loop dilatation generator of ABJ theory is parity invariant. In this letter we derive the non-planar part of the two-loop dilatation generator of ABJ theory in its SU(2)xSU(2) sub-sector. Applying the dilatation generator to short operators, we explicitly demonstrate that, for operators carrying excitations on both spin chains, the non-planar part breaks parity invariance. For operators with only one type of excitation, however, parity remains conserved at the non-planar level. We furthermore observe that, as for ABJM theory, the degeneracy between planar parity pairs is lifted when non-planar corrections are taken into account.