Quantum asymmetry is a physical resource that coincides with the amount of coherence between the eigenspaces of a generator responsible for phase encoding in interferometric experiments. We highlight an apparently counterintuitive behavior that the asymmetry may increase as a result of a decrease of coherence inside a degenerate subspace. We intuitively explain and illustrate the phenomena by performing a three-mode single-photon interferometric experiment, where one arm carries the signal and two noisy reference arms have fluctuating phases. We show that the source of the observed sensitivity improvement is the reduction of correlations between these fluctuations and comment on the impact of the effect when moving from the single-photon quantum level to the classical regime. Finally, we also establish the analogy of the effect in the case of entanglement resource theory.