Recent work applying the notion of pseudospectrum to gravitational physics showed that the quasinormal mode spectrum of black holes is unstable, with the possible exception of the longest-lived (fundamental) mode. The fundamental mode dominates the expected signal in gravitational wave astronomy, and there is no reason why it should have privileged status. We compute the quasinormal mode spectrum of two model problems where the Schwarzschild potential is perturbed by a small "bump" consisting of either a Poschl-Teller potential or a Gaussian, and we show that the fundamental mode is destabilized under generic perturbations. We present phase diagrams and study a simple double-barrier toy problem to clarify the conditions under which the spectral instability occurs.