We present a new optomechanical cavity design, comprising a fiber mirror and a mirror featuring a silicon phononic crystal structure on its backside, termed the "exoskeleton". The fiber mirror's small mass pushes the frequencies of its vibrational modes above that of the mode of interest, while the exoskeleton structure creates a bandgap in the spectrum of the latter mirror, engineered to be centered around the mode of interest. The mechanical mode of interest is a vibrational mode of a "soft-clamped"SiN membrane resonator, exhibiting quality factors well in excess of 100 million at room temperature. Owing to its high quantum cooperativity and low mirror noise, we are able to use the optomechanical system to make an efficient and strong optical measurement of the motion of this mode and feedback-cool it to below 20 phonons at room temperature.