Decoherence in nearly-isolated GaAs quantum dots is investigated using the change in average Coulomb blockade peak height upon breaking time-reversal symmetry. The normalized change in average peak height approaches the predicted universal value of 1/4 at temperatures well below the single-particle level spacing, but is greatly suppressed for temperature greater than the level spacing, suggesting that inelastic scattering or other dephasing mechanisms dominate in this regime.