We explore constraints on the equation of state (EOS) of neutron-rich matter based on microscopic calculations up to nuclear densities and observations of neutron stars. In a previous work we showed that predictions based on modern nuclear interactions derived within chiral effective field theory and the observation of two-solar-mass neutron stars result in a robust uncertainty range for neutron star radii and the EOS over a wide range of densities. In this work we extend this study, employing both the piecewise polytrope extension from Hebeler et al. as well as the speed of sound model of Greif et al., and show that moment of inertia measurements of neutron stars can significantly improve the constraints on the EOS and neutron star radii.