We develop the first model for extreme mass-ratio inspirals (EMRIs) with misaligned angular momentum and primary spin, and zero eccentricity -- also known as quasi-spherical inspirals -- evolving under the influence of the first-order in mass ratio gravitational self-force. The forcing terms are provided by an efficient spectral interpolation of the first-order gravitational self-force in the outgoing radiation gauge. In order to speed up the calculation of the inspiral we apply a near-identity (averaging) transformation to eliminate all dependence of the orbital phases from the equations of motion while maintaining all secular effects of the first-order gravitational self-force at post-adiabatic order. The resulting solutions are defined with respect to `Mino time' so we perform a second averaging transformation so the inspiral is parametrized in terms of Boyer-Lindquist time, which is more convenient of LISA data analysis. We also perform a similar analysis using the two-timescale expansion and find that using either approach yields self-forced inspirals that can be evolved to sub-radian accuracy in less than a second. The dominant contribution to the inspiral phase comes from the adiabatic contributions and so we further refine our self-force model using information from gravitational wave flux calculations. The significant dephasing we observe between the lower and higher accuracy models highlights the importance of accurately capturing adiabatic contributions to the phase evolution.