Scalar fields pervade theoretical physics and are a fundamental ingredient to solve the dark matter problem, to realize the Peccei-Quinn mechanism in QCD or the string-axiverse scenario. They are also a useful proxy for more complex matter interactions, such as accretion disks or matter in extreme conditions. Here, we study the collision between scalar "clouds" and rotating black holes. For the first time we are able to compare analytic estimates and strong field, nonlinear numerical calculations for this problem. As the black hole pierces through the cloud it accretes according to the Bondi-Hoyle prediction, but is deflected through a purely kinematic gravitational "anti-Magnus" effect, which we predict to be present also during the interaction of black holes with accretion disks. After the interaction is over, we find large recoil velocities in the transverse direction. The end-state of the process belongs to the vacuum Kerr family if the scalar is massless, but can be a hairy black hole when the scalar is massive.