Using the UltraVISTA catalogs, we investigate the evolution in the 11.4~Gyr since $z=3$ of the progenitors of local ultra-massive galaxies ($\log{(M_{\rm star}/M_{\odot})}\approx11.8$; UMGs), providing a complete and consistent picture of how the most massive galaxies at $z=0$ have assembled. By selecting the progenitors with a semi-empirical approach using abundance matching, we infer a growth in stellar mass of 0.56$^{+0.35}_{-0.25}$ dex, 0.45$^{+0.16}_{-0.20}$~dex, and 0.27$^{+0.08}_{-0.12}$ dex from $z=3$, $z=2$, and $z=1$, respectively, to $z=0$. At $z 1$, the contribution from star-forming galaxies progressively increases, with the progenitors at $2z 1$, whereas the remaining was assembled via merging from $z\sim 1$ to the present. Most of the quenching of the star-forming progenitors happened between $z=2.75$ and $z=1.25$, in good agreement with the typical formation redshift and scatter in age of $z=0$ UMGs as derived from their fossil records. The progenitors of local UMGs, including the star-forming ones, never lived on the blue cloud since $z=3$. We propose an alternative path for the formation of local UMGs that refines previously proposed pictures and that is fully consistent with our findings.