Utilizing optical and near-infrared broad-band photometry covering >5deg(2) in two of the most well-studied extragalactic legacy fields (COSMOS and XMM-LSS), we measure the galaxy stellar mass function (GSMF) between 0.1 < z < 2.0. We explore in detail the effect of two source extraction methods (SExtractor and ProFound) in addition to the inclusion/exclusion of Spitzer IRAC 3.6 and 4.5 mu m photometry when measuring the GSMF. We find that including IRAC data reduces the number of massive (log(10)(M/M-circle dot) > 11.25) galaxies found due to improved photometric redshift accuracy, but has little effect on the more numerous lower-mass galaxies. We fit the resultant GSMFs with double Schechter functions down to log(10)(M/M-circle dot) = 7.75 (9.75) at z = 0.1 (2.0) and find that the choice of source extraction software has no significant effect on the derived best-fitting parameters. However, the choice of methodology used to correct for the Eddington bias has a larger impact on the high-mass end of the GSMF, which can partly explain the spread in derived M* values from previous studies. Using an empirical correction to model the intrinsic GSMF, we find evidence for an evolving characteristic stellar mass with delta log(10)(M*/M-circle dot)/delta z = , when using SExtractor (ProFound). We argue that with widely quenched star formation rates in massive galaxies at low redshift (z < 0.5), additional growth via mergers is required in order to sustain such an evolution to a higher characteristic mass.