We investigate how the orbital magnetic moments of electron and hole states in a carbon nanotube quantum dot depend on the number of carriers on the dot. Low temperature transport measurements are carried out in a setup where the device can be rotated in an applied magnetic field, thus enabling accurate alignment with the nanotube axis. The field dependence of the level structure is measured by excited state spectroscopy and excellent correspondence with a single-particle calculation is found. In agreement with band structure calculations we find a decrease of the orbital magnetic moment with increasing electron or hole occupation of the dot, with a scale given by the band gap of the nanotube.