We study the link between galaxies and Hi-selected absorption systems at z similar to 3-4 in the MUSE Analysis of Gas around Galaxies (MAGG) survey, an ESO large programme consisting of integral field spectroscopic observations of 28 quasar fields hosting 61 strong absorbers with N-HI greater than or similar to 10(16.5) cm(-2). We identify 127 Ly alpha emitting galaxies (LAEs) around the absorbers, corresponding to a detection rate of 82 +/- 16 percent. The luminosity function of these LAEs is approximate to 5 times higher in normalization than the field population and we detect a significant clustering of galaxies with respect to the gas, confirming that high column density absorbers and LAEs trace each other. Between 30 and 40 percent of the absorbers are associated with multiple LAEs, which lie preferentially along filaments. Galaxies in groups also exhibit a three times higher covering factor of optically thick gas compared to isolated systems. No significant correlations are identified between the emission properties of LAEs and the absorption properties of optically thick gas clouds, except for a weak preference of brighter and multiple galaxies to reside near broad absorbers. Based on the measured impact parameters and the covering factor, we conclude that the near totality of optically thick gas in the Universe can be found in the outer circumgalactic medium (CGM) of LAEs or in the intergalactic medium (IGM) in proximity to these galaxies. Thus, LAEs act as tracers of larger scale structures within which both galaxies and optically thick clouds are embedded. The patchy and inhomogeneous nature of the CGM and IGM explains the lack of correlations between absorption and emission properties. This implies that very large samples are needed to unveil the trends that encode the properties of the baryon cycle.