The study of phases of matter has consistently fascinated and puzzled the condensed matter community, and the discovery of high-temperature superconductors in 1986, and topological phases in 1980, are no exceptions. In particular the iron-base superconductors, one of the later additions to the ever-growing class of high-temperature superconductors, have in recent years attracted a considerable amount of attention, due to their intricate phase diagrams with a multitude of electronic phases in the vicinity of the superconducting state. Such phase diagrams call for an extensive investigation of the surrounding electronic phases, in the hope of elucidating their role in the resulting superconducting instability, and to broaden our understanding of correlated systems in general. A broad range of methods and theories have successfully been applied in the quest of understanding these materials.
Nonetheless, the iron-chalcogenide FeSe appear to be the misfit in the family of iron-based superconductors, due to its lacking long-range magnetic order at ambient pressure, its highly anisotropic Fermi surface in the orthorhombic phase and its peculiar resulting gap structure.
Furthermore, recent theoretical and experimental findings suggest that FeSe also exhibits nontrivial topological phases. More specifically, FeSe doped with Te is argued to be an intrinsic manifestation of the celebrated Fu-Kane proposal, which is believed to harbor the coveted Majorana zero modes in the core of its superconducting vortices. The study of topological phases, and in particular
the topological superconducting phases, has in recent years become one of the most active fields in condensed matter theory, and the indication that FeSe could be the rare case of an intrinsic topological superconductor, opens the door to investigate topological phases in similar materials. In fact, prime candidates are magnetic materials, that either coexist or are in proximity to a superconductor,
since these can, under the right circumstances, demonstrate all the necessary ingredients to enter a topological superconducting phase.
In the first part of this thesis, we study in great detail some general symmetry tools and properties of multi-orbital systems. By keeping this discussion general, we can apply the tools on specific materials, like FeSe, but also on a broader class of systems, such as magnetic materials coexisting or in proximity to a superconductor. Motivated by the peculiar properties of FeSe, previous studies have found that nearest-neighbor Coulomb interactions in monolayer FeSe drive an intra-orbital nematic order with a d-wave form factor. With this in mind, and recent experiments on disorder-induced nematic order in iron-based superconductors, we perform a phenomenological, followed by a microscopic,
study of impurities in systems close to a nematic instability. We find that a single impurity in the tetragonal phase locally induces nematic order, however, due to the specific featureless form of the single impurity, the induced neamtic order spatially averages to zero. On the other hand, under the right conditions a critical density of impurities can modify the Stoner criterion, and thereby raise
the nematic transition temperature. Inspired by the idea of nematic order generated from nearest-neighbor interactions, we perform a thorough symmetry and mean-field study of such interactions, and find, in contrast to previous studies, the leading low temperature neamtic components to be of
inter-orbital d_xz −d_xy character. This specific form of the order parameter give rise to a hybridization gap, which can completely remove the Y pocket from the Fermi level in the 1-Fe unit cell, believed to be consistent with recent experiments. From this highly anisotropic band structure, we calculate the resulting superconducting gap within the theory of spin fluctuation mediated pairing, which also appears to agree with experiments. Lastly, we discuss the discrepancies found between our computed spin susceptibilities and the available neutron scattering experiments, as well as the necessary inclusion
of orbital-dependent quasiparticle weights.
In the last part of this thesis, we proceed and study topological superconducting phases induced by magnetic textures in multi-orbital systems. Specifically, we demonstrate an alternative route to engineering Majorana zero modes, which are trapped in singular vortex defects in magnetic textures. The discussed magnetic textures are assumed to coexist or be in proximity to a nodal superconductor.
Although this study is motivated by the iron-based superconductors, which can potentially have magnetism coexisting with superconductivity, we keep our discussion general, and it can thus be applied to a broad variety of hybrid structures and superconductors. Lastly, we perform a classification of topological superconductors induced by magnetic textures in the absence of singular defects. By taking into account the symmetries tied to the magnetic textures, and allow for general multiband spin-singlet superconductivity, we find a plethora of topological phases leading to flat, uni- or bidirectional, (quasi-)helical and chiral Majorana edge modes.