Tensor network states, and in particular projected entangled pair states (PEPS), suggest an innovative approach for the study of lattice gauge theories, both from a pure theoretic point of view, and as a tool for the analysis of the recent proposals for quantum simulations of lattice gauge theories. In this paper we present a framework for describing locally gauge invariant states on lattices using PEPS. The PEPS constructed hereby shall include both bosonic and fermionic states, suitable for all combinations of matter and gauge fields in lattice gauge theories defined by either finite or compact Lie groups.