We systematically explore the impact of transparency and punishment on cooperation in the provision of public goods. Motivated by problems of embezzlement, we study variations of a public goods game where one player (the official) may embezzle from an existing public good, while others (citizens) can only contribute. We show that transparency induces increased embezzlement in the absence of a punishment mechanism. The qualitative impact of transparency on contributions to the public good is reversed when a punishment mechanism is introduced. We identify stigmatization of the official when actions are not transparent. Only a combination of transparency of actions and peer-punishment options creates full accountability and increases contributions by all players.