We describe the determinants of energy intensity, carbon intensity, and CO2 emissions in the German manufacturing sector between 1995 and 2007, applying the LMDI index decomposition technique not to aggregate but to micro data. We trace back changes in total CO2 emissions from manufacturing to changes in activity level, structural change between sectors, structural change within sectors, energy intensity at the firm level, fuel mix, and emission factors. We use a firm data set on energy use from the AFiD-Panel on German manufacturing plants that allows us to analyze energy use at the firm level with unprecedented accuracy. Our results show that heterogeneity among firms within one sector is a driver of energy intensity, carbon intensity, and CO2 emissions. By stressing the importance of competition between firms for energy efficiency improvements, we highlight a factor that has so far been widely ignored. Firm heterogeneity has so far rarely included in index decomposition analyses. Contrary to wide-spread beliefs, energy intensity improvements at the firm level do not play a significant role in reducing emissions. Based on findings from the decomposition analysis, we use sector-level results on the relative importance of improvements in firm-level energy intensity and intra-sectoral structural change to distinguish two different innovation channels: innovation by technology and by entrants. We show that incumbent firms in a number of sectors, including some of the most energy intensive ones, do not significantly improve their energy efficiency. Innovation takes place via new entrants instead, rendering standard policies targeted at firm-level energy efficiency ineffective.