Fat is only the primary energy source at lower output levels. As power output increases, metabolism shifts increasingly away from fat and towards glycogen. The shape of that curve is different for everyone, but ironically the people who most need to lose fat typically have the least ability to metabolize fat (not fat adapted) and so with any exercise they're mostly running on carbs. Once the glycogen stores run low then their bodies start burning protein as well, even though it's less efficient.

Muscles will also just atrophy if not used, regardless of diet.