I've rooted my Android phone and I've been using LBE Privacy Guard that basically lets you do just what you describe.

I actually just a got a new phone (Nexus 5), and the app is crashing on open right now, I imagine because I'm using KitKat or something like that. But that's besides the point.