I'm really surprised that so many employers use the middle man to find contractors. How do you tap into that market and skip the middle man? Seems like a win-win to me. (Except for the recruiter who's left out in the cold)
It's a critical question you're asking and one I'm actively exploring. It's a huge, huge pay difference if you eliminate these middle men. I think networking plays a massive role in it. I'm not a famous speech giver at conferences nor do I have an online presence past a simple tech blog I don't have time to update. However, it seems people sometimes want to work with me again and pay me for it. I'm learning now the value of this, because there aren't enough yet of these connections. I also lost contact with many people over many years as an FTE.
Conspiracy theory: Follow the money and you will usually find that the staffing firm or consultancy has a "good old boys" relationship with the employer. I've seen it personally, but I don't buy in to the idea that it's pervasive or even occuring more than 20% of the time.
Of course, in the world of sales, those relationships can form quickly. Even if you've never heard of me and my staffing firm, I can present you tickets to a concert or golf tournament, or something similar, and win (buy) favor fairly easily.
Most of these middleman staffing firms are not really differentiated from one another. They find a client and leech on to it.