Eh, I'd say you aren't being fair, no. Not that any interview process is really fair. I once worked with a guy who told me stories about waking up at work at his desk, not remembering how he got there. He never did anything remotely like that working with me. For 3 years he was sharp, punctual, and competent.

And frankly, the reason and manner in which someone left their last job is a piss-poor reason for not hiring someone. What if there was sexual harassment? Stuff like that you can't know from a google search or even an interview.