I had the pleasure of interviewing (and hiring) my first two team members just this past spring. To me, "culture fit" basically meant that the interview consisted mostly of us nerding the fuck out together. That really had nothing to do with sex, race, or any other prejudices.

(FWIW, those two guys have turned out to be truly awesome developers).