> If it's an empirical fact, how much objection from how many (and which) people is enough to cross the line into controversial?

Any. Controversial is a continuous-valued, not binary, attribute.

How controversial is enough to justify a particular reaction? That's a political judgement, and in practice has as much to do with where you stand on the controversy as how much controversy there is.