It's certainly a surprising enough construct that it sounds incorrect, which is the metric that matters most to most people.

> "Whom in this room would you boom?" is not ungrammatical though.

Try polling some people on that.