There's an adjacent option to undo a vote, which will say "unvote" or "undown" depending on which way the vote went.

Designwise that's maybe not ideal, sure. But it is discoverable, consistent, and reliable, which in my view counts for more.