That's a good idea.
The limitation being that your service can redirect go get to a different repository url, but go will still fetch that repository and check out what it thinks is the best branch for that version of go: one with the name of the current go version, or master. So you'd have to play some tricks to keep a full clone-able repo with master locked at some commit for each version of each library, rather than just one repo per library with tags. I guess there are some ways to do that where the clones share the objects so you aren't wasting space, though so not insurmountable.
Space usage doesn't worry me too much. If a "package" is just a set of files, then storing them in some sort of compressed form (.tar.gz) and simply un-tarring them and making the API mimic a git repo with one revision should be fairly straightforward.