You could split on words instead of tokens, but then you need a large vocabulary, you can't deal with inputs that contain a word which is not in the vocabulary, and it's not so clear what a "word" even is.

Instead of coming up with more and more heuristics to chop a sequence of bytes up in "words" in a vocabulary, we could simply set a limit on the size of the vocabulary (number of tokens), put all bytes in there (so we can at least handle any input byte by byte), and pack the remaining space with the most common multi-byte byte sequences. Then you end up with tokens like here.