Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

No, in both tokenizers Unicode tag-block code points like these are converted into bytes (two tokens per character), which is a fallback for code points uncommon enough to not warrant a dedicated token.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: