Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The thing is, the BOM is metadata, it doesn't belong in content. It violates the contract of .txt files, which is: the entire file is a single string of content.

Recognizing it at the edges of your program and stripping it out is not the end of the world, but it's annoying and no other (8-bit) encoding works that way. In fact, I find it hard to believe UTF-8 BOMs in MS programs were anything more than a programmer error. Once such files were out in the wild, everyone else had to deal with them.



There are already plenty of cases that valid UTF-8 readers have to deal with (unused ranges of code points, invalid byte combinations, etc.). Ignoring a BOM is trivial by comparison. A UTF-8 reader honestly doesn't care about the "stringness" of a .txt file because of all the other crap that can be in a byte stream.

Older programs do care, but as I've said elsewhere in the thread an ASCII file can remain ASCII (no BOM). There's no reason to BOM-ify an old ASCII file if it really is ASCII and only ASCII-expecting programs will ever use it.

Over time these old programs will either be upgraded or go away and it will finally be safe to say that inputs must be UTF-8. At that time, the BOM has no reason to exist.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: