I recently ported a program from python to rust and the original author used string regexes. Input and output document encoding mattered but the characters that needed to be matched were always lower ASCII. The python program could have used binary regexes, but instead forced an input encoding (UTF-8) and made the user choose an output encoding. When the input comes from an unknown process or legacy data, however, you don’t always get the luxury of assuming the encoding. Switching to binary regexes and ignoring encoding altogether simplified logic, eliminated classes of errors, and made the program work in scenarios it couldn’t earlier. Getting rid of the last decoding/encoding code gave me so much relief, especially when all of the whacky encoding tests I had already written continued to work.

Author went for Intl.Segmenter too: https://github.com/cheeaun/phanpy/issues/1491

- https://george.mand.is/invalid-surrogate-pairs/

I thought it was something that's easier to play with and feel than necessarily just read about.

Was already bad enough that instead of bytes, we have to worry about code points. Now even that isn’t enough?

It would have been expensive, but all characters should have been fixed size 64bit values.