What do HN people have to say about Unicode and UTF-{8,16,32}? Are there parts you've never really understood? Have you had unexpected bugs due to misunderstood properties of text?
What do HN people have to say about Unicode and UTF-{8,16,32}? Are there parts you've never really understood? Have you had unexpected bugs due to misunderstood properties of text?
> *2.2.3 Characters, Not Glyphs*
> The Unicode Standard draws a distinction between characters and glyphs. Characters are the abstract representations of the smallest components of written language that have semantic value. They represent primarily, but not exclusively, the letters, punctuation, and other signs that constitute natural language text and technical notation. [...] Letters in different scripts, even when they correspond either semantically or graphically, are represented in Unicode by distinct characters.
> Characters are represented by code points that reside only in a memory representation, as strings in memory, on disk, or in data transmission. The Unicode Standard deals only with character codes.
> *2.4 Code Points and Characters*
> The range of integers used to code the abstract characters is called the codespace. A particular integer in this set is called a code point. When an abstract character is mapped or assigned to a particular code point in the codespace, it is then referred to as an encoded character.
> *2.5 Encoding Forms*
This deals with UTF-{8,16,32}, which is a tricky bit and tripped me up for a long time. If the document is too dense here, there's a lot of supplementary material online explaining the different forms, I'll link a Tom Scott video explaining UTF-8.
---
The long and short of it is: the atomic unit of Unicode is the character, or encoded character, which is a value that has been associated with a code point, which is an integer usually represented in hex for as U+XXXX. Unicode doesn't deal with glyphs or graphical representations, just characters and their properties (eg. what is the character name? what should this character do when uppercased?). As you probably know, many characters can combine with others to form grapheme clusters, which may look like a single (abstract) character, but underneath consist of multiple (encoded) characters. Every character is associated with an integer index (a codepoint), and those integers can be represented in three formats (this sort of happened by accident): UTF-32 (just represent the integer directly), UTF-16 (was originally supposed to represent the integer directly, but there were too many and it got extended), and UTF-8 (which has different byte lengths to encode different characters efficiently).
[spec] https://www.unicode.org/versions/Unicode16.0.0/core-spec/
[2.2.3] https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha...
[2.4] https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha...
[2.5] https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha...
[Tom Scott UTF-8] https://www.youtube.com/watch?v=MijmeoH9LT4
The normalization forms are explained, in order of approachability (imo), in this random Youtube video, the Unicode Annex #15, and the Unicode Core Spec:
https://www.youtube.com/watch?v=ttLD4DiMpiQ
https://unicode.org/reports/tr15/
https://www.unicode.org/versions/Unicode16.0.0/core-spec/cha...
Å (ANGSTROM SIGN)
Å (LATIN CAPITAL LETTER A WITH RING ABOVE)
Å (LATIN CAPITAL LETTER A) + (◌̊ COMBINING RING ABOVE)
А̊ (CYRILLIC CAPITAL LETTER A) + (◌̊ COMBINING RING ABOVE)
Of these, the Angstrom Sign is considered deprecated and won't show up in any normal forms. The second is the NFC (composed) form, and the third is the NFD (decomposed) form. The Cyrillic one looks the same, but is not the same abstract character, so isn't connected in any normalization form.
Normal forms also reorder the diacritics if there are multiple. The strings could be compared through their normalized encoded forms (like UTF-8), which I think is what you meant, or their normalized code points directly. I agree it can be messy, but I'm curious what you meant by dubious, do you think there's a better way?
Then came emojis, and now the Unicode Consortium's efforts for Unicode version updates seems to be about adding more different kinds of poop emojis and shades of skin colors. Well, maybe it projects accurately the language and culture of this modern time.
UTF-8 is great because it is a superset of ASCII, but because its byte-width varies, it has more complexity for decoding/encoding it (similar to constant/variable width ISA's in CPUs).
Different languages have different concepts, e.g. text direction==flow (left/right, up/down, characters/logograms, different kind of visual cues etc.). Humans create problems when they want to combine different languages at the same time. E.g. mathematical notation is in my opinion 2D graphics, and it cannot be (usually/always) inlined with text glyphs (to be aesthetically pleasing). Same kind of problems may come when trying to inline e.g. languages with different flow directions. Its like trying to combine native GUI widgets in Win32 and Cocoa/SwiftUI and GTK/Qt/WXwidgets - the (visual) languages doesn't have the same concepts or they are conflicting.
> The Unicode Consortium will no longer accept proposals for flags. Flags that correspond to officially assigned ISO 3166-1 alpha-2 region codes are automatically added, with no proposals necessary.
And they decided against adding Multi-skintoned Families to the RGI, as in, vendors can encode them if they really want to, but it's not recommended. Apple for example replaced their more complex family emoji with the recommended silhouettes afterwards [4].
1. https://blog.unicode.org/2022/03/the-past-and-future-of-flag...
2. https://unicode.org/emoji/proposals.html#Flags
3. https://www.unicode.org/L2/L2020/20114-family-emoji-explor.p...
To say nothing that various emojis have been part of Unicode since pretty much the start, and was part of other encoding schemes as well (notably in Japan, but also e.g. the the "Outlook J").
And if you actually look at the changes in Unicode versions, you'll see there are tons of language-related changes in every one. To say Unicode updates are just about emoji updates is just silly. The reason you don't notice is because this is mostly for small language, obscure features in larger languages, and historical languages, and things like that.
Rendello•7mo ago
Turns out, sometimes changing case changes not only the number of bytes (in UTF-8), but the number of encoded characters! This led to my post "UTF-8 characters that behave oddly when the case is changed" [1], which inspired a lot of conversation that taught me a lot. After that, I started reading Unicode documentation in earnest, and building up an idea of what a new tool should show. I'm trying to make clear things I didn't (and sometimes still don't) understand, so I'd love to know what causes pains in the wild / gaps in people's understanding.
1. https://news.ycombinator.com/item?id=42014045