I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
ivanjermakov•26m ago
Do you know where <||> syntax is coming from? I find it very odd that existing markup wasn't used and there is very little info online about it.
t_mahmood•19m ago
Correct me if I wrong, I saw this when llama.cpp or ollama initializes, it is used as token separator of some sort.
cowlevel•15m ago
GPT-2 used <|endoftext|>
jesdo•14m ago
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
sroussey•3m ago
Constrained decoding came to JS land transformers last month for in browser structured output as well, also used for tool calling.
jesdo•54m ago
ivanjermakov•26m ago
t_mahmood•19m ago
cowlevel•15m ago
jesdo•14m ago
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.