EDIT: One library I found is https://github.com/thetarnav/streaming-markdown which seems to combine incremental parsing with optimistic rendering, which works good enough in practice, I guess.
However, the second thing (not mentioned in the post) is that we are not rendering the markdown to HTML on the server, so []() markdown is sent to the client as []() markdown, not converted into <a href=...>. So even if a []() type link exists in a code block, that text will still be sent to the client as []() text, only sent in a single chunk and perhaps with the link URL replaced. The client has its own library to render the markdown to HTML in React.
Also, the answers are typically short so even if OpenAI outputs some malformed markdown links, worst case is that we end up buffering more than we need to and the user experiences a pause after which the entire response is visible at once (the last step is to flush any buffered text to the client).
Almost every model has a slight but meaningfully different opinion on what markdown is and how creative they can be with it.
Doing it well is a non-trivial problem.
(This comment when subjected to this processing could look like: "Could this result in edge cases with ")
There are some easy wins that could improve this further: line endings within links are generally not valid markdown, so if the code ever sees \n then just flush buffered text to the client and reset the state to TEXT.
sim7c00•8mo ago
it formats its own stuff with markdown, so if i ask it for markdown but dont explicitly specify a downloadable file, it will produce valid markdown up to where it conflicts with its own markdown, and then it gets choppy and chunked.
its an issue of my prompting is what im sure some customer service rep would be told to tell me :p because theres money to be made in courses for prompting skills perhaps, idk. (cynical view).
sure is enjoyable to struggle together with the AI to format its responses correctly :'D
porridgeraisin•8mo ago
monkeycantype•8mo ago