Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

I basically forked the Go scanner/parser, and changed the syntax to have functions return a single value. This enable proper Result/Option type to be used as well as propagating errors with an operator.

I also wanted to have short "type inferred" anonymous functions to be able to use functions like Map/Reduce/Filter, without having to use 100 characters to specify the types.

Comments

mayli•2d ago

That's pretty cool, I ways want Result/Option/Err types in python.

adammarples•2d ago

There is a lib for that I believe

18172828286177•2d ago

Have you seen https://vlang.io/?

https://docs.vlang.io/type-declarations.html#optionresult-ty... in particular

alain_gilbert•2d ago

Yes, I got my inspiration mainly from swift and a little bit from vlang as well.

eg: the @LINE "Compile time pseudo variable"

https://github.com/alaingilbert/agl/blob/8b656a385207e57fd0f...

https://docs.vlang.io/conditional-compilation.html#compile-t...

ben0x539•2d ago

Cool language. Only two weeks for this? Damn!

This seems to nicely smooth out a lot of the pain points I had back when I regularly wrote Go but also switched to other more expression-based languages occasionally. Being able to get all these conveniences while still being able to call into existing Go code seems amazing!

Also, I appreciate the pun, but did you have to use AI for the logo? Haha

alain_gilbert•2d ago

I did use AI for the logo! Using Go libraries is still kinda rough, the standard library works fine, but for other libraries you'd need to (re) define the types manually. I'm trying to make something that would generate it all automatically, but there is some problems that I don't know how I would solve.

For example, I'd like to have the pointers being automatically wrapped in a Option[T], but then if you have something like a linked list, with a struct containing pointers to other nodes, it gets complicated.

LudwigNagasena•2d ago

Wow, looks like a great QoL improvement. I have a few questions though.

Propagation of optional values is an unconventional design decision. Why have you chose it over short circuiting as in C#, JS, Swift?

What’s your opinion on error wrappers and stack trace providers that are often used in go projects complicating simple error propagation? Have you consciously avoided adding similar functionality or just haven’t thought about it yet?

alain_gilbert•2d ago

I haven't think much about it, a lot of things are probably going to change. I tried to get something working as fast as I possibly could, and in two weeks, lots of corners were cut to make it happen.

Thanks for the feedback, it's good to know when something does not make sense ^^

stevedonovan•1d ago

This is how Option works with ? in Rust. But short-circuiting in an expression does make sense

lordofgibbons•2d ago

This is so awesome! I tried building literally this year or so ago after being inspired by Borgo, but my kung fu wasn't as strong as yours.

Are nil pointers still possible/allowed in AGL? After a lack of enums, this is my biggest pet peeve.

alain_gilbert•2d ago

I was also quite inspired by borgo. But unlike borgo, yes, at this time nil is part of the language, which allows you to use other libraries without the need to make laborious wrappers. But if I can manage to create a script to automatically make the wrappers, I'd love to remove the nil keyword entirely.

lordofgibbons•2d ago

There's also the question of how to handle pointer types which get initialized as nil. Specially prevalent in places where you have to deserialize data.