For example, if my intent is to keep the present value, I will send {"foo": 1} or {"foo": 1, "bar": null} as null and no value has the same meaning. On the other hand, I might want to change the existing value to empty one and send {"foo": 1, "bar": []}.
The server must understand case when I am not mutating the field and when I am mutating the field and setting it to be empty.
On the other side, I never want to be sending json out with null values as that is waste of traffic and provides no information to the client, ie {"foo": 1, "bar": null} is the same as {"foo": 1}.
Protocol buffers have the exact same problem but they tackle it in even dumber way by requiring you to list fields from the request, in the request's special field, which you are mutating as they are unable to distinguish null and no value present and will default to empty value otherwise, like {} or [], which is not the intent of the sender and causes all sort of data corruption.
PS: obviosly this applies to pointers as a whole, so if i have type Request struct { Number *int `json:"number"} then sending {} and {"number": null} must behave the same and result in Result{Number nil}
"In v1, a nil Go slice or Go map is marshaled as a JSON null. In contrast, v2 marshals a nil Go slice or Go map as an empty JSON array or JSON object, respectively. The jsonv2.FormatNilSliceAsNull and jsonv2.FormatNilMapAsNull options control this behavior difference. To explicitly specify a Go struct field to use a particular representation for nil, either the `format:emitempty` or `format:emitnull` field option can be specified. Field-specified options take precedence over caller-specified options."
https://pkg.go.dev/github.com/go-json-experiment/json#exampl...
It is good to see some partial solutions to this issue. It plagues most languages and introduces a nice little ambiguity that is just trouble waiting to happen.
Ironically, JavaScript with its hilarious `null` and `undefined` does not have this problem.
Most JSON parsers and emitters in most languages should use a special value for "JSON null".
obj['key']=undefined
was the same as
delete obj['key']No, it's an exception. It was badly designed from the start - it's not just that people's json needs (which hardly changed) outgrew it.
Over time, it became evident that the JSON package didn't meet the needs of its users, and the package has evolved as a result. The size of the evolution doesn't matter.
It's also true that a json IO built-in lib typically wouldn't be so poorly designed in the first release of a language, that it would immediately be in need of maintenance.
JSON library released with Go 1, in 2012. This makes the library 13 years old [0].
If that's immediate, I'm fine with that kind of immediate.
Obviously encoding/json doesn't "work perfectly", the TFA lists several problems it has, and the need for a new version, and that's directly by the horse's mouth. Is the Go team "trolling" as well?
Second, we're not talking whether it "does the job", which is what you might mean by "works perfectly great".
We're talking about whether it's a good design for the problem domain, or whether it has footguns, bad API choices, performance problems, and other such issues.
The latter is a weasely way to put the blame on changing needs - as if initially it was fine, but user needs grew and it's not covering them anymore. Truth is, user needs are the same, we havent had any magical change in JSON use patterns over the last 10 years. The design was just flawed to begin with.
I'd argue it didn't "become evident over time" either. It was evident on day one, and many people pointed it out 10 and 13 years ago.
The types themselves have a way to customize their own JSON conversion code. You could have a struct serialize itself to a string, an array, do weird gymnastics, whatever. The JSON module calls these custom implementations when available.
The current way of doing it is shit though. If you want to customize serialization, you need to return a json string basically. Then the serializer has to check if you actually managed to return something sane. You also have no idea if there were some JSON options. Maybe there is an indentation setting or whatever. No, you return a byte array.
Deserialization is also shit because a) again, no options. b) the parser has to send you a byte array to parse. Hey, I have this JSON string, parse it. If that JSON string is 100MB long, too bad, it has to be read completely and allocated again for you to work on because you can only accept a byte array to parse.
New API fixes these. They provide a Decoder or Encoder to you. These carry any options from top. And they also can stream data. So you can serialize your 10GB array value by value while the underlying writer writes it into disk for example. Instead of allocating all on memory first, as the older API forces you to.
There are other improvements too but the post mainly focuses on these so thats what I got from it (I havent tried the new api btw, this is all from the post so maybe I’m wrong on some points)
Now here’s a new implantation that addresses some of the architectural problems that made the old library structurally problematic for some use cases (streaming large JSON docs being the main one).
The largest problem were around behavior around nil in golang and what to convert into json and vice versa.
* The v2 will now throw an error for invalid characters outside of ut8 (before silently accepted it) which meant one had to preprocess or process again the json before sending it off to the server * the golang nil will be converted to json empty array or map (for each type). previously it was converted to json null. * json field names will be converted to golang names with case sensitivity. before it was case-insentitive and would be lowercased. this kinda caused lots of problems if the field collided. (say there's bankName and bankname in json) * omitempty was problematic as it was used for say golang amount: nil would mean omit the field in json as {} instead of { amount: null}. however it also meant that the golang amount: 0 would also be omitted as { amount: 0 } which surprising. the new omitempty will only do so for nil and empty arrays/hashmaps but no longer for 0 or false. there's a new omitzero tag for that.
How did that make it into the v1 design?
var m map[string]int
println(m == nil)Also, now that nil map is an empty object, shouldn't that extend to every nil struct that doesn't have a custom marshaller? It would be an object if it wasn't nil after all...
And then people complain that Rust doesn't have a batteries-included stdlib. It is done to avoid cases like this.
Yes, we should definitely go with the Rust approach instead.
Anyway, I'd better get back to figuring out which crate am I meant to be using...
Both v1 packages continue work; both are maintained. They get security updates, and were both improved by implementing them on top of v2 to the extent possible without breaking their respective APIs.
More importantly: the Go authors remain responsible for both the v1 and v2 packages.
What most people want to avoid with a "batteries included standard library" (and few additional dependencies) is the debacle we had just today with NPM.
Well maintained packages, from a handful of reputable sources, with predictable release schedules, a responsive security team and well specified security process.
You can't get that with 100s of independently developed dependencies.
That does look a lot cleaner. I was just grumbling about this in golang yesterday (yaml, not json, but effectively the same problem).
analytically•6h ago
https://github.com/centralci/go-benchmarks/tree/b647c45272c7...
tgv•6h ago
gethly•5h ago
So it seems both are operating at the edge of Go's capabilities.
Personally, I think JSON should be in Go's core and highly optimised simd c code and not in the Go's std library as standard Go code. As JSON is such an important part of the web nowadays, it deserves to be treated with more care.
tptacek•5h ago
ronsor•5h ago
dilyevsky•4h ago
Edit: See: https://go.dev/wiki/AssemblyPolicy
nasretdinov•4h ago
I think when it's introduced it might be worth discussing that again. Otherwise providing assembly for JSON of all packages seems like a huge maintenance burden for very little benefit for end users (since faster alternatives are readily available)
CamouflagedKiwi•4h ago
Sonic may be different but I'm feeling once bitten twice shy on "faster" JSON parsers at this point. A highly optimised SIMD version might be nice but the stdlib json package needs to work for everything out there, not just the cases the author decided to test on, and I'd be a lot more nervous about something like that being sufficiently well tested given the extra complexity.
godisdad•3h ago
There is a case to be made here but Corba, SOAP and XML-RPC likely looked similarly sticky and eternal in the past
ForHackernews•1h ago
If you're pushing data around on disk where the serialization library is your bottleneck, pick a better format.
kiitos•2h ago
second of all, sonic apparently uses unsafe to (unsafe-ly) cast byte slices to strings, which of course is gonna be faster than doing things correctly, but is also of course incomparable to doing things correctly
like almost all benchmark data posted to hn -- unsound, ignore
kristianp•33m ago
Thaxll•1h ago
https://github.com/bytedance/sonic/issues/785