The main advantage of ASN.1 (specifically DER) in an HTTPS/PKI context is that it's a canonical encoding. To my understanding Protobuf isn't; I don't know about Thrift.
(A lot of hay is made about ASN.1 being bad, but it's really BER and other non-DER encodings of ASN.1 that make things painful. If you only read and write DER and limit yourself to the set of rules that occur in e.g. the Internet PKI RFCs, it's a relatively tractable and normal looking serialization format.)
However, to have a sane interface for actually working with the data you do need a schema that can be compiled to a language specific notation.
You can parse DER perfectly well without a schema, it's a self-describing format. ASN.1 definitions give you shape enforcement, but any valid DER stream can be turned into an internal representation even if you don't know the intended structure ahead of time.
rust-asn1[1] is a nice demonstration of this: you can deserialize into a structure if you know your structure AOT, or you can deserialize into the equivalent of a "value" wrapper that enumerates/enforces all valid encodings.
> which is that this ends up being complex enough that basically every attempt to do so is full of memory safety issues.
Sort of -- DER gets a bad rap for two reasons:
1. OpenSSL had (has?) an exceptionally bad and permissive implementation of a DER parser/serializer.
2. Because of OpenSSL's dominance, a lot of "DER" in the wild was really a mixture of DER and BER. This has caused an absolutely obscene amount of pain in PKI standards, which is why just about every modern PKI standard that uses ASN.1 bends over backwards to emphasize that all encodings must be DER and not BER.
(2) in particular is pernicious: the public Web PKI has successfully extirpated BER, but it still skulks around in private PKIs and more neglected corners of the Internet (like RFC 3161 TSAs) because of a long tail of OpenSSL (and other misbehaving implementation) usage.
Overall, DER itself is a mostly normal looking TLV encoding; it's not meaningfully more complicated than Protobuf or any other serialization form. The problem is that it gets mashed together with BER, and it has a legacy of buggy implementations. The latter is IMO more of a byproduct of ASN.1's era -- if Protobuf were invented in 1984, I imagine we'd see the same long tail of buggy parsers regardless of the quality of the design itself.
> rust-asn1[1] is a nice demonstration of this: you can deserialize into a structure if you know your structure AOT, or you can deserialize into the equivalent of a "value" wrapper that enumerates/enforces all valid encodings.
Almost. The "tag" of the data doesn't actually tell you the type of the data by itself (most of the time at least), so while you can say "there is something of length 10 here", you can't say if it's an integer or a string or an array.
Could you explain what you mean? The tag does indeed encode this: for an integer you'd see `INTEGER`, for a string you're see `UTF8String` or similar, for an array you'd see `SEQUENCE OF`, etc.
You can verify this for yourself by using a schemaless decoder like Google's der-ascii[1]. For example, here's a decoded certificate[2] -- you get fields and types, you just don't get the semantics (e.g. "this number is a public key") associated with them because there's no schema.
[1]: https://github.com/google/der-ascii
[2]: https://github.com/google/der-ascii/blob/main/samples/cert.t...
So yeah, in that instance you do need a schema to make progress beyond "an object of some size is here in the stream."
PER lacks type information, making encoding much more efficient as long as both sides of the connection have access to the schema.
This has the fun side effect that DER essentially allows you to process data ("give me the 4th integer and the 2nd string of every third optional item within the fifth list") without knowing what you're interpreting.
If the schema uses IMPLICIT tags then - unless I'm missing something - this isn't (easily) possible.
The most you'd be able to tell is whether the TLV contains a primitive or constructed value.
This is a pretty good resource on custom tagging, and goes over how IMPLICIT works: https://www.oss.com/asn1/resources/asn1-made-simple/asn1-qui...
> Because of OpenSSL's dominance, a lot of "DER" in the wild was really a mixture of DER and BER
:sweat: That might explain why some of the root certs on my machine appear to be BER encoded (barring decoder bugs, which is honestly more likely).
You need to populate a string? First look up whether it's a UTF8String, NumericString, PrintableString, TeletexString, VideotexString, IA5String, GraphicString, VisibleString, GeneralString, UniversalString, CHARACTER STRING, or BMPString. I'll note that three of those types have "Universal" / "General" in their name, and several more imply it.
How about a timestamp? Well, do you mean a TIME, UTCTime, GeneralizedTime, or DATE-TIME? Don't be fooled, all those types describe both a date _and_ time, if you just want a time then that's TIME-OF-DAY.
It's understandable how a standard with teletex roots got to this point but doesn't lead to good implementations when there is that much surface area to cover.
no, not at all
they share some ideas, that doesn't make it "pretty much ASN.1". Its only "pretty much the same" if you argue all schema based general purpose binary encoding formats are "pretty much the same".
ASN.1 also isn't "file" specific at all it's main use case is and always has been being used as message exchange protocols.
(Strictly speaking ASN.1 is also not a single binary serialization format but 1. one schema language, 2. some rules for mapping things to some intermediate concepts, 3. a _docent_ different ways how to "exactly" serialize things. And in the 3rd point the difference can be pretty huge, from having something you can partially read even without schema (like protobuff) to more compact representations you can't read without a schema at all.)
At the implementation level they are different, but when integrating these protocols into applications, yeah, pretty much. Schema + data goes in, encoded data comes out, or the other way around. In the same way YAML and XML are pretty much the same, just different expressions of the same concepts. ASN.1 even comes with multiple expressions of exactly the same grammar, both in text form and binary form.
ASN.1 was one of the early standardised protocols in this space, though, and suffers from being used mostlyin obscure or legacy protocols, often with proprietary libraries if you go beyond the PKI side of things.
ASN.1 isn't file specific, it was designed for use in telecoms after all, but encodings like DER work better inside of file formats than Protobuf and many protocols like it. Actually having a formal standard makes including it in file types a lot easier.
Yes, JOSE is still infinitely better than XmlSignatures and the canonical XML madness to allow signatures _inside_ the document to be signed.
- huge braking change with the whole cert infrastructure
- this question was asked to the people who did choose ASN.1 for X509 and AFIK they saied today they would use protobuf. But I don't remember where I have that from.
- JOSE/JWT etc. aren't exactly that well regarded in the crypto community AFIK or designed with modern insights about how to best do such things (too much header malleability, too much crypto flexibility, too little deterministic encoding of JSON, too much imprecise defined corner cases related to JSON, too much encoding overhead for keys and similar (which for some pq stuff can get in the 100KiB ranges), and the argument of it being readable with a text editor falls apart if anything you care about is binary (keys, etc.) and often encrypted (producing binary)). (And IMHO opinion the plain text argument also falls apart for most non-crypto stuff I mean if you anyway add a base64 encoding you already dev need tooling to read it, and weather your debug tooling does a base64 decode or a (maybe additional) data decode step isn't really relevant, same for viewing in IDE which can handle binary formats just fine etc. but thats an off topic discussion)
- if we look at some modern protocols designed by security specialists/cryptographers and have been standardized we often find other stuff (e.g. protobuf for some JWT alternatives or CBOR for HSK/AuthN related stuff).
- ASN.1 is a set of a docent different binary encodings
- ASN.1's schema languages is IMHO way better designed then Protobuf but also more complex as it has more features
- ASN.1 can encode much more different data layouts (e.g. things where in Protobuf you have to use "tricks") each being layout in the output differently depending on the specific encoding format, annotations on the schema and options during serialization
- ASN.1 has many ways to represent things more "compact" which all come with their own complexity (like bit mask encoded boolean maps)
overall the problem of ASN.1 is that it's absurdly over engineered leading to you needing to now many hundred of pages of across multiple standard documents to just implement one single encoding of the docent existing ones and even then you might run into ambiguous unclear definitions where you have to ask on the internet for clarification
if we ignore the schema languages for a moment most senior devs probably can write a crappy protobuf implementation over the weekend, but for ASN.1 you might not even be able to digest all relevant standards in that time :/
Realistically if ASN.1 weren't as badly overengineered and had shipped only with some of the more modern of it's encoding formats we probably would all be using ASN.1 for man things including maybe your web server responses and this probably would cut non image/video network bandwidth by 1/3 or more. But then the network is overloaded by image/video transmissions and similar not other stuff so I guess who cares???!???
I had to look up https://www.merriam-webster.com/dictionary/docent
With the top level encoding solved, we could then go back to arguing about all the specific lower level encodings such as compressed vs uncompressed curve points, etc.
1] the somewhat ironic part is that when it was discovered that using just passwords for authentication is not enough, the so called "lighweight" LDAP got arguably more complex that X.500. Same thing happened to SNMP (another IETF protocol using ASN.1) being "Simple" for similar reasons.
> which can transform ASN.1 into a much more parseable JSON AST
The sign of a person who's been hurt, and doesn't want others to feel the same pain :D
What I disagree is on the disdain being veiled. Seems very explicit to me.
Anyway, yeah, I hadn't heard about it before either, and it's great to know that somebody out there did solve that horrible problem already, and that we can use the library.
(i worked with asn1c (not sure which fork) and had to hack in custom allocator and 64bit support. i shiver every time something needs attention in there)
Honestly any compiler project in pure C is pretty hardcore in my eyes, ASN.1 must amplify the sheer horror.
* unit tests anywhere, so I usually write my methods/functions with unit tests following them immediately
* blocks like version(unittest) {} makes it easy to exclude/include things that should only be compiled for testing
* enums, unions, asserts, contract programming are all great
I would say I didn't have to learn D much. Whatever I wanted to do with it, I would find in its docs or asked ChatGPT and there would always be a very nice way to do things.
From a philosophical/language-design standpoint, it ticks so many boxes. It had the potential to be wildly popular, had a few things gone differently.
If the language tooling and library ecosystem were on par with the titans of today, like Rust/Go, it really would be a powerhouse language.
Still much better than GCCGO, kind of useless for anything beyond Go 1.18, no one is updating it any longer, and may as well join gcj.
D definitely missed a critical period, but I love it all the same.
IMO, the bigger issue is language tooling.
Any system designed picking such standards is basically betraying their client.
I think, if you want to annoy these people maximally, you should write an annotated version of the standard in a mathematical formal language.
I read the table constraints, which try to do something simple, but it's written in the most convoluted way possible.
I think I considered ASN.1 for a system once, but rejected it because of more modern technically superior system.
If the parser for something like ASN.1 doesn't fit in 80 lines of Haskell, perhaps you just shouldn't use it.
I don't know who these assholes are that say "Sure, let's make things slow and buggy, since we all hail Satan after all".
For those of you who missed this, there was a very interesting thing that happened in the growth of the internet.
At the time people were evolving the protocols through the IETF. So all the things that you rely on now - for the most part - just came into being. One day there was email. There was ftp. There was TCP. There were the Van Jacobson TCP mods.
At this time corporate types paid no attention to the internet. Academic types and the IETF were from what I saw the main developers.
Then one day the corporate world realized they might make money. But the development process of the protocols was incomprehensible (and incompatible) with the corporate culture. TCP was clearly a mess, all these protocols like DNS were a mess. From the corporate perspective.
So began the protocol wars https://en.wikipedia.org/wiki/Protocol_Wars.
Whether ASN.1 was a product of that war or just a product of the corporate mentality, it serves as a powerful instance of the what the corporate world looks like vs the academic world looks like. You can find the wreckage from the war littered around. If you see and X.something protocol it could well be one of the relics. There were a few X.things that were adopted and useful, but were others that would haunt your dreams.
Although this is ancient history, and pretty much now told from the corporate perspective, it suggests to us that the corporate process for thinking is not as effective as the alternative - the IETF and Academic.
One is a sort of recipe culture. You write a recipe, everyone follows it and you are happy. The other is a sort of functional culture. If you can make bread and eat it you are happy. When the bread doesn't taste good you fix it.
Given the kind of bread that is commonly available in the US now, we can draw some conclusions about recipe thinking, recipe culture, corporate culture etc. One could even extend this paradigm of thinking to new things like AI. Or not.
I had to pause the movie and explain to my partner just how close the world came to missing out on The Internet, and having instead to suffer the ignominy of visiting sites with addresses like “CN=wikipedia, OU=org, C=US” and god knows what other dreadful protocols underlying them. I think she was surprised how angry and distressed I sounded! It would have been awful!
Poor her!
Monday-morning-quarterbacking is an unproductive pastime, but I don't think it was very close, on account of the Internet side having developed a bunch of useful (if a bit ramshackle) protocols and applications much faster than the ISO team, because the specs were freely available (not to mention written in a much more understandable manner). I still rue the day the IETF dropped the "distribution of this memo is unlimited" phrase from the RFC preambles. Yeah I understand that it originally had more to do with classification than general availability, but it described the ethos perfectly.
It's not all roses and we're paying for the freewheeling approach to this day in some cases, cf. email spam and BGP hijacking. But it gave results and provided unstoppable momentum.
A lot of issues also came from ISO standards refusing to get stuck without known anticipated issues taken care of, or with unextendable lockin due to accidental temporary solution ending up long term one, while IETF protocols happily ran forward "because we will fix it later" only to find out that I stalled base ossified things - one of the lessons is to add randomness to new protocols so that naive implementation will fail on day one.
Then there were accidental things, like a major ASN.1 implementation for C in 1990 being apparently shit (a tradition picked up in even worse way by OpenSSL and close to most people playing with X.509 IMO), or even complaints about ASN.1 encodings being slow due to CPU lacking barrel shifters (I figure it must refer to PER somehow)
I'm sorry you had to waste a year of your life.
There are few things I dislike more in the computing world than ASN.1/BER. It seems to encourage over-specification and is suboptimal for loosely coupled systems.
But it looks like you had a decent time...
Oh and there is also non compliant implementations. E.g. some passports (yes the passports with chip use tons of ASN.1) even have incorrect including of big integers (supposed to be the minimum two complement, as I recall some passports used a fixed non-complement format yanked into the 0x02 INTEGER type... Some libraries have special non-compliant parsing modes to deal with it).
Somehow, despite these specifications being 90% metadata by weight, they seem to consistently forget the part of the spec that lets you actually know what something is. and that part is just left up to context.
I could well be missing something, but a majority of the time it feels to me like they set out to make a database schema, and accidentally wrote the sqlite file format spec instead.
Like thanks, its nice that I can parse this into a data structure :). It would be nicer, however if doing so gave me any idea of what I can do with the data I've parsed.
Though to be fair I feel the same way about the entire concept of XML schemas. The fact that you theoretically can validate an xml document against a schema is honestly completely useless. If I am parsing XML, its because my code already knows what information it needs from the XML document, and really it should also know where that information is. I don't need a separate schema definition to tell me that! its already expressed!! In the part where I am parsing out the data I need!!!
You seem to miss the entire point of XML schemas, or any schema really. Validating a document against a schema isn’t really for your code. It’s for documentation of what can be in a given document and how it needs to be structured. So others don’t need to read your code to understand that.
It then allows editing tools to verify generated documents. Or developers to understand how they can structure XML output properly.
Your code could also use it to verify an XML document before passing it to your code. Then you can inform the user of an invalid document and why instead of just crashing at a random point in code without rolling your own. It can also verify an entire document whereas code may only parse portions leading to later corruption.
1) parse the ASN.1 files, 2) generates the equivalent declaration in a programming language (like C or C++), 3) generates the encoding and decoding functions based on the previous declarations
All of these of exercise are apparently part of data engineering process or lifecycle [1].
Back in early 21st century Python is just another interpreted general purpose programming language alternative, not for web (PHP), not for command tool (TCL), not for system (C/C++), not for data wrangling (Perl), not for numerical (Matlab/Fortran), not for statistics (R).
D will probably follow similar trajectory of Python, but it really needs a special kind of killer application that will bring it to the fore.
I'm envisioning that real-time data streaming, processing and engineering can be D killer utility and defining moment.
[1] Fundamentals of Data Engineering:
https://www.oreilly.com/library/view/fundamentals-of-data/97...
BradleyChatha•4h ago
So I threw a bunch of semi-related ramblings together and I'm daring to call it a blog post.
Sorry in advance since I will admit it's not the greatest quality, but it's really not easy to talk about so much with such brevity (especially since I've already forgot a ton of stuff I wanted to talk about more deeply :( )
whizzter•2h ago
BradleyChatha•2h ago
StopDisinfo910•1h ago
It was hilarious because clearly none of the people who were in favor had ever used ASN.1.
mananaysiempre•43m ago
[1] https://news.ycombinator.com/user?id=cryptonector
[2] https://github.com/heimdal/heimdal/tree/master/lib/asn1
hamburglar•1h ago
You are truly a masochist and I salute you.
cyberax•12m ago
Completely ignoring the ASN.1 support for complicated structures, with more than one CVE linked to incorrect parsing of these text fields m
throw_a_grenade•2h ago
giancarlostoro•2h ago
pjmlp•1h ago
The various WIP features, and switching focus of what might bring more people into the ecosystem, have given away to other languages.
Even C#, Java and C++ have gotten many of features that were only available in D as Andrei Alexandrescu's book came out in 2011.
mort96•49m ago
And still today, the first thought that comes to mind when I think D is "that language with proprietary compilers", even though there has apparently been some movement on that front? Not really worth looking into now that we have Go as an excellent GC'd compiled language and Rust as an excellent C++ replacement.
Having two different languages for those purposes seems like a better idea anyway than having one "optionally managed" language. I can't even imagine how that could possibly work in a way that doesn't just fragment the community.
sfpotter•38m ago
mort96•36m ago
> There are a lot of loud people in the D community who freak out and whine about the GC, and there are plenty more quiet ones who are happily getting things done without making much noise
This sounds like an admission that the community is fractured, except with a weirdly judgemental tone towards those who use D without a GC?
olvy0•55m ago
mananaysiempre•39m ago