In many languages, it’s simply not possible to match function names in definitions or calls using coarse parsing. C is definitely such a language. A large part of the problem is when you don’t have explicit delimiting syntax. That’s what you need. Oils, by contrast, looks to say `proc demo { … }`, so you can look for the `proc` keyword.
Vim’s syntax highlighting is unfortunately rather limited, and if you try to stretch what it’s capable of, it can get arbitrarily slow. It’s my own fault, but the Rust syntax files try to be too clever, and on certain patterns of curly braces after a few hundred lines, any editing can have multiple seconds of lag. I wish there were better tools for identifying what’s making it slow. I tried to figure it out once, but gave up.
I’ve declared coarse parsing rerally good for the basics in almost all programming languages, and that explicit delimiting syntax is necessary. This leads to probably my least favourite limitation in Vim syntax highlighting: you can’t model indent-based mode switching. In Markdown, for example (keep the leading two spaces, they’re fine):
Text
Code
1. Text
Vim says code, actually text
Code
This is a real problem for my own lightweight markup language too, which uses meaningful indentation.
And yes, there are a whole bunch of limitations:
- C function definitions - harder than JavaScript because there's no "function". It's still syntactic, but probably requires parsing, not just lexing.
- C variable definitions - I'd call this "coarse semantic analysis", not coarse parsing! Because of the "lexer hack"
- Indentation as you mention - there was a thread where someone was complaining that treesitter had 2 composed parsers for Markdown -- block and inline -- although I'm not sure if this causes a problem in practice? (feedback appreciated)
---
But I did intend for YSH to be easier to parse than shell, and that actually worked, because it fits quite well in Vim!
I noted here that OSH/bash has 10 lexer modes -- I just looked and it's up to ~16 now
https://www.oilshell.org/blog/2019/02/07.html#2019-updates
Whereas YSH has 3 mutually recursive modes, and maybe 6 modes total.
---
On the "coarse semantic analysis", another motivation is that I found Github's semantic source browser a bit underwhelming. Not sure if others had that same experience. I think it can't really be accurate because it doesn't have a lot of build time info. So I think they could have embraced "coarseness" more, to make it faster
Although maybe I am confusing the UI speed with the analysis speed. (One reason that this was originally a Codeberg link is that Codeberg/Forejo's UI is faster, without all the nav stuff)
There are some related links here, like How To Build Static Analyzers in Orders of Magnitude Less Code:
https://github.com/oils-for-unix/oils/wiki/Polyglot-Language...
This is really no different than how we process language, though? Even using proper names everywhere, turns out proper names get reused. A lot. Such that you pretty much have to have an active simulation of what you are reading in order for most things to attach to identities. No?
And, directly to my second sentence, we have the resources that you don't need to cater to the coarsest possible capabilities. You can augment the things you are looking for just fine. You can see in one pass that a new function named "foo" was added, then in another pass, you can start highlighting the function foo easily enough. Staying in a coarse world, you could probably even add a regex that roughly checks the correct number of parameters. Yes, it takes more passes, but isn't impossible?
Further still, you could basically scan all identifiers and any that are a short hamming distance from each other can be queried to see if they are mistakes. Any that match after case folding? Still coarsely found, but still very helpful.
My point on how we treat language is that we do that more than not. Is my biggest gripe with the jump from a coarse parse straight to a context free grammar approach. We work in contexts. I don't know why we go through so much effort to make the context unnecessary.
As a fun example in natural language where I've seen this. We were going on a trip to Savannah with our kids when they were younger. The oldest was so excited that she was going to get to see cheetahs. An easy mistake to spot when being asked to look for it.
It absolutely nests! Vim's model has recursion, and it works perfectly. Some details here:
https://github.com/oils-for-unix/oils.vim/blob/main/doc/stag...
This highlighter is extremely accurate, and I would call it correct. I list about 3 known issues here, and they are all fixable/expressible in Vim's model:
https://github.com/oils-for-unix/oils.vim/blob/main/doc/algo...
Please install it, and file bugs with any inaccuracies. If YSH code is valid, it should not be mis-highlighted. There is test data in false-postive.ysh and false-negative.ysh.
Try to break it!
---
There are lots of Vim/Textmate plugins that are buggy, but it doesn't mean that all such plugins are.
generate semantic tokens outside
I'd also say that this doesn't really help, since I believe Tree-sitter is the most common way of doing that. I show at the top of the doc that Tree-sitter has issues in practice expressing shell (although admittedly it's not a fair comparison to YSH in Vim. Shell in Vim will have more problems, although in practice I find it pretty good)
https://github.com/oils-for-unix/oils.vim/blob/main/doc/stag...
Stage 1 is non-recursive, but stage 2 is recursive.
tbh the plugin is impressive, no question. but that memoryless model will always need compromises, rule layering, and finetuned sync tricks. treesitter has its issues too, agreed. but having typed nodes and scope trees gives a baseline advantage when meaning depends on ancestry.
ysh.vim solves exactly that problem.
The [ in command mode (test) is not highlighted.
In contrast, the [ within expressions like a[i] is highlighted (currently Normal, but you can make it any color - https://github.com/oils-for-unix/oils.vim/blob/main/syntax/l... )
Again I recommend trying it. Verify what you think the bugs are. I think you have some preconceptions based on using other Vim plugins.
Vim's model is powerful enough to write good plugins, or bad plugins. That is one of the main points of this article.
I'm also curious on how many times you have used something that didn't get syntax highlighting correct? Even using some of the more advanced cweb features of org-mode, it typically gets things more correct than not. And I don't think it is using anything more than regexps? (I have not checked to see how the tree sitter stuff interacts with cweb in many blocks. Will try and look into that.)
e.g. although Vim's syntax highlighting helped me learn shell, it highlights numbers like 'echo 42' in a special way, which is misleading, because shell doesn't have numbers. (On the other hand, YSH does, but not in 'echo 42' either!)
On the other hand, there are also language design issues. Shell also allows MULTIPLE here docs, and I claim that ZERO syntax highlighters handle it correctly - https://github.com/oils-for-unix/oils.vim/blob/main/demo/bad...
(YSH removes here docs in favor of Python-like multi-line strings)
---
But the "surprise" in this article is that Vim is powerful, and you can write a good syntax highlighter or a bad one. There are many possible "programs" to write in this paradigm
I'd also say "completely accurate" highlighting doesn't really exist in practice, and is even problematic in theory.
Tree-sitter grammars are not completely faithful to the original language, because the metalanguage is limited. And highlighters have to deal with incomplete code, so it's not clear what "two parsers being the same" means.
The approach of regions and match items which can contain each other in a hierarchy can handle anything.
By the way, I use Vim for web requests to highlight code served by CGIT.
The vast majority of code you are reading is almost definitionally syntactically correct, unless you are in the process of editing it. In that case, syntax highlighting can provide a lightweight proxy for correctness, which I suspect is where much of the enthusiasm comes from. What I personally want is immediate feedback on the actual correctness of the code, and syntax is just a subset.
That is not to say that highlighting is never useful. I just want it to be manual, like when you search for something with / in vim. Then the highlighted items actually pop and my eyes can go directly to the area I want to focus on. I immediately clear the highlighting as soon as I'm done because otherwise it creates a visual distraction.
In my estimation, what we actually need more of are smarter, faster compilers that can immediately respond to edit changes and highlight only the problem areas in the code. Typically, this should be exactly where my cursor is. I should ideally be programming in a state where everything above the cursor can be assumed correct, but there might be a problem with the current word, which is helpfully reported to me exactly where my focus already lies.
chubot•7mo ago
https://github.com/oils-for-unix/oils.vim/blob/main/doc/algo...
tomhow•7mo ago