(I suspect, to paraphrase Greenspun's rule, any sufficiently complicated app using Web Workers contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of this library...)
If you reference something lexically, your code fails at runtime. Want to use an import? You have to use import() inside the closure you pass to spawn(). Typescript doesn't know this. Your language server doesn't know this. Access a variable that shadows a built in global? Now you're accessing the built in global.
The only way this could even be addressed is by having a full on parser. Even then you can't guarantee things will work.
I think the only "fix" is for JS to introduce a new syntax to have a function that can't access lexical scope, returning a value that either extends a subclass of Function or has a cheeky symbol set on it. At least then, it'll fail at compile time.
The push from language designers (this applies across the high/low level spectrum and at all ranges of success for languages) to make concurrent code ‘look just like’ linearly read, synchronous, single-threaded code is pervasive and seems to avoid large pushback by users of the language. The complaints that should be made against this syntax design become complaints that code doesn’t do what developers think it should.
My position is that concurrent (and parallel) code IS NOT sequential code and languages should embrace those differences. The move to or design of async/await is often explicitly argued for from this position. But the semantic differences in concurrent code IMO should not be obscured or obfuscated by seeking to conform that code to sequential code’s syntax.
Overall it's great, and I'm glad to see a generic implementation of it which will hopefully become a thriving open source project, but ultimately it's a kludge. What's really needed is for JS to introduce a native standardized version of this construct which TypeScript and the rest of the ecosystem have to play nice with.
> If you are using Bun (which doesn't natively support using and uses a transpiler which is incompatible with this library)...
I skimmed the issues but I couldn't find any issues on Bun regarding this except for: https://github.com/oven-sh/bun/discussions/4325
> While Bun is supported and Bun does support the `using` keyword, it's runtime automatically creates a polyfill for it whenever Function.toString() is called. This transpiled code relies on specific internal globals made available in the context where the function is serialized. Because the worker runs in a different isolated context where these globals are not registered, code with `using` will fail to execute.
Writing module bundlers in Javascript had diminishing returns from multi threading because of the overhead of serializing and deserializing ASTs.
I wonder how far something like this would push the ceiling. Would love to see some benchmarks of this thing hauling ASTs around.
The OS does thread management and scheduling, facilitates IPC, locking, etc. All of this is one big largely-solved problem (at least for the kind of things most people are doing in JavaScript today). But because of history, we now have a very popular language and runtimes that are trying to replicate all these features, reinventing wheels, and adding layers on inefficiency to the overall execution.
Sigh.
Lack of easy shared memory has always felt like a problem to me in this space, as often the computation I want to off-load requires (or returns) a lot of data.
> Serialization Protocol: The library uses a custom "Envelope" protocol (PayloadType.RAW vs PayloadType.LIB). This allows complex objects like Mutex handles to be serialized, sent to a worker, and rehydrated into a functional object connected to the same SharedArrayBuffer on the other side.
It's kinda "well, yes, you can't share objects, but you can share memory. So make objects that are just thin wrappers around shared memory"
There’s also the queuing and blocking nature of web-workers, I wish they could asynchronously process messages the same way js IO works, but that’s not the case. Rather you are batching full units of work. The mental model is different.
Anecdotally in Firefox I must have run into some memory leak issues and had to hard restart.
Ultimately I ended up going with service workers, which yes sounds strange but I found to be much easier to work with. Cancellable requests, async, long living in the background … but maybe it just works best for me ;)
One such example: https://github.com/developit/workerize-loader
I want the pre-existing DOM thread to enter into a long term relationship with a particular web worker thread. I'd like the web worker to hold onto a transferrable that has ties back to the DOM thread, the WebGLRenderingContext, and have the DOM thread send draw commands to the worker and its gl context over time.
I imagine this could be achieved by having a dedicated web worker pool of size 1 and allowing the DOM thread to initiate an async send to the worker pool.
Can your API do something like this? Thank you for this early Christmas gift!
Zedriv•2mo ago
dwoldrich•2mo ago