WebGPU is an amazing piece of technology, allowing for truly cross-platform graphics and GPGPU programming. It is being used by frameworks like Three.js and TensorFlow to accelerate their internals. However, when we try to connect libraries together, we soon hit a limit...
Let's say we wanted to pull a texture out of Three.js, and use it as a tensor in Tensorflow.js. If their internal data structure matches, they can just share a VRAM pointer to avoid copying to RAM and back unnecessarily (could be ~100x slower than the actual work we want to do). Unfortunately, it is rare for APIs to be seamlessly compatible with one another, so we need "glue code" for interop. We have two options:
1. Copy data, and transform it in JS/TS.
2. Write a compute shader to operate on VRAM directly, and glue the APIs there.
Well... what if we could write the glue code in TypeScript, and compile it to WebGPU Shading Language instead? We would get hints from the language server about both the output of Three.js, and the input of Tensorflow.js. My team and I have been building a solution just for this, a TypeScript library called TypeGPU!
I like to use the analogy of server & client, as writing both CPU and GPU logic in TypeScript gives you the same benefits here. Write a single code-base, and using modern tooling like Vite, tell the bundler which functions should be executable on the GPU. We hook into the build process with our custom plugin to allow for this. The GPU can be thought of as just an endpoint with an API, and instead of binary blobs and strings, that API can be made type-safe!
And it's not just the "glue code" that becomes better, library APIs can become more customizable! A library can defer shader logic to the user by accepting marked TypeScript functions. Dependency Inversion, without compromising efficiency! We're building libraries on top of TypeGPU as we speak, and we would love to work with everyone building their own GPU-enhanced libraries. You can keep full flexibility of your internals, and still use plain WebGPU & WGSL. We handle the API, so you can focus on your library's core value.
I would love to hear everyone's thoughts on the problem, and the proposed solution
iwoplaza•7mo ago
Let's say we wanted to pull a texture out of Three.js, and use it as a tensor in Tensorflow.js. If their internal data structure matches, they can just share a VRAM pointer to avoid copying to RAM and back unnecessarily (could be ~100x slower than the actual work we want to do). Unfortunately, it is rare for APIs to be seamlessly compatible with one another, so we need "glue code" for interop. We have two options: 1. Copy data, and transform it in JS/TS. 2. Write a compute shader to operate on VRAM directly, and glue the APIs there.
Well... what if we could write the glue code in TypeScript, and compile it to WebGPU Shading Language instead? We would get hints from the language server about both the output of Three.js, and the input of Tensorflow.js. My team and I have been building a solution just for this, a TypeScript library called TypeGPU!
I like to use the analogy of server & client, as writing both CPU and GPU logic in TypeScript gives you the same benefits here. Write a single code-base, and using modern tooling like Vite, tell the bundler which functions should be executable on the GPU. We hook into the build process with our custom plugin to allow for this. The GPU can be thought of as just an endpoint with an API, and instead of binary blobs and strings, that API can be made type-safe!
And it's not just the "glue code" that becomes better, library APIs can become more customizable! A library can defer shader logic to the user by accepting marked TypeScript functions. Dependency Inversion, without compromising efficiency! We're building libraries on top of TypeGPU as we speak, and we would love to work with everyone building their own GPU-enhanced libraries. You can keep full flexibility of your internals, and still use plain WebGPU & WGSL. We handle the API, so you can focus on your library's core value.
I would love to hear everyone's thoughts on the problem, and the proposed solution