Is there any way that TinyKVM + KVM Server could ever be made to work with a GUI program? The sandboxing performance seems free and possibly safer than other solutions.
Instead of firejail or bubblewrap would it ever be possible for me to wrap say Firefox (or a much less complicated GUI program) inside of TinyKVM and restrict it to just network access and reading/writing to ~/Downloads? Likely a way more ambitious target than you had ever imagined, but I can dream.
I am wondering if I could default wrap every command on my terminal to run inside a TinyKVM, no network access, and only permissions to the current directory or below.
wmf•1h ago
It sounds like you're talking about Qubes.
jchw•1h ago
That really isn't unreasonable at all IMO, it's just that it might be hard to do with userspace syscall emulation, since graphical programs will likely need a lot more of the syscall surface. For X11 and Wayland, you'll need some way of handling UNIX domain sockets. Wayland applications will require shared memory too, though you could get away with something like Waypipe instead to serialize everything. You'd probably want some sort of intermediary between any X11/Wayland communications anyways, just to add additional isolation.
It might be easier to adapt gVisor to handle this sort of workload. Adjacent comment mentions Qubes which does the same thing but uses an entire guest kernel.
(If you are creative enough, you can probably come up with some solutions. Qt apps could be made to work with a custom QPA that can somehow funnel information in and out of the sandbox. You could definitely run something like Waypipe or Xpra in the sandbox too, but again I imagine those would wind up requiring a much greater degree of emulation. It's not like I've actually tried this, though, so I could be off.)
laurencerowe•1h ago
TinyKVM is probably most similar to gVisor in KVM platform mode. TinyKVM implements a smaller number of sys calls and is focussed on making resets as fast as possible.
Running sys calls on the host means there is approximately 1µs overhead per syscall from exiting and entering KVM so I'm not sure how well that would work for GUI applications.
And we currently only have very rudimentary support for threads, enough for a server program with ancillary threads to boot up but the expectation is currently that the call into TinyKVM only runs a single thread and we fork multiple copies of the VM to handle requests in parallel.
jchw•21m ago
> Running sys calls on the host means there is approximately 1µs overhead per syscall from exiting and entering KVM so I'm not sure how well that would work for GUI applications.
That made me rather curious how many syscalls a complex GUI application might issue. I wanted to see how many syscalls were happening across my entire system. Thanks to StackOverflow I have a snippet that seems correct[1]:
> perf stat -e raw_syscalls:sys_enter -a -I 1000 sleep 5
Using this, it seems that most programs (as you would probably guess) don't execute a whole lot of syscalls when they're idle. However, starting a complex GUI program definitely causes a pretty massive flurry of syscalls. Starting winecfg without an already-existing wineserver spews a lot of syscalls, somewhere in the neighborhood of 500,000. If we assume that each syscall takes on average around 2µs including the overhead and that they're all serial, I guess that would add up to about 1 second spent on syscalls. That's probably making way too many assumptions, but it does make me feel like it's not completely infeasible to run GUI applications inside of a sandbox like this, though it may very not be compelling when the overhead is factored in.
And of course, just because it could be done does not mean it should, anyway. Even if this is a good idea, I doubt it makes any sense for TinyKVM to be attempting to do it. What TinyKVM does do is already very interesting and probably a lot more practical anyways. It'd probably be better to fork off or build an entire purpose-built sandbox for GUI software, realistically.
Still, pretty interesting stuff to think about.
> And we currently only have very rudimentary support for threads, enough for a server program with ancillary threads to boot up but the expectation is currently that the call into TinyKVM only runs a single thread and we fork multiple copies of the VM to handle requests in parallel.
BTW, I think this design is really cool. This is something I have wanted to exist for a while, even though I don't practically need it.
Given the use of the word "container" that seems to be using Linux namespacing rather than KVM. In case of containers, the isolation is provided solely by the Linux kernel, plus of course any additional defenses you add on top of it. While Guix shell having a built-in way to spawn isolated containers is extremely cool (I use NixOS. As far as I know, Nix does not have an equivalent feature) it seems like from a security standpoint, it would just be similar to using bubblewrap or Firejail directly. Though I like this idea. Seems very useful and convenient.
What I think we're really after though is something like gVisor, where the guest program is completely isolated from the host kernel, and the daemons that allow the guest program to reach the outside world are themselves highly locked down by the host kernel using technologies like seccomp-bpf and namespacing, on top of whatever constraints and validation they apply on their own. While nothing is foolproof, this feels like, if done carefully, it would give you a very good layer of isolation that would be extremely challenging to bypass. I reckon that the sandbox would cease to be the most interesting attack target in a system like gVisor, since in any complicated system, there will probably always be some lower-hanging fruit. (And of course, TinyKVM seems to be basically in the same wheelhouse. None of these solutions are designed to run GUI software, though I reckon it probably could be made to work.)
munchlax•57m ago
The traditional way of doing this is by combining programs. Many programs already do this. e.g.:
time nice distcc ccache gmake
I do this with other tools as well. bwrap, chroot, env, setpriv, xchpst, etc. They all stack.
laurencerowe•1h ago
I'm pretty hopeful that the combination of per-request isolation and the new snapshot functionality we're currently working on will be a big step forward for those running server-side JS at scale.
Having each request start from the exact same program state should make reproducing and fixing production issues easier. In a way it combines the predictability of the CGI programming model with the speed of a warmed modern JIT runtime.
3eb7988a1663•1h ago
Is there any way that TinyKVM + KVM Server could ever be made to work with a GUI program? The sandboxing performance seems free and possibly safer than other solutions.
Instead of firejail or bubblewrap would it ever be possible for me to wrap say Firefox (or a much less complicated GUI program) inside of TinyKVM and restrict it to just network access and reading/writing to ~/Downloads? Likely a way more ambitious target than you had ever imagined, but I can dream.
I am wondering if I could default wrap every command on my terminal to run inside a TinyKVM, no network access, and only permissions to the current directory or below.
wmf•1h ago
jchw•1h ago
It might be easier to adapt gVisor to handle this sort of workload. Adjacent comment mentions Qubes which does the same thing but uses an entire guest kernel.
(If you are creative enough, you can probably come up with some solutions. Qt apps could be made to work with a custom QPA that can somehow funnel information in and out of the sandbox. You could definitely run something like Waypipe or Xpra in the sandbox too, but again I imagine those would wind up requiring a much greater degree of emulation. It's not like I've actually tried this, though, so I could be off.)
laurencerowe•1h ago
Running sys calls on the host means there is approximately 1µs overhead per syscall from exiting and entering KVM so I'm not sure how well that would work for GUI applications.
And we currently only have very rudimentary support for threads, enough for a server program with ancillary threads to boot up but the expectation is currently that the call into TinyKVM only runs a single thread and we fork multiple copies of the VM to handle requests in parallel.
jchw•21m ago
That made me rather curious how many syscalls a complex GUI application might issue. I wanted to see how many syscalls were happening across my entire system. Thanks to StackOverflow I have a snippet that seems correct[1]:
> perf stat -e raw_syscalls:sys_enter -a -I 1000 sleep 5
Using this, it seems that most programs (as you would probably guess) don't execute a whole lot of syscalls when they're idle. However, starting a complex GUI program definitely causes a pretty massive flurry of syscalls. Starting winecfg without an already-existing wineserver spews a lot of syscalls, somewhere in the neighborhood of 500,000. If we assume that each syscall takes on average around 2µs including the overhead and that they're all serial, I guess that would add up to about 1 second spent on syscalls. That's probably making way too many assumptions, but it does make me feel like it's not completely infeasible to run GUI applications inside of a sandbox like this, though it may very not be compelling when the overhead is factored in.
And of course, just because it could be done does not mean it should, anyway. Even if this is a good idea, I doubt it makes any sense for TinyKVM to be attempting to do it. What TinyKVM does do is already very interesting and probably a lot more practical anyways. It'd probably be better to fork off or build an entire purpose-built sandbox for GUI software, realistically.
Still, pretty interesting stuff to think about.
> And we currently only have very rudimentary support for threads, enough for a server program with ancillary threads to boot up but the expectation is currently that the call into TinyKVM only runs a single thread and we fork multiple copies of the VM to handle requests in parallel.
BTW, I think this design is really cool. This is something I have wanted to exist for a while, even though I don't practically need it.
[1]: https://unix.stackexchange.com/a/591299
rolandog•1h ago
[0]: https://www.futurile.net/2023/04/29/guix-shell-virtual-envir...
jchw•54m ago
What I think we're really after though is something like gVisor, where the guest program is completely isolated from the host kernel, and the daemons that allow the guest program to reach the outside world are themselves highly locked down by the host kernel using technologies like seccomp-bpf and namespacing, on top of whatever constraints and validation they apply on their own. While nothing is foolproof, this feels like, if done carefully, it would give you a very good layer of isolation that would be extremely challenging to bypass. I reckon that the sandbox would cease to be the most interesting attack target in a system like gVisor, since in any complicated system, there will probably always be some lower-hanging fruit. (And of course, TinyKVM seems to be basically in the same wheelhouse. None of these solutions are designed to run GUI software, though I reckon it probably could be made to work.)
munchlax•57m ago
time nice distcc ccache gmake
I do this with other tools as well. bwrap, chroot, env, setpriv, xchpst, etc. They all stack.