After 25 years on Windows I recently switched to Linux over the holiday break and came across this with my GeForce 750 Ti which only has 2 GB of GPU memory.
After opening a few Firefox and Ghostty terminals (both apps are hardware accelerated), I notice system instability and things crash or my Wayland compositor (niri) starts failing in unpredictable ways. All journalctl logs point to NVIDIA / kernel errors allocating resources.
Turns out it's likely due to the NVIDIA drivers on Linux not using system memory if the GPU memory is overburdened but on Windows this happened automatically. For reference this is with the official proprietary NVIDA drivers (latest open dkms 580 series for this card).
It's why on Windows I was able to open as many apps as I wanted, record x264 videos with OBS and play games without a single crash in literally 10 years.
There is a multi-year thread on the NVIDIA developer forums with hundreds of people reporting this but NVIDIA says it's not a problem https://forums.developer.nvidia.com/t/non-existent-shared-vram-on-nvidia-linux-drivers/260304/.
It made me think, something must be wrong or mis-communicated because this would be a deal breaker for anyone using NVIDIA GPUs on Linux. I mean, if I open literally 8 Firefox windows or 15 Ghostty terminals, my system crashes but I am only using 3 GB out of 16 GB of system memory but my GPU gets ~75% full and that's when issues start happening. Games are out of the question, but those same games run perfectly well on Windows.
You can verify this by running `nvida-smi` to see your current GPU usage. You can also run `nvidia-smi -q` to get more details. Going by a few replies in that thread, the BAR1 memory usage value is how much memory the GPU has access to which would include system memory if your hardware supports it. Allegedly if you do some tweaking and have a 30XX card you can configure that but everyone else is out of luck. I don't really know for sure what it is, I just know my value is very low and I cannot tweak it.
Here is my nvidia-smi -q output, you can see BAR1 is very low:
FB Memory Usage
Total : 2048 MiB
Reserved : 58 MiB
Used : 923 MiB
Free : 1068 MiB
BAR1 Memory Usage
Total : 256 MiB
Used : 38 MiB
Free : 218 MiB
AMD and Intel cards do not have this issue btw, it's isolated to NVIDIA.
bigyabai•1mo ago
My 3070Ti reports similar memory usage at idle, but even when my VRAM is exhausted I don't get an OOM crash. The only time I'd ever had my GPU OOM panic was with my 1050 Ti on very old Nvidia drivers. You'll probably want a newer GPU for Wayland stuff.
nickjj•1mo ago
NVIDIA's official stance is the DKMS 580 series drivers do support my card. The 590 series do not but I didn't try those. Basically the 75X, 8XX, 9XX and 10XX cards all use the 580 series driver. I don't know about 6XX or less (I didn't check).
Getting a newer GPU is going to be sketchy I think. It's a machine I built up from parts in 2014. The motherboard is an ASRock H97M with PCIe 3.0 x16. I don't think the PSU can even power a 1080 and even then, a 1080 would use the same 580 series drivers as this card so it might have the same problem, just with a higher ceiling (8 GB instead of 2 GB). It's also disgusting that this card is over $200 used in 2025. Using someone else's 10 year old card in itself is a massive gamble.
When the instability starts to happen due to high GPU memory usage I get this in journalctl:
Dec 28 12:41:10 kaizen kernel: [drm:nv_drm_gem_alloc_nvkms_memory_ioctl [nvidia_drm]] ERROR [nvidia-drm] [GPU ID 0x00000100] Failed to allocate NVKMS memory for GEM object
Closing out windows to reclaim GPU memory isn't enough to bring things back to operational. I often get compositor freezes and other weird artifacts like windows taking up space but not being there until I reboot. It's like niri gets in an unrecoverable state that it doesn't recover from. I say reboot but logging out and back in also works, but this is quite disruptive to be doing a few times a day and hinders me from recording videos since I often run into issues while doing my day to day stuff.
I really wonder how to resolve it if it's even possible. This box is my main dev / media creation box and it was super good and stable on Windows for 11 years.