This is a repost from reddit: Hey guys! I have been excited to share this here. This is a project consisting of kernel optimizations for the Tesla p100 series graphics card ($80). I want to start by saying I am 17 years old and do not have a formal degree. I used Ai for a lot of this and while I understand some, I do not understand everything.
Notes: My gpus are capped at 175w/250w each so these numbers may be able to be pushed higher. I also experience minor thermal throttling and sit at a nice toasty 79 degrees, which definitely effect numbers (the table above is while hot, so if you have good cooling expect 5-10% more on prefill and decode). I am using gen3 pcie with two x16 slots. Also, for anyone curious, decode numbers depicted in image were averaged from a list of questions ranging from creative writing and coding.
Improved: tps went from 7-15tps at 0 context to 50-60, 260k context went from 2-4tps to 30-35, prefill went from 220 tps at 0 context to 500, 260k context went from 40tps (as far as i remember, i never really measured cause it was too hard) to 150tps, fixed fp16 math errors by using some mixed fp16/fp32 math operations so rounding errors were eliminated, merged as of sept 22 so it should support qwen 3.8 flash architecture (will merge again when qwen4 releases), and did multiagent work (1 agent 60tps, 2 is 31-38 each, and 3 is 21-24 each)
This setup is somewhat flag specific (ie: (-c 262144 -b 32768 -ub 1024 -np 1 \) without -b 32768 mtp becomes overloaded and drops acceptance to near 0 at full depth) so keep that in mind while setting up.
One more thing, I took regression very seriously in this. Math had to be more accurate or byte identical or it would fail tests. Build, flags, math proofs, and anything else you may need will be linked below.
Enjoy guys! I would love your feedback on this and am looking at pull requests.
Kmic68•59m ago
Notes: My gpus are capped at 175w/250w each so these numbers may be able to be pushed higher. I also experience minor thermal throttling and sit at a nice toasty 79 degrees, which definitely effect numbers (the table above is while hot, so if you have good cooling expect 5-10% more on prefill and decode). I am using gen3 pcie with two x16 slots. Also, for anyone curious, decode numbers depicted in image were averaged from a list of questions ranging from creative writing and coding.
Improved: tps went from 7-15tps at 0 context to 50-60, 260k context went from 2-4tps to 30-35, prefill went from 220 tps at 0 context to 500, 260k context went from 40tps (as far as i remember, i never really measured cause it was too hard) to 150tps, fixed fp16 math errors by using some mixed fp16/fp32 math operations so rounding errors were eliminated, merged as of sept 22 so it should support qwen 3.8 flash architecture (will merge again when qwen4 releases), and did multiagent work (1 agent 60tps, 2 is 31-38 each, and 3 is 21-24 each)
This setup is somewhat flag specific (ie: (-c 262144 -b 32768 -ub 1024 -np 1 \) without -b 32768 mtp becomes overloaded and drops acceptance to near 0 at full depth) so keep that in mind while setting up.
One more thing, I took regression very seriously in this. Math had to be more accurate or byte identical or it would fail tests. Build, flags, math proofs, and anything else you may need will be linked below.
Enjoy guys! I would love your feedback on this and am looking at pull requests.
Github: https://github.com/Kmic-68/llama.cpp/tree/p100-optimizations
Details for build, math proofs, etc: https://github.com/Kmic-68/llama.cpp/tree/p100-optimizations...