Sad they are mostly sitting there unused because very few people know how to program them.

The goal here was an end-to-end RK3588S pipeline rather than comparing detector families: training/export, ONNX graph fixing, INT8 RKNN conversion, C++ postprocessing, and runtime inference across the 3 NPU cores. YOLOv8 has known-good export paths and Rockchip examples, so it was the most practical baseline.

Newer YOLO versions may be possible, but usually require more work around RKNN export compatibility.

This pipeline is processing live camera frames and displaying/streaming annotated output, so latency and frame freshness matter. Increasing batch size would add queueing latency and tends to make the output older, especially when the sensor is producing frames continuously.

The “multithreading” here is not treating the NPU like a CPU in the usual sense. The RK3588S NPU is exposed as 3 cores, and RKNN supports using separate contexts with `rknn_dup_context` and assigning them with `rknn_set_core_mask`. The point was to keep the 3 NPU cores fed while capture, RGA preprocessing, inference, and display are pipelined.

In the single-context loop I was seeing ~31 FPS. With one context per NPU core and pipelined frame handling, it reaches the camera ceiling, around 42–46 FPS depending on the mode. So in this particular real-time streaming setup, parallel contexts/core masks were the practical way to saturate the hardware without adding batch latency.