The basic idea is to take a recorded long gameplay session and automatically turn it into structured gameplay information and highlights. Bridging the performance analysis gap between match stats (too short) & lifetime stats (too long).
The current pipeline is:
Gameplay video → YOLO detection → event tracking → OCR/context filtering → event timestamps → clips → highlight compilation → session statistics
For example, I can fine-tune a YOLO model to recognize game-specific HUD elements and then configure the pipeline to interpret those detections as events such as kills, deaths, or medals.
One thing I wanted to avoid was making the entire system game-specific. Supporting another game should mostly require a new detector/model and configuration rather than rewriting the whole pipeline. And currently, I am working on that.
I also use OCR for states that shouldn't be counted as gameplay events. For example, in Call of Duty games a kill indicator can appear during a "KillCam" or "Spectating" state, so the OCR layer can be used to filter those cases.
The system can then produce clips around detected events and compile them into highlight reels, including vertical versions for Shorts/TikTok-style content.
Recently, I have also been experimenting with using a lightweight Video LLM after event detection. Instead of sending the entire multi-hour gameplay session to the Video LLM model, I take the extracted short clips around an interesting event and ask the model to explain what happened. The goal is to eventually turn this into a lightweight conversational gameplay coach.
The project currently runs locally on NVIDIA GPUs. Have experimented with an old GTX1650 4GB VRAM laptop and it works fine but slow ofcourse.
Finally, I know that real-time event detection should be the way to go and this was my original goal, but I chose offline processing because I didn't want computer vision inference to interfere with the player's actual game performance.
I'd also be interested in hearing how others would approach making the detector/inference layer fast enough for real-time gameplay without causing noticeable GPU/CPU overhead. Keeping in mind, I input 1080p frames into the model.
The project is still evolving, so technical criticism and suggestions are very welcome.
Thanks.