So I am quite amazed with the innovation in DeepSeek-OCR model! I wanted to break it apart and try it out myself, so I asked myself - what if I extract the encoder to fit other existing VLMs?
I didn't have any expectations and was doing this just for fun cos why not? Moving on, after vibe scripting with the encoder, I tried to patch this with Qwen3-VLM 2B. Due to difference in input dimensions of Qwen and the DeepSeek encoder, I pretrained a custom adapter to fit this piece of puzzle.
Long story short - I noticed some performance gains in my experimental synthetic dataset as well as Longbench V2. You can check the project out and try it -
I have added the training and test scripts in the repo.
In a miniscule test run of 50 cases of LongBench V2 benchmark - I noticed that the custom optical encoder with compressed visual tokens performed slightly better than the original Qwen encoder. It could be that 2B model is really weak for this benchmark.
I could be wrong in my approach so I don't want to hype this too much, and I am more curious to find out if this is scalable beyond 2B? I'm GPU poor with a 12 GB 5070 so I would love it if someone gives this a shot and try to take it further? Hope this helps!
volkopat2•3h ago
So I am quite amazed with the innovation in DeepSeek-OCR model! I wanted to break it apart and try it out myself, so I asked myself - what if I extract the encoder to fit other existing VLMs?
https://huggingface.co/Volkopat/DeepSeek-DeepEncoder
I didn't have any expectations and was doing this just for fun cos why not? Moving on, after vibe scripting with the encoder, I tried to patch this with Qwen3-VLM 2B. Due to difference in input dimensions of Qwen and the DeepSeek encoder, I pretrained a custom adapter to fit this piece of puzzle.
https://huggingface.co/Volkopat/Qwen-VLM-Optical-Encoder
Long story short - I noticed some performance gains in my experimental synthetic dataset as well as Longbench V2. You can check the project out and try it -
https://github.com/Volkopat/VLM-Optical-Encoder
I have added the training and test scripts in the repo.
In a miniscule test run of 50 cases of LongBench V2 benchmark - I noticed that the custom optical encoder with compressed visual tokens performed slightly better than the original Qwen encoder. It could be that 2B model is really weak for this benchmark.
I could be wrong in my approach so I don't want to hype this too much, and I am more curious to find out if this is scalable beyond 2B? I'm GPU poor with a 12 GB 5070 so I would love it if someone gives this a shot and try to take it further? Hope this helps!