OP here.
I wrote this implementation to deeply understand the mechanics behind HNSW (layers, entry points, neighbor selection) without relying on external libraries.
While PHP isn't the typical choice for vector search engines, I found it surprisingly capable for this use case, especially with JIT enabled on PHP 8.x. It serves as a drop-in solution for PHP monoliths that need semantic search features without adding the complexity of a separate service like Qdrant or Pinecone.
If you want to jump straight to the code, the open-source repo is here: https://github.com/centamiv/vektor
Happy to answer any questions about the implementation details!
hu3•1mo ago
Great writeup. Thanks for talking the time to organise and share.
It's tempting to use this in projects that use PHP.
Is it useable with a corpus of like 1.000 3kb markdown files? And 10.000 files?
Can I also index PHP files so that searches include function and class names? Perhaps comments?
How much ram and disk memory we would be talking about?
And the speed?
My first goal would to index a PHP project and its documentation so that an LLM agent could perform semantic search using my MCP tool.
centamiv•1mo ago
I tested it myself with 1k documents (about 1.5M vectors) and performance is solid (a few milliseconds per search). I haven't run more aggressive benchmarks yet.
Since it only stores the vectors, the actual size of the Markdown document is irrelevant; you just need to handle the embedding and chunking phases carefully (you can use a parser to extract code snippets).
RAM isn't an issue because I aim for random data access as much as possible. This avoids saturating PHP, since it wasn't exactly built for this kind of workload.
I'm glad you found the article and repo useful! If you use it and run into any problems, feel free to open an issue on GitHub.
Random09•1mo ago
The only small thing you forgot to mention - it requires use of AI. Open Ai to be specific. I've got baited.
centamiv•1mo ago
Apologies if it felt that way! I used OpenAI in the examples just because it's the quickest 'Hello World' for embeddings right now, but the library itself is completely agnostic.
HNSW is just the indexing algorithm. It doesn't care where the vectors come from. You can generate them using Ollama (locally) HuggingFace, Gemini...
As long as you feed it an array of floats, it will index it. The dependency on OpenAI is purely in the example code, not in the engine logic.
devmor•1mo ago
I think you'd get a lot more people interested in trying your project out if you included steps on how to generate vectors for the search as a document.
I love PHP, but I will realistically admit that most people interested in using PHP probably don't have the experience to know how to do such a thing offhand.
centamiv•1mo ago
You are absolutely right. I will update the README with some examples, thanks for the feedback!
lukan•1mo ago
Thanks a lot, I liked the fantasy based examples to explain the concept.
Programming is chanting magic incarnations and spells after all.
(And fighting against evil spirits and demons)
centamiv•1mo ago
I'm really glad you liked the article! Thanks so much for reading the previous one too, I really appreciate it.
hilti•1mo ago
Great article!
I also read your other post and love it! This is exactly my thinking: Locality of Behavior (LoB)
Thanks for checking out the other posts too! I wasn't familiar with the term 'Locality of Behavior' until recently, but it perfectly captures what I strive for: readability and simplicity.
fithisux•1mo ago
It makes perfect sense to implement it in a high level language that allows understandability.
Very good contribution.
centamiv•1mo ago
Thank you! That was exactly the goal. Modern PHP turned out to be surprisingly expressive for this kind of 'executable pseudocode'. Glad you appreciated it!
rvnx•1mo ago
Cool blog post, smart guy, very thoughtful and not a copy-paste of Python code like 99% of folks. Nice to see
centamiv•1mo ago
hu3•1mo ago
It's tempting to use this in projects that use PHP.
Is it useable with a corpus of like 1.000 3kb markdown files? And 10.000 files?
Can I also index PHP files so that searches include function and class names? Perhaps comments?
How much ram and disk memory we would be talking about?
And the speed?
My first goal would to index a PHP project and its documentation so that an LLM agent could perform semantic search using my MCP tool.
centamiv•1mo ago
Since it only stores the vectors, the actual size of the Markdown document is irrelevant; you just need to handle the embedding and chunking phases carefully (you can use a parser to extract code snippets).
RAM isn't an issue because I aim for random data access as much as possible. This avoids saturating PHP, since it wasn't exactly built for this kind of workload.
I'm glad you found the article and repo useful! If you use it and run into any problems, feel free to open an issue on GitHub.
Random09•1mo ago
centamiv•1mo ago
HNSW is just the indexing algorithm. It doesn't care where the vectors come from. You can generate them using Ollama (locally) HuggingFace, Gemini...
As long as you feed it an array of floats, it will index it. The dependency on OpenAI is purely in the example code, not in the engine logic.
devmor•1mo ago
I love PHP, but I will realistically admit that most people interested in using PHP probably don't have the experience to know how to do such a thing offhand.
centamiv•1mo ago
lukan•1mo ago
Programming is chanting magic incarnations and spells after all. (And fighting against evil spirits and demons)
centamiv•1mo ago
hilti•1mo ago
Never heard this term before, but I like it.
https://centamori.com/index.php?slug=basics-of-web-developme...
centamiv•1mo ago