Initially, the architecture was fairly typical:
User Question │ ▼ Vector Search │ ▼ Top K Chunks │ ▼ LLM
It worked well while the knowledge base was small.
As more documents were added, I started seeing a few recurring problems:
More irrelevant chunks being retrieved.
Larger prompts and increasing token costs.
Multiple documents discussing the same topic competing with each other.
Vector search returning semantically similar chunks from documents that weren't actually the best source of truth.
I realized the problem wasn't vector search itself.
It was deciding what should be searched before semantic retrieval even began.
Instead of treating every document equally, I separated the system into two independent stages.
Ingestion
During document upload, every document is processed once.
The pipeline extracts structured metadata including:
document type
business role
departments
topics
planner summary
retrieval keywords
authority score
importance score
answerable questions
That information is stored in a SQL registry, while document chunks and embeddings are stored separately in a vector database.
Document │ ▼ Metadata Extraction │ ├────────► SQL Registry │ └────────► Chunking + Embeddings │ ▼ Vector Store
Query Time
Instead of querying the vector database immediately, the retrieval flow became:
User Question │ ▼ Intent Analysis │ ▼ Registry Ranking │ ▼ Retrieval Planner │ ▼ Selected Documents │ ▼ Vector Search │ ▼ Context Assembly │ ▼ LLM
The registry acts as a lightweight ranking layer.
Rather than searching every document, it produces a ranked candidate set based on signals such as:
authority
importance
approval state
departments
document role
planner summary
retrieval keywords
topic overlap
The planner then decides which documents should actually participate in vector retrieval.
The vector database never searches the entire workspace anymore.
Only the planner-selected documents.
A few other changes made a noticeable difference:
similarity thresholding before accepting chunks
duplicate chunk removal
token budgeting before generation
dynamic chunk limits based on query type
ranking retrieved chunks before assembling context
One interesting observation was that improving retrieval often had a larger impact on answer quality than changing the generation model.
I'm curious whether others have moved beyond "vector search first" architectures.
If you've experimented with retrieval planning, metadata-driven routing, or hybrid retrieval systems, I'd be interested in hearing what worked and what didn't.