Thank you :-)

We use mocking to replace actual LLM calls when testing for the correctness of the ThalamusDB code. In terms of performance benchmarking, we ran quite a few experiments measuring time, costs (fees for LLM calls), and result accuracy. The latter one is the hardest to evaluate since we need to compare the ThalamusDB results to the ground truth. Often, we used data sets from Kaggle that come with manual labels (e.g., camera trap pictures labeled with the animal species, then we can get ground truth for test queries that count the number of pictures showing specific animals).

Well, it definitely goes beyond a traditional DBMS, but yes :-) If processing the same amount of data via pure SQL versus SQL with LLM calls, it will be slower and more expensive when using LLMs. Note that 600s is just the default timeout, though. It's typically much faster (and you can set the timeout to whatever you like; ThalamusDB will return the best result approximation it can find until the timeout). More details in the documentation: https://itrummer.github.io/thalamusdb/thalamusdb.html

We use a vector db (Qdrant) to store embeddings of images and text and built a search UI atop it.

LlamaIndex relies heavily on RAG-style approaches, e.g., we're using items whose embedding vectors are close to the embedding vectors of the question (what you describe). RAG-style approaches work great if the answer depends only on a small part of the data, e.g., if the right answer can be extracted from a few top-N documents.

It's less applicable if the answer cannot be extracted from a small data subset. E.g., you want to count the number of pictures showing red cars in your database (rather than retrieving a few pictures of red cars). Or, let's say you want to tag beach holiday pictures with all the people who appear in them. That's another scenario where you cannot easily work with RAG. ThalamusDB supports such scenarios, e.g., you could use the query below in ThalamusDB:

SELECT H.pic FROM HolidayPictures H, ProfilePictures P as Tag WHERE NLFILTER(H.pic, 'this is a picture of the beach') AND NLJOIN(H.pic, P.pic, 'the same person appears in both pictures');

ThalamusDB handles scenarios where the LLM has to look at large data sets and uses a few techniques to make that more efficient. E.g., see here (https://arxiv.org/abs/2510.08489) for the implementation of the semantic join algorithm.

A few other things to consider:

1) ThalamusDB supports SQL with semantic operators. Lay users may prefer the natural language query interfaces offered by other frameworks. But people who are familiar with SQL might prefer writing SQL-style queries for maximum precision.

2) ThalamusDB offers various ways to restrict the per-query processing overheads, e.g., time and token limits. If the limit is reached, it actually returns a partial result (e.g., lower and upper bounds for query aggregates, subsets of result rows ...). Other frameworks do not return anything useful if query processing is interrupted before it's complete.

:-) Actually, you can also use models of other providers (e.g., Google's Gemini models). You just have to set your access key by the corresponding provider and configure the models you'd like to use in this file: https://github.com/itrummer/thalamusdb/blob/main/config/mode...