What is the go to "production" stack for something like this nowadays? Is Stan dead? Do you do HMC or approximations with e.g. Pyro?

In the technical notes I sort of laid out our model graph on the document branch. We also have a topic branch that is also structured hierarchically: Uniform Prior → High Level Topic Word → Granular Topics → Document Lever Variation in Topics. We just directly visualize that hierarchical representation in the sunburst.

The low level model graph is all written in C and exports granular annotations of the model graph. We use the model output to annotate the original text data. We do some work to store these hierarchical results in a SQL queryable format in DuckDB.

What's cool about this process is it's all annotation based. You can query data at the topic level, analyze topics and sql, and at any point pull up the exact excerpts to which the high level data refers.

Curious what you've been using it to search for?

The doc also explains the UX issue of a simple sunburst graph, thus using a tiered sun burst graph.

Our tool works a little different than LLM style tools. We are doing a bulk search — for academic search, ~1000 papers — and then training a hierarchical Bayesian model to organize the results. Once the model trains, it provides a visual representation of the high level themes that you can then use to explore the results.

The trade off is we are willing to lower the relevance filter to enable a broad set of exploration.

We do offer an api layer (the website is a light layer above this) over the low level statistics code focused on making it super easy to apply to language data if you are interested in playing around with it: https://docs.sturdystatistics.com