Summary:

In this podcast, I talked to Ashish Shubham (VP of Engineering), who's been at ThoughtSpot for 10 years, about AI agents in enterprise analytics. ThoughtSpot started as a search-based analytics company trying to make data accessible to regular business users. In 2019, they tried building natural language interfaces using BERT, but only hit about 50% accuracy. So they shelved the project.

When ChatGPT came out, ThoughtSpot had to act, so Ashish walked me through how they pivoted: they built a 25-30 person team, decided to use prompting instead of fine-tuning, and leveraged their existing semantic data modeling layer to get accuracy into the high 90s. We got into the technical evolution from monolithic systems to agent architectures with tools, how they went from manual human judges to using LLMs to evaluate their outputs, and how enterprise security requirements shaped what they built.

We also talked about how software engineering is changing. Ashish said 50-60% of his code is AI-generated now, and he thinks system design is becoming the most important skill, even for junior engineers.

Chapters:

0:00 Intro and Ashish's journey to ThoughtSpot from GoDaddy 0:13 ThoughtSpot's mission to democratize data analytics for business users 1:26 Early search-based analytics before natural language processing 2:36 ThoughtSpot vs Tableau and the promise of self-service analytics 4:40 The analyst bottleneck problem and how ThoughtSpot aimed to solve it 5:49 Early technical challenges with in-memory databases and data migration 8:11 Semantic data models, joins, and creating abstraction layers for users 11:39 Who builds the data models and the role of analysts 12:22 Pre-LLM natural language processing using BERT and word2vec in 2018-2019 14:43 The accuracy problem and ambiguity in translating user queries 16:58 Trust challenges and why the early NLP product never became core 19:59 Competition with Tableau, Looker, and Power BI 22:44 How analyst roles changed with self-service analytics tools 25:30 The ChatGPT moment and pivoting to LLM-powered natural language 27:48 Early prompt engineering days and generating SQL with LLMs 31:09 Training vs prompting debate and why fine-tuning was eventually abandoned 34:28 Organizational changes and building the NLS team 37:16 Coaching systems for company-specific terminology vs training models 39:02 Evolution of evaluation methods from human judges to LLM-as-judge 43:23 Moving to LangFuse and GCP for agent infrastructure 46:29 How LLM context windows and capabilities evolved their product 50:07 From 30-column limits to agentic systems with 90%+ accuracy 52:52 RAG, column selection, and using proprietary data indexes 54:59 Multi-model support and enterprise data security concerns 59:14 How AI has changed Ashish's personal engineering workflow 1:02:42 Impact of AI on the broader engineering organization 1:04:15 Measuring AI productivity and the challenge of metrics 1:07:26 50-60% AI-generated code and the changing nature of coding 1:09:18 System design skills becoming more important than coding 1:13:00 Junior engineers doing senior-level work and interview changes 1:14:37 Customer conversations about Gen AI adoption across industries 1:17:26 The MIT report on 95% agent failures and why it misses the point 1:22:12 Agent architecture with LangGraph vs Google ADK and building internal agent platform 1:24:26 Where value lies in the next two years: tools, skills, and optimization 1:28:05 Startup opportunities in making AI accessible to non-technical users 1:29:26 Closing remarks