AutoIndex with Sam O'Nuallain - Weaviate Podcast #143!

About this episode

Sam O'Nuallain joins the Weaviate Podcast to discuss AutoIndex, research from UMass Amherst and Databricks on learning representation programs for retrieval. Instead of tuning the retriever or re-ranker, AutoIndex asks how the data itself should be represented: a two-agent system, an analysis agent and a code agent, writes and refines Python programs that chunk, enrich, and reorganize a corpus to optimize downstream metrics like Recall and nDCG.The conversation opens with why indexing is such a natural target for code optimization. Frontier LLMs are exceptional at writing code, a representation program applies cheaply across an entire corpus without passing every document through an LLM, and code is verifiable. Every hypothesis the code agent proposes is gated against a validation set before it is accepted. From there, the discussion dives into the optimization signal. The analysis agent uses tools to read documents, query the retriever, and inspect where gold documents rank, turning a bare score like "recall went up" into rich natural language feedback about why a representation is failing, echoing ideas like GEPA's reflective metrics and the value of small-margin positives for training re-rankers.That leads into why BM25 pairs so well with agents: its lexical transparency makes failures easy to diagnose, illustrated by case studies from the CRUMB benchmark. This includes LaTeX formatting errors sinking Stack Overflow retrieval and Tip-of-the-Tongue movie search, where AutoIndex learned to repeat plots to up-weight terms and expand documents with synonym dictionaries. The conversation moves through connections to document enrichment methods like Anthropic's contextual retrieval, doc2query, and EnrichIndex, generalizing representation programs to text-to-SQL schemas and data lakehouses. The podcast concludes by discussing Sam's lessons transitioning from research to production AI engineering: loop engineering, QA, and evals. It lands on the directions that excite Sam most: harness design, continual learning, and memory as a retrieval problem, squeezing more out of the models we already have without touching the weights. Chapters 0:24 An Overview of AutoIndex 4:04 Retrieval Indexing as Code Optimization 8:47 Optimizing Chunking and Database Schemas 15:57 Feedback for Search Optimization 23:57 Future Directions for AutoIndex 27:34 Document Enrichment for RAG 36:32 Web Search vs. Databases 40:40 AI Engineering 49:30 Exciting Directions for AI