About Lex GPT

Lex GPT is an innovative application that brings AI-powered search capabilities to the Lex Fridman podcast. It serves as a testing ground for exploring the functionalities of Langchain. The application is built on a dataset that includes transcriptions of the first 325 episodes of the podcast, scraped and transcribed using Whisper. The remaining episodes up to 365 are also transcribed with Whisper. The transcribed data is then split and embedded with Langchain.

Features of Lex GPT

  1. AI-Powered Search: The application uses Langchain VectorDBQAChain to embed user queries, perform similarity searches on Pinecone embeddings, and synthesize answers from relevant chunks using GPT 3.5.
  2. Display of Source Documents: Relevant chunks with metadata are displayed as source documents, building on the excellent UI from another project, “wait-but-why-gpt”.
  3. Deployment on fly.io: The application that supports streaming is deployed on fly.io because Vercel requires edge functions for streaming. The team is working on getting edge functions working with Langchain. In the meantime, the more performant app can be accessed at lex-gpt.fly.dev.