Back

Speaker "Nikunj Bajaj" Details Back

 

Topic

LLM economics : The cost of leveraging Large Language Models

Abstract

Most of us are using LLMs and some of us are getting to the point where LLMs are going to production. Honeymoon phase is going to get over soon and practical realities like cost & maintainability are going to become mainstream. However, the cost of running LLMs is not well understood or often not put in perspective. In this talk we will dive deep into what type of costs are involved in building LLM based apps. How do these compare when you run RAG vs Fine tuning, what happens when you use Open Source vs Commercial LLMs? Spoiler- If you wanted to summarize the entire Wikipedia to half its size using GPT-4 8k context window, it would cost a whopping $360K! While there is ample information available online about LLMs and their performances, our session focuses solely on the math-intensive aspect of understanding LLM pricing. We delve into the cost analysis of running popular LLMs, comparing their pricing for a specific task of summarizing Wikipedia. Moreover, we offer valuable knowledge on the levers of pricing in OpenAI and 3rd-party APIs, as well as the costs associated with self-hosted models and fine-tuning. Additionally, we introduce TrueFoundry's innovative solutions, such as a compression API for reducing OpenAI costs and simplified deployment of open-source LLMs through our Model Catalogue and Drop-in APIs. By attending our session, participants gain unique and actionable insights that cannot be easily found online.
Who is this presentation for?
This presentation is for anyone and everyone who is looking to use LLMs either on the enterprise level or for personal use.
Prerequisite knowledge:
None
What you'll learn?
Interact with brilliant minds out there and foster new connections

Profile

Nikunj is the co-founder and CEO of TrueFoundry, a platform empowering ML developers to deploy and optimize Language Models. Prior to this role, he served as a Tech Lead for Conversational AI at Meta, where he spearheaded the development of proactive virtual assistants. His team also put Meta's first deep learning model on-device. Nikunj also led the Machine Learning team at Reflektion, where he built an AI platform to enhance search and recommendations for over 600 million users across numerous eCommerce websites. Fun Fact about Nikunj? He learnt scuba diving and swimming in parallel. His instructor laughed at him saying- "You don't know how to swim? And you thought it would be just fine to jump in the middle of Pacific with 70 lb gear on your back"