Deep Dive
Understanding Cost Metrics
Mandeep explains the common mistake of focusing on cost per token instead of cost per task. He shares a personal story about a chatbot failing to resolve his flight issue, highlighting the need for human intervention which increases costs. He advises businesses to measure efficiency by the overall cost to complete a task, including human resources.
Choosing the Right Model
The discussion shifts to selecting the appropriate AI model. Mandeep suggests defining the task and expected accuracy before choosing a model. He introduces the Pareto curve to balance accuracy and cost. He recommends starting with simpler models like Luna for basic tasks and using more complex models like Astra for advanced needs.
Cost Optimization Strategies
Sapto outlines four strategies to cut AI costs: prompt caching, programmatic tool calling, reasoning effort, and batch API processing. He emphasizes prompt caching, which can drastically reduce costs for repetitive tasks. Programmatic tool calling is highlighted for its ability to reduce token usage significantly without sacrificing quality.
Leveraging Prompt Caching
Sapto dives deeper into prompt caching, advising developers to keep prompts consistent to maximize cache hits. He shares a case study of Blitzy, a customer who improved cache reuse and reduced costs by adopting explicit breakpoints in their prompts, demonstrating significant savings and efficiency gains.