Pratik Bhavsar's Blog Posts

Pratik Bhavsar is an AI Engineer at Splunk focusing on agent evaluation, reliability, and observability. He has spent a decade building across the full AI stack, from training transformers, building semantic search, and shipping ML systems to designing agentic architectures.

He joined Cisco through the acquisition of Galileo, where he led open-source evaluations and developer relations. He built the Agent Leaderboard, an open benchmark measuring AI agent performance on real-world tasks, and the Hallucination Index, a systematic study of factual reliability across foundation models.

He is the author of five technical books covering eval engineering, agentic systems, RAG, multi-agent architectures, and LLM-as-a-Judge methodologies. Prior to Galileo, Pratik was a founding NLP Scientist at Enterpret and Senior Data Scientist at Morningstar. Pratik holds an M.Tech from IIT Bombay and loves to share his thoughts on Substack.

Artificial Intelligence

8 Minute Read

Token Meter: A Live Cost Meter for Your Coding Agents

See what your coding agents cost before the bill arrives. Token Meter tracks Claude Code, Codex, and Cursor usage live — free, open source, and 100% local.

Learn

5 Minute Read

Character Error Rate (CER): Meaning, Formula, and How to Use It in 2026

Character error rate (CER) is a an AI accuracy metric. Learn how to ntegrate CER into a quality stack for transcription, OCR, and extraction workflows.

Artificial Intelligence

8 minute read

How to Build an Agent Evaluation Framework for Production AI

Learn how to evaluate agents for production AI with this framework approach.

Artificial Intelligence

7 MINUTE READ

Evaluating AI Agents on Tool Calling and Planning

Evaluate AI agent performance on tool calling and planning by prioritizing consistency metrics, multi-turn state handling, and domain-specific assertions over public benchmark scores

Artificial Intelligence

10 Minute Read

4 Key RAG Metrics to Improve Retrieval and Generation

Use these four key metrics—Context Relevance, Chunk Relevance, Context Adherence, and Completeness—to diagnose whether your RAG system is failing at retrieval or generation.

Artificial Intelligence

9 minute read

How To Choose a Vector Database Architecture

Choose the right vector database for your RAG systems. Compare database index structures, quantization methods, and deployment options to select the most suitable architecture.

Artificial Intelligence

15 Minute Read

Context Engineering for AI Agents in Production: Architecture, Failure Modes, and Metrics

Diagnose and resolve agent context failures by identifying common patterns, applying architectural strategies, and implementing a robust production observability loop.

Artificial Intelligence

11 Minute Read

Scaling Laws of AI Tokenomics

Tokenomics isn’t about reducing tokens – it’s about understanding the marginal return of inference.

Artificial Intelligence

10 Minute Read

How To Continuously Improve Your LangGraph Multi-Agent System: A Tutorial

Improve the performance and reliability of your LangGraph multi-agent with observability to trace agent decisions, isolate failure patterns, and optimize workflows.

Artificial Intelligence

8 Minute Read

How to Reduce Agent Cost by Model Routing

Learn how model routing helps AI agents reduce token costs by using the right model for each task while maintaining performance.

Artificial Intelligence

8 Minute Read

What is Agent Tokenomics?

Agent tokenomics is the discipline of classifying and governing how tokens move through an agent system.