Leon Zhao on AI product evaluation, AI-native workflows, causal inference, and data science. https://leonz.site/ What sustained agentic work taught me about exit criteria, evaluation, context, and review - The least useful way to evaluate a coding agent is to ask whether it produced an answer. Answers are easy. Completed work is harder. A plausible patch can still misunderstand the system. A passing test can miss the behavior that matters. A confident explanation can rest on a capability the... Wed, 09 Sep 2026 09:00:00 -0700 https://leonz.site/2026-09-17-claude-code-is-a-workflow-not-a-chat-box/ https://leonz.site/2026-09-17-claude-code-is-a-workflow-not-a-chat-box/ A practical measurement stack for quality, cost, latency, and product value - AI products are often evaluated as if the central question were simple: Did the model produce a good answer? That question matters, but it is rarely enough. A response can be accurate and still fail to help someone complete a task. It can be useful but arrive too slowly. It... Tue, 14 Jul 2026 09:00:00 -0700 https://leonz.site/2026-09-17-evaluating-ai-products-beyond-accuracy/ https://leonz.site/2026-09-17-evaluating-ai-products-beyond-accuracy/ Why useful AI spreads through pain discovery, ownership, contribution paths, and habit formation - I have seen a recurring pattern in how organizations approach AI. A team discovers a new model or tool. Someone demonstrates what it can do. A workshop is scheduled. Documentation is published. People are encouraged to experiment. Then, a few weeks later, the excitement can be gone while the work... Fri, 29 May 2026 09:00:00 -0700 https://leonz.site/2026-09-17-the-missing-layer-in-ai-adoption-is-workflow-design/ https://leonz.site/2026-09-17-the-missing-layer-in-ai-adoption-is-workflow-design/ How to validate signals, separate mechanisms from artifacts, and make uncertainty useful - I have learned to distrust models that look impressive too quickly. A strong coefficient, a polished dashboard, or a compelling narrative can create the feeling that we understand a system. Often, we understand only one slice of it. The result may be driven by a shared time trend, a mechanical... Wed, 18 Mar 2026 09:00:00 -0700 https://leonz.site/2026-09-17-trustworthy-decision-systems-begin-with-model-skepticism/ https://leonz.site/2026-09-17-trustworthy-decision-systems-begin-with-model-skepticism/ Retrieval optimization, prompt engineering, and A/B testing for enterprise LLMs - When you build an AI product on top of a third-party LLM, you don’t control the model weights. You control everything around it. That constraint turns out to be more of an advantage than a limitation. After launching Slackbot, a context-aware conversational AI built into Slack, early beta feedback made... Sat, 01 Mar 2025 00:00:00 -0800 https://leonz.site/2025-03-01-reducing-hallucinations-enterprise-llms/ https://leonz.site/2025-03-01-reducing-hallucinations-enterprise-llms/ Using quasi-experimental methods to evaluate feature value at scale - If you ship fast enough, you’ll eventually outrun your ability to measure. That’s exactly what happened with Slack AI. Over the course of a year, we launched AI features at roughly one per month. The goal was clear: help users navigate information overload. But because Slack users are clustered in... Wed, 15 Jan 2025 00:00:00 -0800 https://leonz.site/2025-01-15-measuring-ai-impact-without-ab-tests/ https://leonz.site/2025-01-15-measuring-ai-impact-without-ab-tests/ Implementing commonly used models from scratch - This is an attempt to build a sentiment classifier for movie reviews. We’ll create a binary linear classifier that reads reviews, like you’d see on Rotten Tomatoes, and predicts whether the sentiment is positive or negative. The goal is to classify given movie reviews as “positive” or “negative” using a... Mon, 01 May 2023 00:00:00 -0700 https://leonz.site/2023-05-01-sentiment_classification/ https://leonz.site/2023-05-01-sentiment_classification/