I learned early that winning that standoff had nothing to do with being right. It had everything to do with speaking the right language. IT is optimizing for zero-surprise bills and zero data leaks. If you want access, you have to make their job easier, not harder. This article is the playbook I refined over seven years of doing exactly that.

IT Always Defaults to No

Before you write a request that gets approved, understand why no is the default. It’s risk aversion driven by three factors: compliance, cost, and liability.

Cloud cost exposure. A single analyst running SELECT * FROM large_table on a dashboard can double a cloud data warehouse bill overnight. A virtual warehouse consumes Snowflake credits while it runs. Credit limits don’t cut off spending instantly. When a threshold is reached, the assigned warehouse may continue running for a period before it is suspended, consuming additional credits in the process. This is true even when the most aggressive suspension setting is used. IT’s budget is on the line, and even well-intentioned queries can cause cost spikes when analysts don’t understand consumption-based pricing.

Identity and Access Management attack surface. Every additional person with a direct database connection string is a potential entry point. Unauthorized disclosure of credentials is among the leading causes of data incidents across enterprise environments. The principle of least privilege, which governs how IT controls access, means database access should be granted only to the degree necessary to accomplish a specific task. Access is meant to be the exception, not the baseline.

Liability and regulatory exposure. If you export sensitive data to an unencrypted Excel file on your laptop and that laptop goes missing, regulators can fine the company. Cybersecurity frameworks treat the loss of control over sensitive data as one of the most serious risks a company can face, and often one that must be disclosed. Storing sensitive data on laptops in unencrypted Excel files, without any record of who accessed it or when, is exactly the kind of control failure that creates regulatory and legal exposure. Excel sprawl creates audit nightmares that governed database access prevents.

What read-only access means to IT. Analysts say read-only and think ‘I can’t delete rows.’ IT hears read-only and thinks about all the things that become possible with that access: pulling down the entire customer database, running queries so heavy they slow down core business systems, or viewing compensation data and confidential deal information without authorization.

When to accept no versus push back. Accept no if IT offers a curated view, reporting replica, or sandbox environment. These signal a mature data organization that balances security with analyst access. If IT’s proposed alternative is to manually pull data into Excel, push back. Spreadsheets saved on local computers with no access controls, no version history, and no audit trail are a far bigger security and compliance problem than a governed database connection. That conversation needs to go up the chain.

If I’ve gotten your attention so far, what follows is the part that took seven years to learn. The templates, the compliance documentation, the risk mitigation proposals, the escalation scripts, and the three case studies that show how this all plays out. It’s all below.

Read more

The beta is finally ready for macOS and Linux, and I’d love for you to test the architecture. If you’ve ever felt paralyzed by a project that felt too big, your feedback is exactly what I need to refine the system. You can try it out and share your experience through our survey at the links below.

DOWNLOAD THE ASAURA AI BETA

TAKE THE ANONYMOUS USER SURVEY

I’ve been documenting the build over a 100 Days of Building AI video marathon. You can check that out on Substack notes or LinkedIn. Today is Day 91.

I’ve also been writing technical deep dives about the Asaura AI build on my other Substack, Between Thinking and Doing. Feel free to subscribe to that as well, or connect with me on LinkedIn.

Now that the main build is stable, I’ve got the space to get back into the rhythm of writing here every week.

Today’s article covers BigQuery features I've found myself teaching constantly over the past year. I share at least three of these with every data team I meet. These features solve problems people have worked around for years.

Part of what makes them so useful is that SQL has a structural quirk that we all know too well: you write clauses in a fixed order that doesn’t match the order they actually execute. BigQuery’s GoogleSQL dialect has been shipping fixes for this. The five features below are the ones I keep coming back to.

Read more

Why Ad-Hoc Prompting Breaks at Scale

Ad-hoc prompting works fine for prototypes. You’re iterating fast, the model’s responses are good enough, and the cost is trivial. When a prompt is serving real users under real SLAs with real billing attached, the economics look very different from a prototype.

A prompt is a dependency, just like a library version or a database schema. When it changes, outputs change. When outputs change, downstream parsers, classifiers, and user interfaces can break. Teams that don’t version control prompts lose institutional knowledge, can’t reproduce bugs, and have no rollback path.

The compounding factor is model updates. When using AI via third-party APIs, the LLMs behind those APIs can unexpectedly change. Like traditional ML models, LLMs can be refreshed or tuned without a significant version bump, meaning the model’s performance on your set of prompts can change without any notice. Without a versioning system, you can’t determine whether a degradation came from a prompt change or a model change, and that ambiguity is expensive.

Prompts Drift Like Code

Software engineers have spent decades building systems to manage code drift: version control, branching strategies, code review, CI/CD. Prompts need the same discipline, and most teams apply almost none of it.

There’s an unavoidable tension between keeping prompts close to the code versus an environment that non-technical stakeholders can access. Leaving it unresolved gets more expensive as the team grows.

Prompt management tools from observability platforms like Arize, Braintrust, and LangSmith offer Git-like version control for prompts and allow rollback if changes reduce quality, but tools alone won’t help if the discipline isn’t there. Prompt management tools are inherently limiting because they can’t easily execute your application’s code. Even when they can, there’s often significant indirection involved, making it difficult to test prompts with your system’s full capabilities, including tools, RAG, and agents. Keeping prompts in the same repository as application code, with the same review gates, removes that indirection.

A prompt change is a logic change. It controls what your system does and how it responds, and it deserves the same review process as any other change to your codebase.

Subscribe now

Testing Prompts vs. Testing Code

Testing code is relatively well understood. You write unit tests, integration tests, and end-to-end tests. Give a traditional function the same input twice, and you’ll get the same output twice, which makes it straightforward to write tests that check for exact results.

Prompt engineering is about conditioning a probabilistic model to generate a desired output. Each additional instruction or piece of context steers the model’s generation in a particular direction. That probabilistic nature means you can’t assert exact outputs. Instead of checking for an exact answer, you check whether the response has the right shape: whether it’s valid JSON, whether the label the model chose is one you’d accept, and whether the thing it extracted actually appeared in the source text.

Structured evaluations tell you whether a prompt change made things better or worse before it reaches users. Without them, you’re iterating on feel, which works fine until something goes wrong in a way you didn’t anticipate and can’t reproduce.

Eyeballing is useful as a final check, but it doesn’t hold up at volume or across teams.

When Prompt Engineering Pays Off (and When It Doesn’t)

Before committing engineering resources to a prompt infrastructure, it’s worth being honest about ROI.

Prompt engineering pays off when a model’s output goes directly into another system without a human reviewing it first, when the same prompt runs thousands of times a day, when each call carries a meaningful cost, or when output quality has a direct and measurable effect on what users experience. Without reliable evaluation, you can’t iterate, and without iteration, you can’t improve. Projects stall here more than anywhere else in the LLM lifecycle.

For tasks reviewed by humans before use, or where call volume is low enough that failures surface quickly, a lightweight log of prompt versions in a shared document may be sufficient. For anything in the first category, treating prompts as software assets is worth the investment.

In the next section, we get into the practical architecture: how to structure and version prompts, how to run A/B tests without exposing users to untested changes, how to build a test suite that runs automatically when a prompt changes, how to organise a prompt library across a growing team, how to track cost at the prompt level, how to detect when a deployed prompt starts degrading, and two failure patterns that illustrate what happens when these systems aren’t in place.

Read more

Inference doesn’t work like that. Inference is a subscription you didn’t fully price. Once a model hits production, inference consistently accounts for 60-80% of total LLM operational spend across a 12-month horizon. For teams shipping to real users at scale, that number trends toward the high end.

A team that spends $5K on training will often spend $40K-$50K running that model in year one. This happens consistently across production teams.

Inference Costs Catch Production Teams Off Guard

Training has a natural end state. You’re optimizing toward a metric, you hit it, and spending stops. Inference has no finish line. Every user request is a billing event, and billing events compound.

Three mechanisms drive the bulk of that spend:

KV cache memory consumption. When your model reads a prompt, it saves a record of every word it processes so it doesn’t have to re-read them for each new word it generates. That saved record is called the KV cache, and it sits in GPU memory. The longer the prompt, the more memory it occupies. For long prompts, 40-60% of your GPU’s available memory can go to KV cache alone.

Idle GPU time. The GPUs powering your model rent for $2-$8/hour, whether they’re processing requests or sitting idle. User traffic isn’t steady; it spikes during business hours and drops overnight, but you can’t spin up a new GPU instantly. Bringing one online takes between one and three minutes, which is too slow to respond to a traffic spike. So teams keep extra GPUs running at all times just in case, and pay for them around the clock.

Per-token economics. Every word sent to the model and every word it sends back costs money. Tokens are just the unit the model uses to measure text, roughly three-quarters of a word each. What teams often miss is how quickly prompt length adds up. A request that includes 2,000 words of retrieved documents, a 500-word system instruction, and a 200-word question costs three to four times as much as a tightly written 700-word equivalent. Bloated instructions, uncompressed retrieved text, and redundant examples all push costs up faster than usage alone would.

Standard Optimizations Don’t Finish the Job

The first instinct is usually to optimize at the model level: quantize to INT8, swap to a smaller model, apply distillation. These are legitimate tools, but they’re not a complete strategy.

Quantization can yield significant savings on its own (up to 50% at INT8 and higher at INT4), but model-routing or batching-only approaches typically plateau at 20-30%, and even quantization’s gains erode if retrieval context and serving efficiency aren’t addressed alongside it. Smaller models help when the workload is simple, but they degrade on complex instructions. Teams frequently apply one of these levers in isolation, see diminishing returns, and assume they’ve hit a ceiling.

They haven’t. The ceiling is generally much higher.

Inference cost reduction isn’t a one-dimensional problem. It spans caching strategy, batching behavior, model selection, serving architecture, and hardware configuration simultaneously. Pulling a single lever and treating it as a complete solution is why teams frequently plateau at 20-30% savings with most optimization approaches, whereas a systematic approach can achieve 60-70%.

How to Know Which Lever to Pull First

The rest of this article walks through five specific optimization levers and the signals that tell you which to prioritize for your workload type.

I also cover a full case study from my most recent client: they were spending $50K/year on inference, and we reduced that to $15K without affecting model quality. Every tradeoff gets named.

LLM Inference Optimization: Cutting Costs by 70% Without Sacrificing Quality

This content is for paid subscribers. If you’re reading this and haven’t upgraded yet, you can do so below.

Read more

This problem is everywhere. Data teams build beautiful visualizations that stakeholders glance at once, nod appreciatively, then ignore for every decision that matters. You see the same confession across analytics forums, Slack channels, and team retrospectives. The analytics industry has mastered the art of building. We’ve failed at measuring whether it changes anything.

Welcome back to The Data Letter! Here are some recent articles you may have missed:

• From Manual Export to Automated Pipeline: A Practical Playbook

• LLM Fine-Tuning on a Budget

• Deepseek for Data Analysis

Subscribe now

Vanity Metrics vs Decision Velocity

There’s a difference between dashboards that look right and dashboards that change behavior. Most teams optimize for the first. They measure views, shares, and executive compliments. These are vanity metrics dressed up as success signals.

What matters is decision velocity: how quickly can someone move from seeing data to taking action? If your stakeholder opens your dashboard, scrolls through twelve trend lines, then schedules a meeting to ‘discuss next steps,’ you’ve built decoration, not infrastructure.

Back to diagnosing the problem…

Kill This Dashboard Test

Here's how you diagnose it. Ask your stakeholders these questions:

1. What decision did you make because of this dashboard in the last two weeks?

If they pause or pivot to talking about how useful it ‘could be,’ you’re watching theater.

2. If I removed this chart tomorrow, what would break?

If nothing breaks, nothing mattered.

3. When this metric moves, what do you do differently?

If they can’t name a specific action, the metric is decorative.

Signs you’re building something nobody uses:

• Stakeholders request more charts instead of fewer

• They want ‘visibility’ but can’t define what changes when they have it

• They complement your color palette more than your insight clarity

Subscribe now

Building for Impact, Not Applause

Start with the decision, not the data. Before you touch Tableau, ask: ‘What’s the one thing this person needs to do differently?’ Then build backward from that action.

Strip everything else. Your stakeholder doesn’t need context, trends, and breakdowns. They need the number that triggers the action, and they need it unmissable.

Make the action as easy as the view. If someone sees ‘conversion rate dropped 12%’ and then opens Jira to create a ticket, you’ve failed. The button to create that ticket should be right there.

Test by removing features, not adding them. Show version A with ten charts. Show version B with two. Measure which one changes behavior faster.

Subscribe now

How to Build Dashboards That Drive Action

Ask yourself whether what you’re building will change anything.

Close the dashboard and watch what happens. If nobody’s behavior shifts, if no tickets get created, if no priorities change, you’ve answered your own question. Strip out everything that doesn’t trigger an action. Give stakeholders fewer charts and clearer next steps. Measure whether anyone does something different after seeing your work, not whether they say they like it.

Next up: a decision-first framework for building dashboards that stakeholders use to take action, complete with templates and a stakeholder questionnaire.

The first pattern uses Google Apps Script to automatically pull CSV attachments from Gmail and load them into a Google Sheet. The second uses Python and pandas to consolidate multiple CSV exports into a single clean output file on a schedule. Both have been tested and work.

The methods are practical enough for analysts automating their first workflow and straightforward enough for engineers who want a low-overhead solution they can hand off to a less technical teammate. Each section includes the code, setup instructions, and scheduling guidance.

All three files referenced in this article (both scripts and one sample CSV) are available in a secret gist for paid subscribers. The link is at the end.

Read more

This is the analyst-as-middleware problem, and it’s more common than most organizations admit. Someone with Python skills and SQL fluency spends their days downloading CSVs and managing attachment threads. No queries. No transformations. No insight generation. Just scheduled, manual data transport dressed up with an analytical job title.

Why Organizations Create Data Dead Ends

This doesn’t happen because companies are malicious. It happens because systems calcify. IT locks down database access for security reasons that made sense in 2011 and never got revisited. Legacy platforms don’t expose APIs or connect cleanly to modern tooling. Business units developed their own manual workflows years ago, and ‘how we’ve always done it’ carries more institutional weight than efficiency ever could.

The result is an informal architecture built from browser downloads, shared drives, and email attachments. Staying sharp is possible without system access. What it requires is intentional practice with the materials you already have.

Subscribe now

How Skill Atrophy Happens to Good Analysts

The cognitive cost isn’t obvious at first. You’re still technically working with data, but there’s a meaningful difference between analyzing data and transporting it.

When your job doesn’t require you to write queries, you stop writing them. Python loops you wrote from memory are starting to require documentation lookups. SQL syntax blurs at the edges. The joins you knew cold become joins you have to think about. Skills erode in proportion to how rarely they’re exercised, and manual export workflows exercise almost none of the skills that define analytical competence.

Six months in, you’re slower. A year in, you’re rusty. Two years in, you may start to believe the rusty version is the real one.

If your daily work has drifted away from the technical side, the previous issue is worth your time. LLM Fine-Tuning on a Budget walks through dataset preparation, how LoRA works, and how to evaluate a model when you don’t have labeled test data. It’s structured, technical, and built for people who want to stay capable.

Red Flags That Confirm You’re Trapped in Manual Export Work

Watch for these patterns in your own workflow. Your browser history is dominated by downloads, report portals, and file transfers. Shared drives function as your team’s data architecture, with folders named by date and maintained by convention rather than design. Email is your ETL pipeline, literally, attachments moving data between people because systems won’t talk to each other. You haven’t written a query against a live database in months. Your ‘analysis’ begins after someone else pulls the data and ends before anyone asks how it was prepared.

If more than two of those describe your week, you’re not doing data analysis. You’re doing data custody.

Subscribe now

How to Preserve Your Skills Without Changing Your Job

Getting out of this situation runs through your relationship with the data you already handle, not through a job search.

Local sandboxing means running every export you receive through a script before you manually touch it. Parse it with Python. Load it into a local database. Write the query that answers the question, even if you’re just going to paste the answer into a report anyway. The analysis muscle stays active because you’re actually using it.

Shadow analysis means building things no one asked for. You have the data. Model something. Build a dataset that answers a question your team hasn’t thought to ask yet. Don’t send it anywhere. Just build it. The practice is the point.

Document the waste. Track, in hours, how much time you spend manually transporting data each week. This isn’t venting. It’s evidence. Quantified inefficiency is the clearest argument for change.

Turning Manual Exports into Automated Pipelines

Getting out of this situation runs through your relationship with the data you already handle, not through a job search. Escaping manual export workflows doesn’t always mean landing somewhere better. More often, it means automating the situation you’re already in.

‘Automation Playbook for Trapped Analysts’ gets into that next. Specific tools and workflows. No direct database access required. Just the CSV in your inbox and a decision to stop moving it by hand.

Happy Sunday! See you next week.

Paid subscribers get the full archive. That includes the vector database implementation guide covering Pinecone, Weaviate, and Chroma, with cost formulas and query optimization patterns; the advanced model drift detection guide with diagnostic workflows and threshold tuning; and the fine-tuning issue guide with dataset templates and a cost-tracking spreadsheet. The Automation Playbook for Trapped Analysts will be paid. If automating your way out of manual export work is relevant to you, it’s worth being there for it.

Subscribe now

This guide gives you the preparation toolkit needed for successful fine-tuning. You’ll get dataset format examples, an explanation of how LoRA works, platform research to compare your options, a cost tracking spreadsheet template, and evaluation methods for models without labeled test data. This guide focuses on preparation and planning rather than training code. The planning infrastructure covered here determines whether your fine-tuning project succeeds or wastes resources.

Read more

Choosing between prompt engineering, RAG, and fine-tuning means matching your requirements to the right level of investment. Get this wrong, and you’ll spend weeks building something you could’ve solved in an afternoon.

Let’s fix that.

3-Layer Stack: What You’re Actually Choosing Between

Prompt engineering means crafting better instructions for a base model. You’re working entirely at the inference level, modifying inputs to get better outputs. The model stays the same, and you’re not pulling in external data.

RAG (Retrieval-Augmented Generation) adds a knowledge layer. You’re pulling relevant context from your documents or databases and injecting it into prompts at runtime. The model stays unchanged, but now it has access to your specific information.

Fine-tuning means retraining the model itself on your data. You’re modifying the model’s weights to specialize its behavior, teaching it patterns, formats, or domain knowledge that weren’t emphasized in the base training.

Subscribe now

Three Questions That Determine Your Approach

Does the model already know how to do this task? If you’re asking GPT-5 to write marketing copy or summarize documents, the base capability exists, even if you need to refine the outputs through better prompts. Begin with prompt engineering to dial in the style and format. If you’re asking it to generate outputs in a proprietary format or follow domain-specific conventions it’s never seen, you might need fine-tuning.

Does the task require knowledge the model doesn’t have? When you need the LLM to reference your internal documentation, product specifications, or company policies, that’s a RAG problem. The model can reason about information, but it can’t memorize your entire knowledge base. Don’t fine-tune when you just need to feed in context.

Is consistent behavior across thousands of examples more important than flexibility? Fine-tuning shines when you need the model to reliably produce outputs in a specific style, follow particular reasoning patterns, or handle specialized scenarios that would require increasingly complex prompts. If you’re solving one-off tasks or experimenting with different approaches, the rigidity of a fine-tuned model works against you.

This week’s paid article:

A senior data scientist’s framework for using DeepSeek effectively, including three workflow patterns for messy CSVs, slow SQL, and legacy code.

UPGRADE

Once you’ve answered these questions, consider the cost implications of each approach.

Cost Comparison at a Glance

Prompt engineering costs you time upfront, but keeps inference costs standard. You’re paying for the experimentation phase, not the deployment.

RAG adds infrastructure overhead. You need vector databases, embedding models, and retrieval logic. Your per-query cost increases because you’re making multiple API calls and processing additional tokens.

Fine-tuning flips the cost structure. High upfront investment in compute, data preparation, and evaluation. Lower per-query costs if you’re using a smaller model, higher costs if you’re fine-tuning large models and still using them at scale.

Use Case Examples

Here’s how these questions play out in real projects:

1. You’re building a customer support chatbot that needs to reference your FAQ documents and troubleshooting guides. That’s RAG. The model can already reason with the information you provide. RAG lets you surface the right documents at query time.

2. You’re generating SQL queries from natural language, and your database schema uses unconventional naming patterns that confuse base models. That’s a fine-tuning candidate. You want the model to internalize your specific patterns.

3. You’re summarizing meeting transcripts in a specific format with particular sections and bullet styles. Start with prompt engineering. Most models can follow formatting instructions if you’re clear enough. Only escalate to fine-tuning if prompt complexity becomes unmanageable.

4. You’re extracting structured data from unstructured medical notes, where the model needs to understand clinical abbreviations and context-dependent terminology. This might need fine-tuning if your domain vocabulary differs significantly from what’s in the training data.

   Subscribe now

‘Is This Overkill?’ Checklist

Before you commit to RAG or fine-tuning, run through these checks:

• Can you solve this with 10 examples in your prompt instead of 10,000 examples in a training set? (If yes, you don’t need fine-tuning.)

• Does your use case actually require real-time access to changing information? (If yes, RAG. (If no, maybe just better prompts.)

• Will you be running this model hundreds of thousands of times? (If no, the cost optimization from fine-tuning probably doesn’t matter.)

• Do you have clean, representative training data that teaches the model something it doesn’t know? (If no, don’t fine-tune.)

• Can you describe your desired behavior in words? (If yes, try prompt engineering first.)

  ---

Bottom Line

Most teams skip straight to the expensive solution because it feels more sophisticated. But prompt engineering handles more use cases than people expect, RAG solves the knowledge problem without model changes, and fine-tuning should be your last resort, not your first instinct.

Start simple, escalate when you have evidence that simpler approaches won’t work.

This Wednesday, for paid subscribers, I’m sharing a complete fine-tuning guide, including dataset preparation templates, a conceptual LoRA walkthrough, platform research comparisons, and a cost-tracking spreadsheet you can adapt for your own projects.

Become A Paid Subscriber

A decision framework and three reusable workflows

I don’t use ChatGPT for my data work. For public datasets and personal projects, like the examples in this article, I use DeepSeek.com. It has a 1 million token context window, it’s free, and it supports complex reasoning without losing track during a multi-table join.

When I’m working with sensitive data (which is often the case for my work), I run DeepSeek models locally using Ollama + Continue.dev. The local versions support 128k tokens. 128k tokens sounds abstract, but in practice, it means I can paste the entire schema of a 50-table data warehouse and still have room to ask follow-up questions. I’ve never hit the limit during normal analysis.

This article gives you a decision framework for when to use DeepSeek and when to code manually, a quality control checklist for catching AI mistakes before they cost you hours, and reusable workflow patterns you can adapt to your own analysis tasks. You’ll walk away knowing exactly when to delegate work to an LLM and when to just type the damn code yourself.

Why DeepSeek? And Addressing Privacy Concerns

There are two camps. One group won’t touch DeepSeek because of data residency concerns. The other is excited about publicly available models that match or beat proprietary options on reasoning benchmarks. I’m in the latter camp for my work, unless a client I’m contracting with already has an enterprise account with OpenAI or Anthropic that they prefer I use (which happens most of the time).

My position is that DeepSeek.com is fine for public datasets and the kind of examples I’m sharing here. For sensitive work, the local setup I mentioned earlier is straightforward: install Ollama, pull the DeepSeek model, and point Continue.dev at your local instance. You get the reasoning power without sending anything to external servers.

This isn’t a setup guide, so I won’t walk through every step. The point is that local deployment is viable if privacy is important to your use case. For everything else, the web interface works perfectly well.

Decision Framework: When to Use DeepSeek vs. Coding Manually

DeepSeek handles tasks that require understanding context, reasoning through approaches, or generating code you’d need to look up anyway. You should code manually when the task is faster to type than to prompt, when you’re learning something new, or when you need absolute certainty.

Here’s how I decide:

If the task requires understanding why something works or what approach to take, use DeepSeek. If you already know what to write, just type it.

Here’s how that plays out across three common scenarios:

Read more

“I hate leet coding interviews.”

My friend, a senior analyst, told me that over coffee last month. Five years in, crushing it at her job, but dreading the thought of ever having to switch roles because of the interview gauntlet. And honestly? I get it. When you’re spending your days designing data strategies and managing stakeholders, being asked to write a flawless binary search on a whiteboard feels absurd. Especially when Copilot’s sitting right there in VS Code, ready to handle that instantly.

So yeah, the “stop learning SQL and Python” crowd has a point. Just not the one they think they’re making.

📌 Quick note: This is a free article on concepts. Paid subscribers get implementation guides midweek, every week.

Upgrade

Why Everyone’s Fed Up

Two things are happening at once. First, as you get more senior, your job changes. You’re not writing queries all day anymore. You’re figuring out what questions to ask, which systems to build, and how to structure teams around data. Second, AI assistants have gotten really good at syntax. I don’t keep the exact parameters for window functions memorized anymore, and I don’t need to. That’s what documentation is for. GitHub Copilot does.

These tools aren’t the problem. Most interview processes are still stuck in 2015, testing whether you’ve memorized edge cases instead of whether you can actually solve problems. That’s what’s driving people up the wall.

But Here’s Where It Gets Tricky

There’s a huge difference between “I don’t need to memorize syntax” and “I don’t need to understand how this works.” Some people are conflating the two, and that’s dangerous.

AI is incredible at generating code. It’s also incredibly bad at knowing whether that code is right for your specific situation. I’ve watched Copilot suggest a JOIN that would absolutely wreck performance on our data model. If I didn’t understand what was happening under the hood, I’d have shipped it and spent the next week debugging production issues.

What hasn’t changed is that you still need to debug things. You still need to spot when something’s wrong. And if you’re senior, you’re definitely mentoring people and making architectural calls. You can’t do any of that without understanding the underlying concepts. The tool that writes the code might be new, but the requirement to know what good code looks like isn’t going anywhere.

🔒 This Week’s Paid Guide

A technical guide to building production RAG systems that covers vector database selection, AI embedding model costs, chunking strategies, and optimization patterns that control your LLM infrastructure spend.

Upgrade

What Matters Now

So if mindless memorization of code snippets is out, what’s in? I think about it as four core things.

First: problem framing. Can you take a messy stakeholder request and turn it into something concrete? That’s half the battle right there.

Second: critical thinking. When your model spits out results, can you tell if they make sense? Can you reason about uncertainty, edge cases, and what might be missing?

Third: working effectively with AI. Writing good prompts, integrating outputs, and knowing when to trust the suggestion and when to override it. This is a skill now.

Fourth: technical judgment. Whether a human or an LLM wrote it, can you review the code and evaluate whether it’s efficient, maintainable, and correct? Can you explain why one approach is better than another?

These aren’t easier skills than syntax memorization. They’re harder, but they’re the ones that actually matter.

Subscribe now

Where This Is Heading

The good news is that some teams are figuring this out. I’ve seen interviews shift from “write this algorithm from scratch” to “here’s an ambiguous business problem, walk me through how you’d approach it.” You can use whatever tools you want. The interviewer wants to see how you think, how you break down problems, and what trade-offs you consider.

That’s not a lower bar. It’s a higher one. It’s just measuring the right things.

And honestly, that’s the direction all of us need to be moving into, whether we’re hiring or getting hired. Let AI handle the mechanical stuff. That’s what it’s good at. Focus your energy on the conceptual work, the strategic thinking, the judgment calls. That’s where humans still have the advantage, and probably will for a while.

Here’s what paid subscribers got this past month:

Advanced Model Drift Detection - A technical deep dive into statistical methods (PSI, Wasserstein, KL divergence) for catching data distribution shifts before they tank your models, plus a diagnostic framework that takes you from alert to root cause without manual investigation.

MLOps on a $50 Monthly Budget - The complete architecture for running production ML inference at founder-friendly costs using serverless GPUs, DuckDB querying S3, and automated shutdown scripts that prevent runaway cloud bills.

AI Agent Starter Kit - Production-grade agent infrastructure with built-in cost tracking, compliance engines for EU AI Act/GDPR, and structured logging that makes debugging agents forensic analysis instead of guesswork.

DIY Data Catalog Template - A working metadata management system you can test locally in under 2 hours, with validation scripts, tier-based governance, and a Google Sheets backend that proves the pattern before you invest in vendor platforms.

Not a paid subscriber yet?

UPGRADE NOW

Read more

Vector databases provide the necessary architecture to maintain low-latency retrieval as your data scales, solving the engineering bottlenecks that cause simpler systems to fail under load.

Vector Indexing for Semantic Retrieval

Production AI systems require a shift in how we handle data retrieval. While standard database operations rely on rigid filters and specific keys, AI applications depend on the relationship between concepts. A vector database specializes in these relationships by treating data points as coordinates in space rather than entries in a list. It serves as a high-performance engine for approximate nearest neighbor (ANN) search over dense vector embeddings, enabling the system to navigate concepts rather than keywords.

To put it simply, instead of scanning for matching words, the system organizes information based on how closely related the ideas are. It functions like a digital library that has already grouped similar topics together across an expansive 3D map. This allows the software to jump straight to the most relevant section and find the best answers instantly, rather than checking every single page one by one.

Subscribe now

Retrieval Engine for Production RAG

In a live AI application, the speed and quality of the information the system finds are vital. If the search takes too long, the user waits. If it encounters incorrect information, the AI gives the wrong answer.

A vector database acts as a high-speed search engine, finding the most useful facts in a fraction of a second. By feeding the AI only the most relevant data, the system remains accurate while reducing the costs of running large models.

Implementation Trade-offs: Build, Buy, or Hybrid

Choosing the right path depends on your team’s bandwidth and your project’s specific constraints.

Managed Services (e.g., Pinecone)

• Pros: Zero ops overhead, fast integration for rapid deployment, and guaranteed availability via SLAs.

• Cons: Recurring monthly costs that scale with data volume, less control over underlying infrastructure, and potential data egress fees.

Cloud-Hosted Open Source (e.g., Weaviate Cloud)

• Pros: Flexibility of an open source core combined with managed scaling and often granular, usage-based pricing models.

• Cons: You still own some configuration logic, and you’re still paying a cloud premium.

Self-Hosted (e.g., pgvector, Milvus)

• Pros: Maximum architectural control, cost-effective at high volume, and integrates directly with existing data infrastructure (like PostgreSQL).

• Cons: Significant engineering burden for initial deployment, ongoing monitoring, and manual performance optimization.

Subscribe now

Use these three questions to evaluate your current trajectory:

1. Is your project transitioning from a proof-of-concept to a deployed application with actual user traffic?

2. Does your retrieval latency or recall accuracy degrade when your document volume exceeds 10,000 chunks?

3. What’s your team’s higher priority: developer velocity or long-term infrastructure cost control?

If you answered ‘yes’ to the first two questions, you’ve reached the point where a dedicated vector database is a requirement. Your answer to the third question will dictate whether you should lean toward a managed service or a self-hosted solution.

Vector databases are the industry standard for a specific engineering requirement: scalable similarity search in AI applications. By offloading the complexity of high-dimensional indexing to a dedicated system, you ensure your RAG pipelines remain responsive as your data grows.

Building these systems at scale involves nuanced architectural decisions that go far beyond basic implementation. For exclusive access to deep-dive technical reports and advanced strategies for engineering production AI, consider upgrading to a paid subscription to The Data Letter.

Upgrade to Paid

Most production ML systems monitor drift by running statistical tests on a schedule, comparing recent data to a baseline, and triggering alerts when thresholds are crossed. This generates two problems: gradual, meaningful drift gets missed, while natural variance creates false alarms. Models can pass scheduled retrain validations with strong offline metrics while business outcomes quietly degrade because the monitoring approach lacks the sophistication to catch what matters.

Data scientists have access to sophisticated statistical methods for detecting distribution shifts. Most monitoring failures stem from applying these tools without understanding what they measure, when they’re appropriate, or how to tune them for specific systems. Teams often implement drift detection as a checkbox compliance task rather than a diagnostic system. They pick a metric (often PSI because it’s popular), set an arbitrary threshold (0.1 or 0.2 because that’s what a blog post suggested), and wait for alerts. When those alerts arrive, there’s no clear path from “PSI exceeded threshold” to “here’s what’s wrong and what to do about it.”

Effective production monitoring requires understanding what each statistical method actually measures, when it’s appropriate, and how to tune it for your specific system. More critically, it requires a diagnostic framework that takes you from an alert to a root cause without manual data spelunking every time.

👋🏿👋🏿👋🏿 Welcome to all new TDL subscribers this week! Here are some recent popular articles you might be interested in as well:

MLOps on a $50 Monthly Budget

Data Catalog Implementation

DIY Data Catalog

Production Hell of AI Agents

AI Agent Starter Kit

Paid subscribers get full access to all technical deep dives, implementation guides, and operational playbooks.

Upgrade to Paid

Before you can diagnose drift effectively, you need to understand which statistical methods detect which types of changes.

Five Statistical Tools for Drift Detection

Think of drift detection methods as specialized instruments, each designed to answer a specific question about your data. Using the wrong tool or misinterpreting its output leads to either missed drift or alert fatigue.

Population Stability Index (PSI) asks: Has the proportion of data falling into predefined buckets changed? It divides your feature values into categories (bins) and compares the proportions of data in each category between your baseline and current data. Think of it like dividing ages into ranges (18-25, 26-35, 36-45) and checking whether the percentages in each range have shifted. PSI is particularly useful for monitoring discrete or naturally bucketed features, such as credit score ranges or price tiers. Its strength is interpretability: you can immediately see which bins shifted. Its weakness is sensitivity to how you define those bins.

Wasserstein Distance (also called Earth Mover’s Distance) asks: How much work would it take to reshape one distribution into another? It measures the minimum cost to transport probability mass from one distribution to match another. This makes it well-suited for continuous features where you care about the magnitude of shifts, not just their presence. A small Wasserstein distance indicates that the distributions are similar. A large one indicates they’ve moved apart. It handles heavy-tailed distributions better than variance-based metrics.

Kullback-Leibler Divergence asks: How much information would I lose if I used my old distribution to model the new one? KL divergence quantifies the difference between two probability distributions by measuring how one distribution diverges from the other. The calculation works differently depending on which distribution you treat as the reference (KL(P||Q) ≠ KL(Q||P)). It’s also highly sensitive to changes in rare events or edge cases. This sensitivity makes it powerful for detecting subtle shifts in uncommon scenarios, but creates problems when certain values appear in new data that never appeared in your baseline (the math breaks down when probabilities hit zero).

Maximum Mean Discrepancy (MMD) asks: Do these two high-dimensional distributions differ in a statistically significant way? It compares distributions by transforming them into a special mathematical space (using a kernel function) and measuring how far apart they are in that transformed view. Think of it like comparing two cities not by their street layouts, but by converting each into a set of aggregate statistics (population density patterns, commercial vs. residential ratios) and comparing those. MMD works well for complex data types such as text embeddings, image representations, and other multi-dimensional features where simpler metrics struggle. It requires more computation but detects complex, multivariate shifts that single-variable methods miss.

The Page-Hinkley Test asks: Has this metric started showing a consistent pattern of deviation from its typical level? Unlike previous methods that compare two static distributions, Page-Hinkley tracks a cumulative metric over time and signals a change point when it crosses a threshold. This makes it ideal for monitoring prediction metrics (like average predicted probability) where you want to detect the moment drift begins, not just that it occurred.

Tool selection is just the starting point. Operational challenges intensify when you need to set thresholds that separate meaningful changes from random fluctuations, and when an alert triggers but you can’t determine whether it indicates a data pipeline bug, legitimate distribution shift, or actual concept drift. Most drift detection implementations fail at these two points.

Read more

As a first time founder building my own product, I’m prioritizing cost control from day one. I’m determined to reach product-market fit without infrastructure costs consuming my runway. This article documents the stack I’m building (designed to start under $50 monthly and scale only when usage justifies the cost).

The alternative requires a different mental model. Orchestrate poverty, don’t provision wealth. Build systems that wake up on demand and sleep when idle. Design for right-sized compute, not always-on excess. Automate ruthlessly so frugality compounds without manual intervention.

Subscribe now

Foundations of Frugal MLOps: Rethinking Defaults

Most MLOps tutorials push you toward the enterprise playbook (managed feature stores, always-on inference endpoints, dedicated GPU instances). This works when you have venture capital. For solo founders and indie hackers bootstrapping their way to revenue, these costs can consume months of runway before you serve a single paying customer.

Consider the typical quick start path. Spin up a SageMaker inference endpoint (starting at $0.526/hour for g4dn.xlarge with a T4 GPU), provision a managed feature store, and maybe add DataDog for monitoring (starting at $15 per host monthly). You’re immediately committed to $378+ monthly before serving a single prediction (one g4dn.xlarge instance running 24/7 at $0.526/hour = $379 monthly). Scale that across development, staging, and production environments, and you’re burning $500-1,000 monthly on infrastructure alone.

Treat compute as a scarce resource that must justify its existence every second. Default to stateless, ephemeral architecture. Embrace serverless patterns that bill you for actual usage, not provisioned capacity. Replace managed services with open-source tools running on minimal infrastructure.

The constraint forces better architecture while preserving runway until revenue justifies higher spending (stateless services, efficient batching, aggressive caching, and clear separation between hot and cold paths).

Subscribe now

Blueprint: A $50 Monthly MLOps Stack

Here’s the architecture designed for production ML inference at under $50 per month.

Read more

A little about my guest today:

documents the history and future of operating companies at , where he examines how elite organizations build competitive advantage through operational excellence. A Harvard graduate who began his career as a PhD student in economics at the University of Chicago, John brings both academic rigor and practical experience to his analysis. Having sold his family’s B2B industrial distribution company in 2021 before founding 6A East Partners, a research and advisory firm exploring the fundamental question: What is the future of companies?

His writing spans Operating Economics (dissecting how companies like Amazon and, of course, JPMorgan build optimization engines), Operating History (connecting contemporary strategy to legends like Andy Grove and Alfred Sloan), and the AI Upskilling Playbook (research backed frameworks for career resilience in the AI era). With popular pieces like The AI Skills Crisis Isn’t What You Think and Operating Economics: Building Antifragile Companies, John has built a community of operators, founders, and builders seeking to understand not just what companies do, but how they actually work.

He helps business owners, founders, and investors optimize their operations, translating insights from Fortune 500 transformations into actionable frameworks for companies at any scale. John creates content daily (despite the occasional protests from his beloved wife, Fabiola) and still cringes at his early LinkedIn posts.

Part 1 examined JPMorgan’s financial performance and strategic decisions from 2021-2026: $2 billion in annual AI investment generating $2 billion in returns, 450 production models scaling to 1,000 by year-end, and a workforce transformation that included a 10% reduction in operations staff.

JPMorgan Chase generates $2 billion in annual business value from artificial intelligence by treating model deployment as an industrial engineering challenge. While most organizations prioritize algorithms or talent, JPMorgan focuses on the operational plumbing required to scale.

Success derives from data pipelines processing $10 trillion in daily transactions with zero downtime tolerance, feature stores eliminating training-serving skew across 450 production models, and deployment infrastructure pushing updates to 250,000 employees every eight weeks while maintaining regulatory compliance across 120 jurisdictions.

JPMorgan’s AI returns reflect a decade of data engineering that preceded the current AI wave. While competitors scrambled to hire data scientists after ChatGPT’s November 2022 launch, JPMorgan already operated 400+ production use cases spanning fraud detection, credit decisioning, and algorithmic trading. The bank employs 900 data scientists, 600 ML engineers, and a 200-person AI research team. This infrastructure provides a structural advantage that most enterprises cannot replicate, regardless of budget.

This article examines the technical systems generating those returns. These include cloud migration architecture moving petabytes off mainframes, ML lifecycle management serving 1,700 AI specialists, LLM gateway processing millions of employee queries, and real-time fraud detection operating at millisecond latency.

Subscribe now

Cloud Migration Strategy: High-Volume Data Systems Transformation

JPMorgan processes nearly $10 trillion in payments daily across 120 currencies. Moving that volume from mainframe systems to cloud infrastructure presents technical challenges most engineers never encounter: dual-write architectures maintaining consistency across legacy and modern systems, schema translation for decades-old data models, and testing strategies that verify correctness without access to production traffic patterns.

The bank increased cloud application deployment from 38% in 2022 to 65% by 2025, with approximately 80% now running on modern infrastructure. That three-year timeline reflects the actual complexity of financial systems migration, not the “lift and shift in six months” narrative common in cloud vendor marketing.

The migration challenge requires running old and new systems simultaneously. JPMorgan uses change data capture technology that copies every transaction from mainframe computers to cloud databases in real-time, like carbon paper, creating a duplicate. Both systems process the same transactions for months while engineers verify that the cloud version produces identical results.

Old mainframe systems store data differently from modern cloud databases. Records from the 1980s use outdated formatting that doesn’t translate directly to today’s standards. A single customer’s information might be scattered across 15 different mainframe files that need to be combined into a single modern database record.

Testing presents unique constraints. Banks cannot experiment with real customer transactions due to regulatory rules. Instead, JPMorgan runs shadow tests where cloud systems process copies of live transactions without affecting actual payments. Engineers compare outputs between old and new systems. Only after weeks of matching results with total precision does JPMorgan shift real traffic to cloud infrastructure.

The bank invested over $2 billion in building cloud data centers with computing power that mainframes cannot provide. Modern fraud detection requires specialized processors that analyze transactions instantly, achieving the 300x speed improvement documented in Part 1, where older systems took 24 hours running overnight batch processes.

Subscribe now

OmniAI Platform: ML Lifecycle Management for 1,700 Specialists

JPMorgan operates OmniAI, an internal ML platform serving 1,700 AI specialists across 450+ production models. The platform addresses problems obscured from organizations running 5-20 models but essential at scale: feature definition inconsistencies that cause training-serving skew, model versioning across teams that independently update shared features, and deployment pipelines that balance engineering velocity and regulatory audit requirements.

Feature stores solve a coordination problem. When different teams build models independently, they often define the same metric differently. One team calculates ‘customer transaction velocity’ over 7 days, another over 30 days. Models trained on one definition but deployed using another produce incorrect predictions.

JPMorgan’s feature store provides a single definition that all teams share. Engineers define each metric once, and the platform ensures every model uses the same calculation for both training and production. This prevents errors where models work in testing but fail in real-world deployment.

The system maintains two versions: a storage system for historical data used in model training, and a fast-access system delivering metrics in milliseconds for real-time predictions. Some metrics update nightly (such as customer credit history), while others update instantly (such as current transaction counts).

Deployment follows a careful progression. New models first run alongside existing models without affecting actual decisions (’shadow deployment’). Engineers compare predictions between old and new versions. Next, new models handle 1-5% of real transactions (’canary deployment’) and are gradually increased if performance remains stable. Automated systems monitor for problems and can reverse deployments if metrics degrade.

Monitoring 450+ models requires tracking whether model predictions remain accurate as real-world patterns change. Each model logs its decisions to a central system. Nightly analysis checks whether current predictions match patterns from training data, alerting engineers when models need retraining.

This platform infrastructure enables JPMorgan’s distributed AI organization: 900 data scientists, 600 ML engineers, and 1,000 data management specialists working across lines of business while maintaining centralized governance. Without OmniAI, coordinating 1,700 specialists across 450+ models would create competing definitions, inconsistent deployments, and regulatory compliance failures.

Subscribe now

LLM Suite Gateway: Powering 250,000 Employees

The platform manages multiple AI providers (OpenAI and Anthropic) through a single interface. Employees request AI assistance without choosing specific vendors; the system routes requests based on cost and capabilities. This prevents dependence on any single vendor.

Cost tracking monitors AI usage by person, department, and application. The system records how many ‘tokens’ (roughly equivalent to words) each request consumes, which AI model processed it, and when it was processed. Finance teams receive monthly reports showing AI costs by business unit, enabling budgeting and cost allocation.

The system connects AI models to JPMorgan’s internal documents using retrieval-augmented generation (RAG). When employees ask questions, the system searches millions of internal documents for relevant information, then provides that context to the AI before generating answers. This grounds responses in JPMorgan’s actual policies and data rather than the AI’s general training.

Vector databases enable semantic search across internal documents. Unlike keyword search, which matches exact words, semantic search understands meaning. A search for ‘loan approval process’ finds relevant documents even if they use different terminology, such as ‘credit decisioning workflow.’

Access controls ensure employees only receive information they’re permitted to see. Investment bankers cannot access retail customer data; retail employees cannot access merger documents. The system filters search results based on employee permissions before providing context to the AI.

The eight-week update cadence manages LLM Suite platform changes: evaluation → testing → limited rollout → full deployment. Evaluation assesses new models against existing models using golden datasets (curated question-answer pairs covering common use cases). Limited rollout serves 5-10% of production traffic and monitors for quality degradation. Full deployment occurs only after validating metrics across a limited rollout.

The platform enables measured productivity gains: investment bankers automating 40% of research tasks, portfolio managers cutting research time by up to 83%, and wealth advisors finding information 95% faster.

Subscribe now

Real-Time Fraud Detection at $10 Trillion Daily Scale

JPMorgan’s fraud detection operates at millisecond latency while processing $10 trillion in daily transactions. The system achieves 40% higher accuracy than traditional rule-based approaches, enabling real-time blocking of fraudulent transactions before they are completed.

The system must analyze each transaction in milliseconds before completing payment. At $10 trillion in daily volume across 8 billion transactions, JPMorgan processes roughly 93,000 transactions per second on average, peaking above 200,000 during peak periods.

Fraud detection examines dozens of signals: transaction amount, merchant type, location, time of day, account age, typical spending patterns, recent transaction frequency, and connections to known fraudulent accounts. The system computes these signals instantly by maintaining pre-calculated summaries that update with each transaction rather than recalculating from scratch.

The architecture uses streaming technologies (Kafka and Flink) to continuously process transactions. As transactions occur, the system computes risk signals, scores fraud probability, and returns decisions before payments are complete.

Reducing false positives represents the primary challenge. Traditional systems block many legitimate transactions, frustrating customers. JPMorgan’s 40% improvement in accuracy means the system correctly identifies actual fraud while allowing more legitimate transactions through.

The system continuously re-trains on recent fraud patterns. Fraudsters constantly evolve their tactics, making yesterday’s model less effective today. JPMorgan deploys updated models through the same careful progression used for other AI systems (shadow testing, then gradual rollout).

Fraud attacks grow 12% annually. Without AI, fraud losses would escalate exponentially. By holding fraud costs flat despite surging attack volumes, the system avoids hundreds of millions in potential losses, contributing significantly to the $2 billion in annual business value detailed in Part 1. This represents defensive value that’s easily overlooked: preventing losses rather than generating new revenue.

Subscribe now

Strategic Maturity: Structural Barriers to AI Replication

Data infrastructure maturity distinguishes successful AI initiatives from failed attempts.

JPMorgan spent a decade building foundations before deploying AI at scale, unifying customer information across 30+ disconnected legacy systems. This unification ensures every department accesses identical customer data. The bank built pipelines moving data from mainframe computers to cloud databases instantly. In contrast, many organizations rely on inconsistent customer records, daily batch updates, and manual error tracking.

Testing requirements reflect banking regulations. JPMorgan must explain every credit decision to regulators, prove fraud models treat all customers fairly, and demonstrate AI systems meet safety standards. Most companies deploy models after basic accuracy testing without rigorous validation or explanation capabilities.

Subscribe now

Legacy system integration determines what’s possible. JPMorgan spent years building connections, allowing modern AI systems to interact with decades-old mainframe applications. Companies with similar old systems but without this integration cannot deploy AI, requiring instant access to core data.

The $2 billion in returns comes more from data infrastructure, testing frameworks, and operational systems than from sophisticated algorithms. Any company can buy the same cloud services, use the same open-source software, and hire comparable talent. What cannot be replicated quickly is the decade of data engineering, laying the foundations for deployment.

Organizations should assess their readiness through honest questions: Do you have consistent customer information across all systems, or does marketing see different data than sales? Can you calculate customer metrics instantly, or does data processing take hours? Do you have shared tools preventing teams from defining the same metric differently? Can you test new models safely without affecting customers? Do you monitor whether models stay accurate as conditions change? Can you explain to regulators why models make specific decisions?

Honest answers reveal gaps. Closing them requires multi-year infrastructure investment, producing no immediate revenue. Executives must approve spending millions on data quality tools and deployment automation before the first AI model delivers business value. This patience separates JPMorgan from competitors seeking immediate AI returns.

Part 1 documented JPMorgan’s competitive positioning: the bank ranks #1 on Evident AI’s Index for AI maturity, commands a valuation premium of 2.68x book value versus peers, and outperformed bank indices by 35% in 2025. That market recognition reflects investor confidence that competitors cannot quickly replicate this infrastructure advantage. Wells Fargo, Citigroup, Goldman Sachs, and Bank of America all acknowledge AI investments, but none disclose production use case counts or specific financial returns approaching JPMorgan’s transparency.

Subscribe now

Data Infrastructure Foundations: Prerequisites for AI Success

JPMorgan’s $2 billion annual return stems from a technical infrastructure built to handle 450 production models scaling toward 1,000. These results include 40% higher fraud detection accuracy and wealth advisors finding information 95% faster. Such outcomes originate from a decade of foundational engineering rather than the recent adoption of popular algorithms.

Many organizations struggle with AI because they lack foundational data architecture. Without consistent identifiers, real-time pipelines, and automated drift monitoring, data scientists spend 80% of their time on infrastructure maintenance. JPMorgan’s success proves that shifting this ratio through centralized platforms creates a structural lead over competitors.

Technical practitioners should recognize that AI returns require ML operations to be mature, which can take years to build. Organizations starting AI initiatives today must invest in data infrastructure, feature engineering capabilities, deployment automation, and monitoring systems before expecting $2 billion returns. Machine learning models represent a widely available commodity. The infrastructure surrounding those models determines success.

JPMorgan’s AI leadership persists because competitors cannot quickly replicate the data engineering foundation. Regional banks can hire data scientists, but cannot build equivalent data pipelines without a multi-year investment. Fintech startups begin with modern infrastructure but lack transaction data at JPMorgan’s scale.

For executives considering AI investment, JPMorgan’s experience teaches patience. The bank spent years building data infrastructure starting from 300 use cases and $100 million in value (2022) to reach 450+ use cases and $2 billion in value (2025). Expecting immediate ROI from AI initiatives without addressing foundational data quality, infrastructure, and governance challenges leads to failed projects and abandoned strategies.

The future of enterprise AI belongs to organizations willing to invest in infrastructure first and algorithms second. JPMorgan proves the thesis: $2 billion in annual returns from infrastructure that enables 1,700 specialists to operate 450+ models, delivering measurable business value. That infrastructure includes cloud migration, feature stores, deployment pipelines, and monitoring systems.

A special thanks to John Brewton for collaborating on this analysis!

The Data Letter delivers breakdowns like this on the systems and strategies behind the world's most important companies. For deep dives into data infrastructure, MLOps, and the operating models that create billion-dollar advantages, become a paying subscriber.

Upgrade to Paid

This happens everywhere. Teams choose Spark when DuckDB would suffice. They deploy Airflow when cron would work. They architect for scale they’ll never reach, then spend years paying the operational cost. This pattern repeats so consistently that I’ve developed a checklist to separate genuine requirements from resume-driven development.

Subscribe now

Two diseases that plague modern data teams

The first disease is the future proofing fallacy. Engineers convince themselves they’re building for tomorrow’s scale, but tomorrow never arrives. I once worked with a team that implemented a complex real-time streaming pipeline because “we might need low-latency analytics eventually.” Two years later, they were still running batch reports overnight. The streaming infrastructure sat there, consuming cloud credits and engineering time, solving a problem that existed only in planning documents.

The second disease is tool selection by conference talk. An engineer attends a presentation about how Netflix uses Flink, then returns to their 50-person company convinced they need the same architecture. I call this conference driven development. The engineer forgets that Netflix processes petabytes of data daily, with hundreds of engineers. Their company processes gigabytes with three engineers. The scale mismatch renders the comparison meaningless.

Both diseases stem from the same root cause: engineers optimize for perceived sophistication rather than actual requirements. The cure is to ask better questions before writing any code.

👋🏿👋🏿👋🏿 Welcome back to The Data Letter! Here are some recent articles you may have missed:

Production Hell of AI Agents

AI Agent Starter Kit

DIY Data Catalog Template

Five diagnostic questions for architecture decisions

Before choosing any data tool, I run through these five questions. They’ve saved me from countless overengineering disasters.

Question 1: What problem am I solving right now, not hypothetically? Write down the specific pain point. If you can’t articulate it in one sentence without using “might” or “could,” you don’t have a clear requirement. Real problems are concrete: “The daily report takes 6 hours to run” or “Users wait 30 seconds for search results.” Hypothetical problems sound like: “We might need to scale to millions of users.”

Question 2: What’s the actual data volume today, and what will it realistically be in 12 months? Measure in concrete units. If you’re processing 50GB daily now, honest growth projections rarely exceed 200GB in a year unless you’re experiencing explosive user acquisition. I’ve never seen a team accurately predict they’d 10x their data volume. Most teams 2x or 3x over years, not months.

Question 3: Can I solve this with tools I already understand? Familiarity has enormous value. A tool you know well will always outperform a superior tool you barely understand. I’d rather maintain a slightly awkward Postgres solution than a theoretically elegant Kafka setup that requires constant Stack Overflow searches.

Question 4: What’s the operational burden of this choice? Count the hours weekly. Will someone need to monitor dashboards? Investigate failures? Tune performance? Upgrade versions? If the answer exceeds 5 hours per week, you need strong justification, as operational costs compound while delivering zero feature value to users.

Question 5: What happens if I’m wrong and need to migrate later? Most engineers fear this scenario excessively. Migration projects are common in data engineering. I’ve migrated from Postgres to distributed databases, from cron to Airflow, from batch to streaming. None took more than three months. Compare that to the years you’ll spend maintaining infrastructure you don’t need.

Subscribe now

Choosing between dull and complex tools

Here’s how I evaluate tool pairs for common scenarios:

Postgres vs. Kafka: Use Postgres for anything under 100GB daily or where hour-level latency is acceptable. Use Kafka only when you need sub-second latency across multiple consumers, or when the daily data volume exceeds 500 GB. The operational complexity difference between them is substantial. Postgres requires basic SQL knowledge. Kafka requires understanding partitions, consumer groups, retention policies, and cluster management.

DuckDB vs. Spark: Use DuckDB for anything that fits in memory on a single machine, roughly 100GB of source data. Use Spark only when your data genuinely spans terabytes and requires distributed processing. I’ve seen DuckDB outperform badly configured Spark clusters on datasets under 500GB. The performance gap narrows as data grows, but the operational burden stays disproportionately high.

Cron + SQL vs. workflow orchestration engines: Use cron for dependency graphs with fewer than 20 steps or when failures can wait hours for manual intervention. Use Airflow or Prefect when you have complex dependencies, need sophisticated retry logic, or require detailed execution history. I’ve maintained cron-based pipelines for years with zero issues. I’ve also maintained Airflow deployments that required weekly troubleshooting.

The pattern is consistent: simple tools excel when you operate below certain thresholds. Complex tools become justified only at specific scale or complexity levels.

Subscribe now

Mental model for tool selection

I visualize tool selection as a decision tree that starts with one question: Can this wait hours or days? If yes, proceed to batch processing territory. If no, advance to real-time considerations.

For batch processing, the next question becomes: Does this fit on one machine? If your data compresses to under 100GB, stay with single-machine tools like DuckDB or Postgres. If it exceeds that, advance to distributed processing tools.

For real-time requirements, ask: Do multiple systems need this data simultaneously? If no, consider whether a faster database query or materialized view solves the problem. If yes, you’ve arrived at legitimate streaming territory.

At each decision point, the answer “I’m not sure” defaults to the simpler option. You can always migrate up in complexity. Migrating down is harder but still possible.

Subscribe now

When complex tools are actually justified

I’m not arguing against sophisticated data infrastructure entirely. I can think of three scenarios that genuinely require complex tools:

First, when the data volume exceeds what one server can handle. If you’re processing multiple terabytes of data daily and a single Postgres instance can’t handle the load, even with optimization, distributed systems become necessary. But verify you’ve actually hit this limit through measurement, not assumption.

Second, when latency requirements demand it. If users expect sub-second responses on queries that scan billions of rows, or if downstream systems require millisecond-level data freshness, streaming architectures and specialized databases earn their complexity. Ensure these requirements come from user needs, not engineering preferences.

Third, when regulatory or business requirements mandate specific capabilities. Some industries require audit trails, point-in-time recovery, or cross-region replication that simpler tools can’t provide. These are external constraints that override technical preferences.

Notice how all three of these scenarios involve measurable thresholds or external requirements, and not hypothetical future needs or architectural aesthetics?

Subscribe now

Action plan for escaping overengineering

Start by auditing your current data infrastructure. For each component, answer the five diagnostic questions above. If you can’t justify a tool with concrete measurements, add it to your migration backlog.

Next, establish volume thresholds for tool selection. Write them down and share them with your team. Mine are: under 100GB daily on Postgres, under 1 TB daily on single-machine processing, and under an hour of latency in batch territory. Your thresholds might differ based on team expertise and existing infrastructure.

Then implement a complexity budget. Each new tool requires approval from the full engineering team. The proposer must demonstrate that simpler alternatives are inadequate using actual measurements, not projections. I’ve seen this single practice eliminate 80% of overengineering proposals.

Finally, celebrate boring solutions. When someone solves a problem with cron and SQL instead of a new framework, recognize it explicitly. Engineering culture often rewards complexity over simplicity. Reversing this incentive structure takes deliberate effort but pays enormous dividends.

Most data problems are small data problems wearing big data costumes. Strip away the costume, and you’ll find that simple tools, wielded competently, solve the vast majority of real requirements.

I’m currently documenting the build of Asaura AI, an AI personal assistant for people with ADHD and executive dysfunction. If you’re interested in watching a product get built from scratch (complete with user research, design decisions, and technical choices), I’m writing about it on my Asaura substack.

Three articles so far:

Executive Dysfunction ≠ Laziness

Patterns From User Research

Asaura’s Training Manual

Support The Data Letter

If you found the small data manifesto useful, consider becoming a paid subscriber. You’ll get access to deeper technical breakdowns, case studies from real projects, and the occasional rant about data engineering decisions that keep me up at night.

Become a Paid Subscriber

Teams need an engineered response: a starter kit that embeds solutions for cost control, observability, and resilience directly into an agent’s architectural foundation. The power of this approach lies in deliberate structure and explicit data flow. By organizing an agent system around interconnected design patterns, each addressing a specific failure mode, we transform debugging from archeology to engineering.

The AI Agent Starter Kit creates an agent architecture where every decision, action, and resource consumption leaves a structured, queryable trace. When something goes wrong, and it will, the system provides the precise diagnostic data needed to identify root causes and implement fixes.

Subscribe now

Design Patterns for Agent Resilience

Four foundational principles guide this architecture, each targeting a specific production failure mode.

Cost-Awareness by Default: Every interaction with a language model must emit structured cost data before execution proceeds. This principle addresses the cost spiral problem by making resource consumption a first-class concern, tracked and logged at the same level as functional outputs. The architecture treats cost as an intrinsic state, never an afterthought.

Observability-First Design: The system records detailed, machine-readable logs for every decision, API call, and action the agent takes. This requirement applies to every component without exception. This counters the debugging nightmare by ensuring diagnostic data is available before problems occur. Observability becomes part of the contract each component must fulfill.

Explicit State Management: All task context, execution history, and intermediate results reside in a single, inspectable state object that flows through the system. This addresses brittleness by eliminating implicit assumptions and hidden dependencies. When the agent fails, its complete context remains available for analysis.

Defensive Tooling Patterns: Every external integration point includes automatic input validation, output verification, and error logging before results reach the agent’s reasoning layer. This mitigates integration complexity by standardizing how the agent interacts with external systems and ensuring failures at API boundaries generate diagnostic data rather than cascading into the agent’s logic.

These four principles shape every component in the system. Understanding how they translate into concrete architectural patterns requires examining each layer in detail.

Read more

What doesn’t exist yet is the operational discipline to make agents reliable at scale. Teams are discovering that building an agent is straightforward. Making it survive contact with real users, real data, and real business constraints is an entirely different engineering challenge.

Welcome to production hell.

This isn't a failure of ambition or technology. It exposes the chasm between demo-stage potential and production-stage engineering requirements. Every team building agentic AI systems encounters this descent: a series of compounding technical, operational, and economic challenges that transform a promising prototype into a reliability nightmare. According to recent surveys, 57% of organizations have agents in production, yet quality remains the top barrier, with 32% citing it as their primary challenge.

Survival requires building a new discipline: agent operations, not waiting for the next model release. They’ll do it through pragmatic design, rigorous observability, and the wisdom to know when a deterministic workflow beats an autonomous agent.

Subscribe now

Descending Through Agent Hell: Where Production Dreams Die

Building production-ready AI agents involves overcoming different layers of difficulty, with each layer building on the previous one. Each demands engineering solutions that don’t yet exist in standardized form.

Spiraling AI Agent Costs: When LLM Expenses Devour Your Budget

A single GPT-4 call costs fractions of a penny. Then you chain six calls together: planning, tool selection, execution, validation, error recovery, and final synthesis. Now you’re at several cents per task attempt. Multiply by failed attempts. Multiply by the debugging runs that never make it to production.

The economics collapse fast. Production agents orchestrate complex workflows: state management, validation logic, retry mechanisms, and logging systems. Each piece adds cost. Most of these pieces serve as operational overhead to make LLMs semi-reliable rather than LLM calls themselves.

According to industry data, GPT-4 Turbo costs $0.01–$0.03 per 1,000 tokens, and complex agents can burn 5–10 million tokens monthly. For mid-sized deployments, LLM operational costs range from $1,000 to $5,000 per month. Long context windows and chained reasoning multiply costs exponentially. One e-commerce brand saw token usage spike 300% after enabling order-tracking workflows, pushing monthly costs from $1,200 to $4,800.

Research from 2025 shows that 80% of enterprises underestimate their AI infrastructure costs by more than 25%. This is why agent projects stall: the unit economics don’t work until reliability crosses 90%, and that threshold remains elusive.

Subscribe now

Debugging AI Agents: Tracing Failures Through Stochastic Fog

Traditional software fails the same way every time. Same input, same bug, every time. You set a breakpoint, step through the code, find the bad line, and fix it.

Agents behave differently.

Run the same prompt twice, and get two different reasoning paths. One succeeds; the other halts with a malformed JSON response. Which step broke? Was it the prompt? The model’s sampling randomness? An edge case in your validation logic? A transient API timeout that corrupted the state three steps ago?

You have no breakpoints. You have logs: thousands of lines of LLM outputs that read like stream-of-consciousness reasoning, peppered with tool calls and intermediate results. Somewhere in that morass, the agent decided to skip a necessary validation step. Why? The model doesn’t tell you. The logs show it happened, not why.

The community’s recent enthusiasm for models like DeepSeek R1 reflects this pain. DeepSeek R1’s explicit reasoning traces provide unprecedented transparency, exposing the model’s decision process through structured sequences that show problem definition, iterative refinement, and final solution articulation. Engineers want visibility into why the agent chose path A over path B, why it hallucinated a parameter value, and why it decided the task was complete when it clearly wasn’t.

Better reasoning traces help diagnosis, yet they don’t solve the core operational problem. Even with complete visibility, you’re still debugging a stochastic system. The fix for one failure pattern might introduce a new one. Prompt engineering becomes a game of whack-a-mole: patch the issue where the agent misinterprets datetime formats, only to discover it now fails on timezone conversions.

Research shows that DeepSeek R1’s transparent reasoning makes models vulnerable to safety threats and jailbreak attacks precisely because detailed traces expose the full decision-making process. The transparency that aids debugging also increases the attack surface.

You can’t write unit tests for “the agent should reason correctly.” You can write integration tests for specific scenarios, watch them pass, and hope the model generalizes. When it doesn’t, you’re back in the logs, searching for patterns in randomness.

Subscribe now

Hitting AI Agent Reliability Ceilings: When Brittleness Wins

You’ve tuned your prompts. You’ve added validation. You’ve implemented retry logic. Your agent now succeeds 75% of the time. Then you plateau.

No amount of prompt engineering gets you past 80%. The remaining failures are edge cases: ambiguous inputs, unexpected API responses, scenarios that require contextual judgment the model simply doesn’t have. You’re stuck in the reliability gap: good enough to be tantalizing, too unreliable to ship.

In demos, teams rarely mention that a simple decision tree or state machine often beats an autonomous agent on reliability, cost, debuggability, and predictability.

A decision tree with 15 branches costs nothing to run and succeeds 99% of the time. Your clever agent costs $0.30 per attempt and succeeds 78% of the time. Which one ships?

Real-world data confirms this pattern. UC Berkeley found that reliability remains the top development challenge for production agents. Rather than develop technical innovations, developers dial down their ambitions and adopt simpler methods. Most use off-the-shelf models with no fine-tuning and hand-tuned prompts. Agents have short run-times, with 68% executing fewer than 10 steps before requiring human intervention.

Testing of autonomous agents like Devin revealed stark limitations: while they excel at isolated tasks like API integrations, they achieve only 3 successes out of 20 end-to-end tasks. Simpler, developer-driven workflows using tools like Cursor avoid many issues encountered with autonomous agents.

A 5% error rate might be acceptable for a chatbot, but it becomes a massive problem for agents that place orders, update databases, or make automated decisions. One corrupted database entry can shut down operations. The brittleness ceiling forces a brutal reckoning: maybe autonomous agents aren’t the right architecture for this problem. Maybe hybrid approaches combining deterministic workflows with LLM-powered flexibility at specific decision points deliver better outcomes.

Subscribe now

Integration Labyrinth: Connecting AI Agents to Real-World Systems

While you’ve been wrestling with prompt reliability, your agent still needs to talk to the outside world. Every real production agent requires integration with existing systems: CRMs, databases, payment processors, internal APIs, and third party services. This is where demo magic dies in a pile of authentication flows and schema mismatches.

Your demo agent “sends an email.” In production, that means:

• OAuth 2.0 flow with token refresh logic

• Rate limiting that varies by provider

• Handling bounces, spam filters, and delivery failures

• Parsing varied response formats (JSON, XML, GraphQL)

• Managing API versioning when providers change schemas

• Timeouts that corrupt the multi-step agent state

• Webhook verification for async confirmations

• Error codes that mean different things across different APIs

Each integration multiplies the failure modes. Your agent might execute flawless reasoning yet hit errors because the calendar API returned a 429 rate limit, the CRM schema changed and broke your field mapping, or the payment processor requires 3D Secure authentication that your agent can’t handle.

Industry practitioners confirm this is the hardest part of production deployment. One analysis found that the most challenging part of building a production-ready agent is the “body” layer: secure authentication with third-party applications, credential management for thousands of users, and reliable execution via well-formed API calls. This is the time-consuming plumbing that separates a clever prototype from a scalable product.

Authentication complexity alone creates massive overhead. AI agents require robust, automated, and cryptographically secure authentication rather than traditional human-centric methods. Best practices include using short-lived certificates from trusted PKIs, using hardware security modules (HSMs) to store keys, and workload identity federation to tie agent identities directly to organizational infrastructure. Token rotation must happen automatically every 24-72 hours to maintain security.

Engineers spend weeks building resilient connectors, implementing exponential backoff, and writing parsers for inconsistent responses. None of this shows up in demos. All of it determines whether your agent survives contact with production.

The integration layer becomes your agent’s largest codebase, with more lines of defensive error handling than actual agent logic. You realize you’re building a traditional distributed system that happens to have an LLM in the middle, with all the operational complexity that entails.

Subscribe now

Voices from the Production Trenches: Community Sentiment on AI Agent Challenges

The disillusionment is widespread and well-documented. Across engineering communities, two narratives emerge repeatedly:

Engineers Pursuing Solutions: 89% of organizations have implemented some form of observability for their agents, and 62% have detailed tracing that allows them to inspect individual agent steps. Without visibility into how an agent reasons and acts, teams can’t reliably debug failures, optimize performance, or build trust with stakeholders.

The enthusiasm for transparent reasoning models like DeepSeek R1 reveals how starved teams are for debuggability. Engineers know the problems are solvable; they need the right instrumentation.

Business Anxiety: Organizations face significant obstacles in translating agentic pilots into deployable solutions. Even among enterprises with agents in production, maturity remains low. Only 5% of engineering leaders cited accurate tool calling as a major challenge, suggesting most production systems focus on surface-level behavior rather than deeper reasoning.

The gap between promised business value and delivered reliability creates existential pressure. According to MMC Ventures data, 42% of organizations deployed at least some agents in Q3 2025, up from 11% two quarters prior, yet 68% of employees interact with agents in fewer than half their workflows. Higher accuracy correlates with lower production autonomy. Healthcare founders report 90% accuracy rates but admit this isn’t sufficient to remove human oversight.

Teams are frustrated, not defeated. They’re sharing workarounds, comparing architectures, and debating hybrid approaches. Production hell is painful, but it’s where real engineering disciplines are forged.

Subscribe now

Standardizing Agent Infrastructure: AAIF’s Emerging Role

While teams struggle with production challenges, the industry is standardizing the infrastructure layer that could ease the pain. In December 2025, the Linux Foundation announced the Agentic AI Foundation (AAIF), bringing together Anthropic’s Model Context Protocol (MCP), Block’s goose, and OpenAI’s AGENTS.md under neutral governance.

This represents a strategic recognition: agent execution infrastructure, not model capabilities, currently blocks progress. MCP alone went from an internal Anthropic project to an industry standard in 12 months, with over 10,000 published servers now connecting agents to tools, data, and applications. The protocol has been adopted by Claude, Cursor, Microsoft Copilot, Gemini, VS Code, and ChatGPT.

The formation of AAIF, backed by platinum members including AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, Microsoft, and OpenAI, signals that the unstructured experimentation phase is coming to an end.

For engineers in production hell, this standardization offers a path forward. Rather than building custom integration layers for every external system, teams can build on MCP’s universal connection standard. Rather than inventing project-specific agent guidance, they can adopt AGENTS.md’s Markdown convention, already used by over 60,000 open-source projects.

Teams now face a different question: “How do we make this work reliably within established protocols?” That’s still hard, but it’s engineering hard, not research hard.

Subscribe now

Survival Guide: Engineering Your Way Out of AI Agent Production Hell

Production hell ends when you stop fighting the fundamental nature of LLMs and start building systems that account for their limitations. Here’s how teams are escaping:

Prioritize Deterministic Workflows Over Autonomous AI Agents

Start with the hardest question: Does this problem actually need an agent? Map your workflow. Identify which steps truly require flexible reasoning. You’ll often find 80% of your logic is deterministic and well-suited for state machines or decision trees.

Build that deterministic skeleton first. Use LLMs only for genuinely ambiguous decision points: parsing unstructured user input, choosing between semantically similar options, and generating natural language responses. This hybrid architecture delivers:

• Predictable behavior for most cases

• Debuggable logic flows

• Cost control (LLM calls only where needed)

• Clear failure boundaries

Production data validates this approach. Analysis shows that prompt and sequence lengths are steadily growing for programming use cases, while all other categories remain stagnant. Agent builders are keeping their agents simple and short to achieve reliability. Outside of coding agents, prompt and sequence complexity remain stagnant. These simpler agents are the ones reaching production.

An agent that’s 40% state machine and 60% LLM-powered flexibility often outperforms a 100% autonomous agent on every production metric.

Subscribe now

Design AI Agents for Human Oversight from Day One

Autonomous agents are a goal, not a starting point. Systems built for production assume human intervention and design for it gracefully. Research confirms that 92.5% of in-production agents deliver their output to humans, not to other software or agents. Chatbot UX dominates because it keeps a human in the loop.

Key strategies include:

• Confidence thresholds: When the agent’s certainty drops below a threshold, escalate to human review before proceeding

• Preview and confirm: Show users the agent’s planned actions before execution, especially for high-stakes operations

• Intervention points: Build explicit handoff mechanisms where humans can step in, correct course, and hand back to the agent

• Audit trails: Log every decision and action in a human-readable format for post-hoc review

Companies emphasize “co-pilot positioning” even when full autonomy is technically possible. They discovered employees either overrely or underrely on outputs, never achieving optimal collaboration. Winners aren’t building autonomous systems; they’re building narrow, high-frequency task executors with human oversight.

Measure “mean time to human intervention” as a key metric. You want this number trending down over time, but you never want it to reach zero. That signals you’ve eliminated necessary oversight.

Subscribe now

Build AI Agent Observability Before Scaling Logic

You can’t fix what you can’t see. Before adding more sophisticated agent behaviors, instrument everything. The importance of this cannot be overstated: 89% of organizations with agents in production have implemented observability, and adoption is even higher (94%) among those with mature deployments.

Essential observability components:

• Structured logging: Every tool call, every reasoning step, every state transition gets logged with context

• Distributed tracing: Track requests across the entire agent execution path, including external API calls

• Evaluation frameworks: Build automated tests that run your agent against a growing suite of real-world scenarios, not just happy paths

• Failure classification: Tag and categorize every failure mode so you can identify patterns

For production agents, teams typically track a mix of quality and performance metrics, including accuracy, task completion rate, latency, error rate, and resource usage. For many customer-facing flows, teams aim for high-90s accuracy on core tasks, completion rates above 90%, sub-second response times for simple interactions, and low single-digit error rates.

Teams that build observability first can iterate rapidly. Teams that bolt it on later spend months retrofitting logging into complex agent logic.

Subscribe now

Measure AI Agent Operational Metrics, Not Just Model Performance

Forget about vibes. Track metrics that connect to business outcomes:

• Cost per successful task: Total spend (LLM + infrastructure) divided by tasks completed successfully

• Success rate by task complexity: Segment simple vs. complex tasks to understand where brittleness lives

• Retry rate: How often does the agent need multiple attempts to complete a task?

• Human intervention rate: What percentage of tasks require human oversight or correction?

• Time to resolution: How long does the full task take, including retries and interventions?

These metrics tell you whether your agent is getting more reliable and cost-effective over time. They also give you the data to make architectural decisions. Research shows that enterprises face $50,000 to $200,000 in integration costs and take 3-6 months to deploy production agents. Without clear operational metrics, it’s impossible to determine if this investment is worthwhile.

One analysis found that startups claiming broad autonomy inevitably pivot to narrow, high-accuracy verticals. Infrastructure costs force model selection trade-offs. High-regulation industries lock into 90% accuracy, 40% autonomy configurations. Others optimize for 70/70 configurations with lower costs.

Subscribe now

Production Hell Forges Better Engineering Practices for AI Agents

Production hell feels like failure. Demos worked. Prototypes impressed stakeholders. Then reality intervened.

But this phase matters more than the demo ever did.

Right now, engineers worldwide are learning which agent architectures survive contact with production. They’re discovering that hybrid workflows beat pure autonomy. They’re building observability tools that make stochastic systems debuggable. They’re developing operational metrics that connect AI capabilities to business value. They’re establishing patterns for human-agent collaboration that acknowledge the limitations of current models while leveraging their strengths.

This is how new engineering disciplines are born. Not from triumphant product launches, but from the grinding work of making unstable technology stable. The teams enduring production hell today are writing the playbooks others will follow tomorrow.

The defining story of 2025 was not which models topped benchmarks, but which organizations successfully moved from experimentation to scaled production. Three barriers consistently prevented pilots from reaching production: reliability requirements (a 5% error rate becomes massive for agents making automated decisions), integration complexity (integrating with Oracle, Salesforce, legacy databases, security protocols, and compliance requirements often exceeded expected value), and cost-benefit analysis (proving ROI at scale).

The next generation of agentic applications won’t emerge from better models alone, though better models will help. They’ll emerge from the operational practices being forged right now: knowing when to use agents and when to use state machines, building for human oversight rather than pure autonomy, and measuring operational reliability rather than demo polish.

Production hell reveals that polished demos and deployable systems require fundamentally different engineering approaches.

Your agent will fail. Your agent will fail. The question is whether you’ve built systems to understand why, fix it, and prevent it from happening again. That capability, not model performance, determines whether you ship or stall.

Stay ahead of the shift in agent infrastructure.

Subscribe for in-depth analysis of production AI systems, protocol standards, and the teams building reliable agents at scale.

Become a paid subscriber

References

LangChain. “State of AI Agent Engineering 2025.” November-December 2025. https://www.langchain.com/state-of-agent-engineering

AgentiveAIQ. “AI Agent Cost Per Month 2025: Real Pricing Revealed.” August 2025. https://agentiveaiq.com/blog/how-much-does-ai-cost-per-month-real-pricing-revealed

Greenice. “AI Agent Development Cost 2025: Expectation vs Reality.” October 2025. https://greenice.net/ai-agent-development-cost/

Emergent Mind. “DeepSeek-R1 Reasoning Traces.” 2025. https://www.emergentmind.com/topics/deepseek-r1-reasoning-traces

Drew Breunig. “Enterprise Agents Have a Reliability Problem.” December 2025. Analysis of UC Berkeley’s MAP research and other 2025 agent studies. https://www.dbreunig.com/2025/12/06/the-state-of-agents.html

Carl Rannaberg. “State of AI Agents in 2025: A Technical Analysis.” Medium, January 2025. https://carlrannaberg.medium.com/state-of-ai-agents-in-2025-5f11444a5c78

Arion Research. “The State of Agentic AI in 2025: A Year-End Reality Check.” December 2025. https://www.arionresearch.com/blog/the-state-of-agentic-ai-in-2025-a-year-end-reality-check

Composio. “The 2026 Guide to AI Agent Builders (And Why They All Need an Action Layer).” December 2025. https://composio.dev/blog/best-ai-agent-builders-and-integrations

Obsidian Security. “Security for AI Agents: Protecting Intelligent Systems in 2025.” November 2025. https://www.obsidiansecurity.com/blog/security-for-ai-agents

Obsidian Security. “The 2025 AI Agent Security Landscape: Players, Trends, and Risks.” November 2025. https://www.obsidiansecurity.com/blog/ai-agent-market-landscape

Local AI Zone. “DeepSeek AI Models 2025: Revolutionary Reasoning AI for Education & Research.” October 2025. https://local-ai-zone.github.io/brands/deepseek-ai-coding-expert-guide-2025.html

Machine Learning Mastery. “7 Agentic AI Trends to Watch in 2026.” January 2026. https://machinelearningmastery.com/7-agentic-ai-trends-to-watch-in-2026/

Cleanlab. “AI Agents in Production 2025: Enterprise Trends and Best Practices.” August 2025. https://cleanlab.ai/ai-agents-in-production-2025/

TechUpkeep. “The Uncomfortable Truth About AI Agents: 90% Claim Victory While 10% Achieve Adoption.” Analysis of MMC Ventures data. November 2025. https://www.techupkeep.dev/blog/state-of-agentic-ai-2025

Maxim AI. “Understanding AI Agent Reliability: Best Practices for Preventing Drift in Production Systems.” November 2025. https://www.getmaxim.ai/articles/understanding-ai-agent-reliability-best-practices-for-preventing-drift-in-production-systems/

Thammineni, Prasad. “The Complete Guide to AI Agent Pricing Models in 2025.” Agentman, Medium. January 2025. https://medium.com/agentman/the-complete-guide-to-ai-agent-pricing-models-in-2025-ff65501b2802

The Linux Foundation. “Linux Foundation Announces the Formation of the Agentic AI Foundation (AAIF), Anchored by New Project Contributions Including Model Context Protocol (MCP), goose and AGENTS.md.” December 9, 2025. https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation

I’ve encountered the same story with Enterprise data catalogs multiple times. Organizations invest six figures in vendor platforms, assign implementation teams, conduct training sessions, and watch adoption collapse within months. Engineers return to Slack channels and tribal knowledge because the catalog provides worse answers than asking a colleague.

The failure pattern stems from misdiagnosis. Teams treat catalog implementation as a technology problem that requires vendor solutions, when metadata management fundamentally requires process change and behavioral integration. Technology alone cannot fix missing workflows, undefined ownership, or unclear documentation standards.

The three-pillar framework introduced in my Data Catalog Implementation article addresses root causes: Decoupled Architecture separates concerns that scale differently, Enforced Contracts make metadata non-optional through CI/CD integration, and Ruthless Curation focuses effort where business value concentrates. That framework explains why traditional approaches fail at 60,000+ tables.

Experienced practitioners choose DIY approaches for specific reasons. They need metadata validation integrated into existing CI/CD pipelines, not separate catalog interfaces. They require flexibility in the storage layer as infrastructure evolves. They want ownership of metadata schemas without vendor upgrade cycles forcing breaking changes. Most importantly, they recognize metadata management as a team capability issue that requires cultural change, not a technology gap that calls for vendor solutions.

The implementation pattern described here embodies all three pillars of the framework. Decoupled architecture separates metadata ingestion from storage and presentation. Enforced contracts make validation mandatory in deployment workflows. Ruthless curation implements tier-based documentation requirements that scale to production environments.

Below you’ll find complete, downloadable components for a working implementation: template structures, contract schemas, validation patterns, and ready-to-use automation scripts. I’ve tested these components locally in under 2 hours.

After the paywall: I’ll show you how to test everything locally, set up manual Google Sheets integration, and discuss when and why to consider migration paths or advanced patterns. These are production-tested components that work today.

Read more