numpy (pip install numpy) -- no GPU, no PyTorch, because today is about process, not tensors;Learn AI Series):I ended episode #134 on a deliberately unsatisfying note. We had spent the whole thing learning to calculate our way through infrastructure -- GPU rates, spot instances, the on-prem crossover, make-versus-buy reduced to arithmetic you can actually run. And then I pointed at the thing the spreadsheet cannot price: that every one of those clean decisions quietly assumed someone owned the shutdown script, someone chose the provider, someone was actually allowed to say "we build this in stead of renting it". Compute you can budget. The humans around the compute you cannot, and that is exactly where most AI projects go to die.
So today there is almost no math and no model at all. Today is about the messy, unglamorous stuff that decides whether all the technical skill this series has drilled into you actually ships anything: roles, lifecycle, documentation, review, debt, and the organizational landmines nobody warns you about. If you are a team of one -- and quite some of you reading this are -- do not tune out. Process is not bureaucracy for solo builders; it is insurance against your own forgetting. Here we go ;-)
Homework first, as always -- full code, no hand-waving.
Exercise 1 -- turn the estimator into a budget tool. The task was to take a fixed budget ceiling and, for every provider/instance in the rates table, print how many GPU-hours that budget buys (on-demand and spot), then declare which single option lets you train the longest.
def budget_hours(budget, use_spot=False):
"""For a fixed budget, how many GPU-hours does each option buy?"""
rates = { # per-GPU $/hr, on-demand, early 2026
("A100", "aws"): 4.00, ("A100", "gcp"): 3.67, ("A100", "lambda"): 1.25,
("H100", "aws"): 12.25, ("H100", "gcp"): 11.00, ("H100", "coreweave"): 3.00,
("L4", "aws"): 0.80, ("L4", "gcp"): 0.70,
}
best, best_hours = None, -1.0
for (gpu, prov), rate in rates.items():
r = rate * 0.3 if use_spot else rate # spot ~ 70% off
hours = budget / r
if hours > best_hours:
best, best_hours = (gpu, prov), hours
print(f"{prov:>9} {gpu:<5} {'spot' if use_spot else 'on-demand':<9} "
f"${r:>6.2f}/h -> {hours:>7.1f} GPU-h")
print(f"LONGEST under ${budget:,.0f}: {best[1]} {best[0]} ({best_hours:.1f} GPU-h)\n")
return best
budget_hours(1000, use_spot=False)
budget_hours(1000, use_spot=True)
The one sentence: with a flat dollars-per-hour table the "longest" option and the "cheapest per hour" option are always the same thing -- lowest rate mechanically buys the most hours -- and they only diverge once you weight hours by throughput, because an H100 at triple the price but quadruple the speed finishes more actual work per dollar even though it buys you fewer raw hours on the clock.
Exercise 2 -- model your own crossover. Plug a realistic monthly query volume and today's API price into api_vs_custom, find the breakeven month, then redo it assuming traffic grows 15% month-over-month in stead of staying flat.
def api_vs_custom_growth(q0, api_cost, training_cost, infra_monthly,
months=24, growth=0.0):
"""Cumulative API spend vs. own-the-model, with optional monthly traffic growth."""
api_cum, q, breakeven = 0.0, q0, None
for m in range(1, months + 1):
api_cum += q * api_cost # this month's API bill
custom_cum = training_cost + infra_monthly * m
if breakeven is None and custom_cum < api_cum:
breakeven = m
q *= (1 + growth) # next month's volume
tag = f"{growth:.0%}/mo"
print(f"[{tag:>7} growth] breaks even at "
+ (f"month {breakeven}" if breakeven else "never within horizon"))
return breakeven
# 100k queries/month at $0.03, $5k to train, $500/mo to host
api_vs_custom_growth(100_000, 0.03, 5000, 500, months=24, growth=0.00)
api_vs_custom_growth(100_000, 0.03, 5000, 500, months=24, growth=0.15)
At flat traffic the custom model overtakes the API somewhere around month three; switch on 15% monthly growth and that crossover marches earlier, because every extra query you serve widens the gap the API is charging you per call while your training cost sits fixed. The lesson is not the exact month -- it is that growth is a thumb on the scale that always pushes toward owning the thing, and the faster you grow the sooner "rent" turns into "buy".
Exercise 3 -- make the idle guard real-ish. Rewrite the shutdown guard so it consumes a stream of simulated utilization readings across a whole week, kills the box after patience idle checks, and reports the dollars saved versus a machine left humming the entire week at $32/hr.
import numpy as np
def simulate_week(patience=6, low=0.05, check_every_min=10, hourly=32.0, seed=0):
rng = np.random.default_rng(seed)
checks = 7 * 24 * 60 // check_every_min # one week of periodic checks
idle_streak, minutes_billed, shut = 0, 0, False
for _ in range(checks):
if shut:
continue # box is off -> stops billing
# mostly idle, with an occasional genuine burst of work (~20% of checks)
util = rng.uniform(0.4, 0.9) if rng.random() < 0.2 else rng.uniform(0.0, 0.03)
minutes_billed += check_every_min
idle_streak = idle_streak + 1 if util < low else 0
if idle_streak >= patience:
shut = True
hours_billed = minutes_billed / 60
full_week = hourly * 7 * 24
saved = full_week - hours_billed * hourly
print(f"patience={patience:>2}: billed {hours_billed:>5.1f}h "
f"(${hours_billed*hourly:>6,.0f}) saved ${saved:>6,.0f} of ${full_week:,.0f}")
return saved
for p in (3, 6, 12):
simulate_week(patience=p)
One sentence on choosing patience: set it so that patience * check_every_min comfortably exceeds the longest legitimate idle gap in your workload -- the pause between epochs while validation and checkpointing run, say -- otherwise the guard will cheerfully execute a job that was merely catching its breath between rounds. Right, homework done. Now let us talk about the people who were supposed to write that shutdown script.
In traditional software the roles are worn smooth by decades of use -- frontend, backend, DevOps, QA, everyone knows roughly where the lines are. AI teams are younger, the titles are mushier, and the responsibilities overlap in ways that cause real friction. Here is the honest breakdown.
Data Scientist. Explores data, builds prototypes, runs experiments. Lives in notebooks, fluent in statistics and visualization, brilliant at finding signal and turning a business question into something measurable. The classic weakness: produces code that sings in a notebook and falls over the instant it meets production traffic.
ML Engineer. Takes that prototype and makes it survive -- training pipelines, serving infrastructure, performance work. This is the bridge between "it worked on my laptop" and "it serves a million users". Everything we slogged through in episodes #117-#126 is this person's daily life.
Data Engineer. Builds and guards the pipelines that feed the models -- ETL, data quality, feature stores, versioning (episode #118 was the technical half of this job). Underinvest here and your ML engineer is quietly training on stale or corrupted data and nobody notices until the metrics fall off a cliff.
MLOps Engineer. Owns the operational surface: CI/CD for models (episode #124), monitoring (episode #123), rollouts, drift alerts. The role exists because shipping a model is genuinely not the same problem as shipping a web service -- models rot, and someone has to watch them rot.
Research Scientist. Reads papers, implements new architectures, pushes the frontier. Mostly a luxury of larger labs; in a small team the data scientist or ML engineer just wears this hat on Fridays.
The single most common staffing blunder -- and I have watched it play out more than once -- is hiring a pile of data scientists and no engineers. You end up with a team that can prototype anything and ship nothing, a beautiful gallery of demos that never reach a user. As a rough starting ratio for a product team, think something like one data scientist to two ML engineers to one data engineer. Having said that, the ratio matters far less than the principle: prototyping capacity without shipping capacity is a very expensive way to produce slide decks.
Software has a development lifecycle. ML has one too, but the shape is different -- it curls back on itself, because models degrade and data moves. Six stages:
A tiny state tracker -- the kind of unglamorous scaffolding teams actually build so nobody has to ask "wait, which stage is the churn model in?":
from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional
@dataclass
class ModelLifecycle:
"""Track a model through its lifecycle stages."""
model_name: str
problem_statement: str
owner: str
created: datetime = field(default_factory=datetime.now)
stage: str = "framing" # framing, data, experiment, production, deployed, retired
metrics: dict = field(default_factory=dict)
data_version: Optional[str] = None
model_version: Optional[str] = None
notes: list = field(default_factory=list)
def advance(self, new_stage, note=""):
self.notes.append(f"[{datetime.now():%Y-%m-%d}] {self.stage} -> {new_stage}: {note}")
self.stage = new_stage
def record_metric(self, name, value, split="test"):
self.metrics.setdefault(f"{split}/{name}", []).append(
(datetime.now().isoformat(), value))
def summary(self):
print(f"Model: {self.model_name} (stage: {self.stage}) owner: {self.owner}")
print(f"Problem: {self.problem_statement}")
for key, values in self.metrics.items():
ts, val = values[-1]
print(f" {key}: {val:.4f} ({ts[:10]})")
churn = ModelLifecycle("churn-predictor-v2",
"Predict 30-day churn probability for active subscribers",
owner="ml-team")
churn.advance("data", "Historical data extracted, labels verified")
churn.advance("experiment", "Baseline gradient-boosted tree running")
churn.record_metric("auc_roc", 0.847)
churn.record_metric("precision_at_10pct", 0.72)
churn.summary()
It looks almost too simple to bother with. That is the point -- the teams that skip this are the ones where, three months in, nobody can say with a straight face which model is live, what it was trained on, or when it last got looked at.
A model card is a short, standardized document describing a model: what it does, how it was trained, where it works, and -- the part everyone skips and everyone regrets skipping -- where it fails. Google put the idea forward back in 2019 and it has since become basic hygiene for responsible deployment.
Why bother? Because six months from now, when the model starts misbehaving at 2am, someone needs to know what data it saw, what its known limits are, and what evaluation was actually done. That someone might be a colleague who has never seen the code. More likely it is you, having comprehensively forgotten every detail. Write it down while you still remember.
model_card = {
"model_name": "churn-predictor-v2",
"version": "2.1.0",
"date": "2026-02-15",
"type": "Binary classification (gradient-boosted trees)",
"description": "Predicts probability of customer churn within 30 days.",
"training_data": {
"source": "internal CRM database, Jan 2023 - Dec 2025",
"size": "245,000 customers, 18 features",
"label_definition": "Customer inactive for 30+ days after observation date",
"known_gaps": "Missing partner-channel signups before 2024",
},
"evaluation": {
"test_set": "Jan-Mar 2026, 32,000 customers (temporal split)",
"auc_roc": 0.847,
"precision_at_10pct_recall": 0.72,
"f1_score": 0.68,
},
"intended_use": "Prioritize retention outreach for at-risk customers",
"out_of_scope": "Do NOT use for credit decisions or service denial",
"limitations": [
"Lower accuracy for customers with < 3 months history",
"Not validated on enterprise accounts (trained on consumer only)",
"Degrades if the pricing structure changes significantly",
],
"ethical_considerations": [
"Age and location features may introduce demographic bias",
"Regular fairness audits required across customer segments",
],
"maintenance": {
"retraining_schedule": "Monthly on a fresh 3-month window",
"monitoring": "Daily AUC tracking, weekly drift detection",
},
}
Every field earns its place, but the limitations and out_of_scope sections are the ones that actually save you -- they tell a future user what not to trust the thing for. A model card without a limitations section is a sales brochure, not documentation. And because a document nobody validates rots into fiction, it is worth a five-line guard that simply refuses to let an undocumented card through review:
REQUIRED = ["model_name", "version", "training_data",
"evaluation", "intended_use", "limitations"]
def validate_card(card):
missing = [k for k in REQUIRED if not card.get(k)]
if missing:
raise ValueError(f"model card incomplete -- missing: {missing}")
if not card["limitations"]:
raise ValueError("a model card with an empty 'limitations' list is a lie")
print(f"model card OK: {card['model_name']} v{card['version']}")
validate_card(model_card)
Wire that into CI and "we will document it later" quietly stops being an option -- the build fails until the card is honest. Nota bene: the same idea applies to datasets, where the equivalent artifact is called a datasheet (origin, collection method, consent, known biases). Same discipline, different noun.
Standard code review catches bugs, style slips, broken logic. ML review needs all of that plus a handful of checks that ordinary review sails straight past -- and the nastiest ones are the ones where the code is perfectly correct and the reasoning is wrong.
Data leakage. Is test data bleeding into training? Are features computed using information from the future? Was the scaler fitted on the whole dataset in stead of just the training fold? This is the most common and most devastating ML bug in existence, and it slides past normal review precisely because the code looks fine -- there is no crash, no exception, just a validation score that is a beautiful, seductive lie.
Reproducibility. Are the random seeds set? Run it twice, do you get the same answer? Can a colleague rebuild your result from the code and config alone? Episode #119 handed you the tools; review is where you enforce that they were actually used.
Metric selection. Is the metric right for the business problem? Accuracy on an imbalanced dataset (episode #13, forever haunting us) passes review and ships a useless model. The reviewer has to understand the problem, not just read the diff.
Training-serving skew. Does the feature computation at training time exactly match the one at serving time? "days_since_last_purchase" computed one way in the batch pipeline and another way at inference -- a timezone here, a null handled differently there -- and your live model silently sees features it was never trained on.
review_checklist = {
"data_leakage": [
"Preprocessing fitted ONLY on the training split",
"No future-derived features in a time-series context",
"Target variable not accidentally leaking into the features",
"Holdout set truly held out (used for no decisions at all)",
],
"reproducibility": [
"Random seeds set for every source of randomness",
"Data version pinned (a hash or a snapshot date)",
"Dependencies pinned (requirements.txt with versions)",
"Hyperparameters logged, not buried in a comment",
],
"evaluation": [
"Metric matches the business objective",
"Realistic split (temporal for time-series, not random)",
"A baseline to compare against",
],
"production_readiness": [
"Training and serving feature logic are identical",
"Edge cases handled (nulls, out-of-range, unseen categories)",
"Model size and latency inside the deployment budget",
"Monitoring hooks actually wired in",
],
}
def review_gaps(pr_claims):
"""pr_claims: the set of checklist items the author swears they handled."""
for area, items in review_checklist.items():
missing = [it for it in items if it not in pr_claims]
status = "OK" if not missing else f"{len(missing)} UNCONFIRMED"
print(f"{area:<20} {status}")
for it in missing:
print(f" - {it}")
review_gaps(pr_claims=set()) # nothing confirmed yet -> the full checklist lights up
Run that against an empty set and every item lights up red, which is exactly the state every pull request starts in. The checklist is not there to slow you down -- it is there so the leakage bug gets caught in review, where it costs a comment, in stead of in production, where it costs a quarter.
Google's much-cited 2015 paper, "Hidden Technical Debt in Machine Learning Systems", named something every practitioner feels but few articulate: ML systems accumulate debt faster than ordinary software, and the debt is far harder to see. The reason is that most of it breaks silently.
Data dependency debt. Your model leans on an upstream pipeline. That pipeline changes its schema one Tuesday. Your model keeps running, keeps returning numbers, and quietly serves garbage until accuracy tanks days later. Code dependencies break loudly -- an import fails, a build goes red. Data dependencies break in a whisper.
Configuration debt. The sheer number of hyperparameters, feature flags, data paths and environment knobs in an ML system is enormous, and a single wrong one -- a stale data path, a learning rate off by a zero -- produces a model that trains successfully and performs terribly. It does not crash. It just gets worse, and you go looking for a bug in the model when the bug is in a YAML file.
Experimental debt. Every untracked experiment, every "temporary" hack that put down roots, every model variant living in someone's home directory but not in the registry. Over a year the team forgets what was tried, what worked, and why any given decision was made -- institutional amnesia with a loss curve.
Feedback-loop debt. When a model's own predictions shape the data it is later trained on. A recommender that only surfaces popular items gathers data proving popular items get clicked, which trains the next model to push popular items even harder -- a system optimizing itself into a filter bubble, one retrain at a time.
The cure is not zero debt -- that is a fantasy. It is making debt visible and paying it down on purpose: track experiments (episode #119), version your data, document decisions, review on a cadence. A quick sketch of the "quiet" part made loud -- a guard that screams when an upstream schema drifts before the model eats the bad data:
def check_schema(expected, incoming):
"""Fail loudly the moment an upstream data contract changes."""
missing = set(expected) - set(incoming)
added = set(incoming) - set(expected)
if missing or added:
raise RuntimeError(f"schema drift! missing={missing or '-'} added={added or '-'}")
print("upstream schema OK")
expected_features = {"age", "tenure_months", "monthly_spend", "n_logins_30d"}
todays_feed = {"age", "tenure_months", "monthly_spend"} # someone dropped a column
check_schema(expected_features, todays_feed) # -> RuntimeError, not silent rot
Eight lines turns a silent, week-long accuracy bleed into a loud crash at 9am on the day it happens. That trade -- loud-and-early over quiet-and-late -- is the whole philosophy of managing ML debt in one idea.
The technical failures are the ones you can fix. The people failures are the ones that sink projects.
Expectation management. Stakeholders expect AI to be a magic wand -- "just train a model to predict X". They do not see the data, the labels, the evaluation, the iteration, the endless maintenance. Setting honest expectations upfront ("three months to a prototype, six to production, then it never really stops") is unglamorous and it is the single highest-leverage thing you can do to keep organizational trust intact.
The proof-of-concept trap. A data scientist builds a dazzling demo in two weeks. Management sees it, greenlights production, and budgets none of the six months of engineering needed to make it reliable. The demo becomes production, held together with duct tape and optimism, and the team spends the next year firefighting in stead of improving. I have seen this exact movie more than once, and it never ends well.
Incentive misalignment. Data scientists are rewarded for trying new things; ML engineers are rewarded for keeping things stable. Those goals pull against each other by design. A healthy team names the tension out loud and builds process that balances novelty against reliability, in stead of pretending everyone wants the same thing.
The "just retrain" reflex. A model degrades and the instinct is to retrain on fresher data. But if the real cause is a data-quality issue, a concept shift, or a feedback loop, retraining on bad data makes it worse. Diagnosing why a model degraded is harder than training a new one -- and that diagnostic skill is precisely what separates a senior practitioner from a junior with a .fit() habit.
Working alone -- as quite some of you are -- the same principles apply, just scaled down:
The solo builder's signature failure is the "it is all in my head" trap. Everything runs fine until you take a two-week break and come back to a system you no longer understand. Process is not corporate ritual. It is a note passed forward to a version of you who has forgotten everything -- and that person always shows up eventually.
Three to gnaw on before next time -- and, as always, I will walk through full solutions at the top of the following episode.
Grow the lifecycle tracker a spine. Extend ModelLifecycle so that advance() refuses illegal transitions -- define the legal order (framing -> data -> experiment -> production -> deployed -> retired), allow going backwards to "data" or "experiment" (that is iteration, and it is legal), but reject nonsense like jumping straight from "framing" to "deployed". Add a days_in_stage() helper and a one-line health check that flags any model stuck in "experiment" for more than, say, 60 days. One sentence on why "stuck in experiment" is a smell worth alerting on.
Write your own model card -- and break it. Take any model you have actually trained in this series (the churn tree, a CNN from #46, anything) and fill in a real model_card dict for it, honestly. Then run validate_card against it. Now deliberately empty the limitations list and confirm the validator rejects it. Write two sentences: one naming a genuine limitation of your model that you would not have written down without the card forcing you to, and one on why that omission would have bitten a future user.
Simulate feedback-loop debt. Build a tiny recommender that always recommends the current top-3 most-clicked items, then simulate 20 rounds where each round's "clicks" are drawn mostly from whatever was recommended (with a small chance of a random discovery). Track how the diversity of recommended items changes over the rounds and show the system collapsing toward a handful of winners. One sentence on the cheapest intervention (hint: think exploration, and cast your mind back to episode #103) that would keep the catalogue from strangling itself.
We have spent this whole series building the pieces -- the math, the models, the pipelines, the infrastructure, and today the humans and process that hold it all together. So here is the question I want you carrying into next time, and it is a big one: could you take everything from the last hundred-odd episodes and stand up a single, coherent, end-to-end system -- data in one side, a monitored, documented, retrainable model serving predictions out the other, with the roles and the review and the lifecycle all wired in? Not a notebook. A platform. That is where we head next, and we are going to build it ;-)
@scipio