The model was not broken. The world the model predicted changed when Zillow stopped observing and started intervening.
On November 2, 2021, Zillow announced it was winding down Zillow Offers, its algorithmic home-buying business, after suffering a $381 million loss in the third quarter of that year. About 27,000 homes had been purchased through the program; thousands were now flagged for sale below cost. The company laid off roughly 2,000 people — a quarter of its workforce. The sting came not from algorithmic naivety, but from a failure so structural that it hides in plain sight: the model had been trained on a world in which Zillow was a small participant, observing prices. When Zillow became a major buyer — tens of thousands of homes annually — it stopped observing the market and started changing it. Its purchases moved local prices. Its renovation demand drove up labor costs. Its inventory backed up. The model had no mechanism to understand any of this. It kept making predictions with the same confidence it had earned when those predictions didn't have to survive the consequences of acting on them.
Understanding when and why a model fails under intervention separates organizations that learn from their data from those that discover $381 million gaps too late.
Every recommendation a predictive model makes is, at its foundation, a claim about what will happen when you act on it. Yet nearly every consequential failure of predictive analytics in organizational decision-making follows the same structural pattern: the model was trained on data collected while the organization was behaving one way, and deployed in a scenario where the organization behaves differently. The model's accuracy metrics — its precision, its recall, its confidence intervals — all came from a world in which the model's predictions didn't have to survive the consequences of being acted upon. When you change what you're doing, you change the world the model is trying to predict. The model sees this as noise. You see it as bankruptcy. This chapter teaches you to diagnose that gap before it manifests in eight-figure losses.
The promise is silent but absolute: new situations are like old situations in the ways that matter. When that promise holds, the model works. When it breaks, you have confident wrongness — worse than uncertainty, because confidence suppresses the doubt that triggers checking.
Every predictive model, regardless of sophistication, follows the same three-step process. First, it observes a historical record of situations paired with outcomes. Second, it fits a function — a mathematical relationship — that captures how the features of each situation relate to the outcome that followed. Third, it uses that function to predict outcomes in new, unseen situations. Implicit in this entire architecture is a single assumption, so foundational that it is rarely named: the new situations will be statistically similar to the old situations in all the ways that matter for prediction. This assumption holds perfectly in stable environments where nothing the model predicts changes how the world behaves. But organizational decision-making almost never occurs in stable environments. When you act on a model's prediction, you change the context in which that prediction was made.
In an observational distribution, P(Y | X) tells you: if we see X occur naturally, what Y typically follows. It does not tell you what Y will follow if you deliberately set X.
A predictive model trained on historical data is learning from what statisticians call the observational distribution. This is the distribution of outcomes that results when the world unfolds without your deliberate intervention on the variables you're trying to predict from. In Zillow's case, the training data captured home prices, renovations, resale timelines, and market movements under a regime where Zillow was one of many participants, responsive to market conditions but not a dominant actor. The relationships between home features, market conditions, and resale prices made sense in that world. The model learned those relationships accurately. It could predict resale prices well because the mechanisms generating those prices — supply and demand in a market where Zillow was not the dominant buyer — remained stable. The observational distribution P(Y | X) means: if we observe X in the natural world, what Y typically follows. It means nothing about what Y will follow if we intervene to set X.
In an interventional distribution, P(Y | do(X)) is fundamentally different from P(Y | X). The model trained on observation has no data on what happens when you act.
When Zillow moved from observing the market to intervening in it at massive scale, it entered what statisticians call the interventional distribution. This is the distribution of outcomes that results when you deliberately set or change a variable, rather than simply observing it. When Zillow bought tens of thousands of homes, it didn't just observe the market — it changed the supply of homes available, increased demand for renovation contractors, backed up its own processing pipeline, and altered price expectations in neighborhoods where it was concentrating purchases. These consequences didn't appear in the training data because Zillow had never intervened at that scale before. The observational distribution and interventional distribution can look identical in theory but behave completely differently in practice. P(Y | X) — the probability of Y given that you observe X — is not the same as P(Y | do(X)) — the probability of Y given that you deliberately intervene to set X. The model was trained on the first. It was deployed in the second.
The do-operator logic asks: will intervening on this variable break relationships that held in the observational data?
Statisticians and causal inference researchers use the do-operator notation P(Y | do(X)) to denote the probability of an outcome Y when you deliberately intervene to set X to some value, as opposed to simply observing X. This distinction is not pedantic. It marks the deepest failure mode in applied predictive modeling. Consider Zillow's case concretely. In the observational distribution, home price was related to features like square footage, location, market conditions, and inventory levels. When those variables took on certain values naturally, certain prices followed. But when Zillow intervened to buy homes at scale, it didn't just observe that relationship — it changed it. The intervention on "homes Zillow buys" severed paths in the original causal graph and opened new ones. Zillow's own demand for contractors altered labor costs. Its inventory concentration altered supply expectations. The historical relationship P(resale price | features, market conditions) no longer held under P(resale price | do(scale up home purchases)).
Zillow faced all three, but intervention was the core failure: the model could not account for how its own actions changed the world it was predicting.
Divergence between training and deployment environments happens through three distinct mechanisms, each leaving different fingerprints in the data. Concept drift occurs when the world itself changes — consumer preferences shift, economic conditions evolve, competitors emerge, seasons turn. The relationship between features and outcomes can remain unchanged, but the distribution of outcomes drifts as the world's behavior drifts. Covariate shift occurs when the distribution of input features X changes, but the relationship between X and Y remains stable. You might be training on a uniform distribution of neighborhoods and deploying on a concentration in expensive urban areas — same relationship, different region of the feature space. Intervention, the most damaging, occurs when your organization deliberately modifies X, which severs paths in the causal graph that existed in the observational data and activates new causal mechanisms the model never encountered. Zillow faced all three simultaneously: drift in interest rates, shift in the mix of homes bought, and direct intervention through scale changes that altered contractor costs and inventory dynamics.
Zillow's iBuying model was trained on observational data from a market where Zillow bought homes opportunistically. The historical relationship P(resale price | home features, market conditions, renovation costs) was accurate for that regime. When Zillow scaled to tens of thousands of purchases monthly, it intervened. Its purchasing volume moved prices. Its demand for contractors and construction workers pushed up labor costs beyond what the model had assumed. Its inventory of homes awaiting resale exceeded its processing capacity; timelines stretched from the modeled three-to-six months to much longer. Market participants began to anticipate Zillow's buying and adjusted their own behavior — a feedback loop entirely absent from the training distribution. The model kept generating price estimates with the same confidence intervals it had earned when predicting in a world where the model's decisions didn't cascade back into the mechanisms it was trying to predict. The model was mathematically sound. The failure was structural: it lacked any mechanism to represent how its own recommendations, once executed at scale, would transform the world it was modeling.
Zillow's model had proven itself on historical data. That very track record became the mechanism by which false confidence spread through an organization that was about to lose $381 million.
The most insidious failure mode in organizational analytics is not a model that performs poorly; it is a model that performs well on training data and then fails catastrophically in deployment. Poor performance breeds skepticism. It prompts questions. It triggers checking. But a model with high accuracy and small confidence intervals earns trust. When that model makes recommendations, the organization acts on them with conviction. The evidence of past success becomes a shield against doubt. This is precisely what happened at Zillow. The model had worked for years. Its predictions of home prices and resale timelines were, by machine learning standards, accurate. The organization had validated it thoroughly. So when the model recommended buying homes at ever-increasing scale, leadership acted without hesitation. The model's proven track record became the mechanism by which the organization scaled into a regime where the model's assumptions no longer held. Uncertainty would have prompted caution. Confidence prompted conviction. And by the time the mismatch became visible in Q3 2021, Zillow had already committed to a strategy that could not be unwound without massive losses.
The Living Model architecture is built to surface this gap before it costs millions. It monitors whether the relationships the model learned are still holding when decisions are executed. It asks, in real time: is the world the model is operating in still the world the model was trained on? If not, why not, and what does that mean for the next decision?
Nearly every consequential failure of predictive analytics in organizational decision-making follows the same structural pattern. A model is trained on historical data collected in a regime where the organization's behavior was set by prior decisions and market conditions. The model learns the relationships that held in that regime. It is validated thoroughly. Its accuracy is demonstrated. Then it is deployed in a scenario where the organization uses the model's recommendations to change what it does. This change in organizational behavior alters the world the model is trying to predict. Relationships that held under observation break under intervention. New causal mechanisms activate. Feedback loops that were dormant in the training distribution become active in the deployment distribution. The model has no language for any of this. It makes predictions with the same confidence it had earned when predicting in a more stable world. The Innovator's Dilemma is not primarily a perception problem — it is an incentive and time-horizon problem. Organizations with a working business model face strong incentives to stick with it, and they move faster by accumulating small scale-up wins than by pausing to check whether the next ten-fold increase will break the fundamental assumptions on which those wins were built.
The difference between a model that works and a model that costs you $381 million is not mathematical sophistication. It is the willingness to ask, in real time, whether the world the model predicted is the world the model is operating in. Living Models build this question into the system itself.
The central claim of this chapter is that every predictive model — no matter how accurate, no matter how carefully validated — is fundamentally a model of observation. It captures relationships that held true in a world the model was observing, not acting upon. When an organization uses a model's recommendation to change what it does, it enters a different statistical regime: the interventional distribution. The relationships that held in the observational distribution can break. Causal paths can sever. New mechanisms activate. The model has no data on this regime. It cannot predict its own consequences. This is not a failure of the modelers. It is a structural property of learning from historical data. The only antidote is explicit, continuous vigilance: monitoring whether the relationships the model learned are still holding as the model's recommendations reshape the world. The Living Model is built around this vigilance. It doesn't solve the problem of intervention. But it makes the problem visible before it becomes a $381 million loss.
Living Models · Chapter 2 · The Map That Doesn't Move