Part of the The Bayesian Investor — A Primer for the CIO series.
Phase 3 — From the Committee to the GPS: Tracking Things That Move
There is one problem with the hiker-in-the-fog approach: it is thorough, but it is slow. The hiker needs tens of thousands of steps to map the landscape. For a one-time analysis, that’s fine. But what if the landscape itself is moving? What if you need to track an unobserved target in real time — the market’s hidden volatility, the “true” inflation rate, the fair value of an asset that is only indirectly observable?
This phase is about the pivot: from a slow, thorough committee review to a fast, streaming instrument.
Why the pivot is necessary
Think of the difference between a formal committee review and a GPS navigation system.
A committee gathers, considers all the evidence, deliberates, and produces a considered judgment. It is excellent — but it is slow, and by the time it finishes, the situation may have changed. A GPS does something different: it never stops updating. Every second it receives new measurements, fuses them with its current estimate, and produces a current position with a circle of uncertainty around it — and it does this continuously, forever, as the target moves.
Some economic targets are committee-shaped (you analyze them occasionally, in depth). But many — especially volatility, the market’s hidden fear gauge — are GPS-shaped. They change continuously, they’re never directly observable, and decisions need a now answer, not a last-quarter answer. This phase builds the instruments for GPS-shaped problems.
The Kalman filter: the GPS itself
The centerpiece is the Kalman filter, and the analogy is almost too good: it is a GPS, mathematically.
Here’s the problem it solves. You have many noisy sensors — each one individually unreliable. A GPS has satellite signals that bounce and drift; your sensors might be price data, option-implied signals, transaction prints, all of them noisy. You also have a belief about where the target is and how it’s moving. How do you combine all of it into one good estimate?
The Kalman filter does two things in a loop, forever:
- Predict: “Given where I think the target is and how it’s moving, where should it be now?” (The bookmaker’s prior, updated for time passing.)
- Correct: “Now the sensors have spoken. How much should I trust them versus my prediction?”
And here’s the elegance: the filter weighs each source by how trustworthy it is. A very noisy sensor gets little weight; a very precise one gets a lot. If the sensors suddenly disagree wildly with the prediction, the filter notices and starts trusting them less. It is self-correcting — a smooth, continuously updated estimate that never panics and never ignores evidence.
For your firm: this is how you track an unobservable target like market volatility — the hidden state of fear and calm that drives option prices and risk budgets — in near real time, from noisy observable signals, with an honest uncertainty circle that shrinks as the sensors get better.
The particle filter: thousands of tiny scouts
The Kalman filter is brilliant but has limits: it assumes the target moves in fairly smooth, well-behaved ways. What if the truth is wilder — jumping regimes, sudden shifts, non-linear dynamics? Enter the particle filter, which is the scout version.
Instead of one estimate with one uncertainty circle, imagine dispatching thousands of tiny scouts, each carrying a guess about where the target is. The scouts wander according to your model of how the target moves. Then the evidence arrives — and each scout is graded on how well its guess matches what was actually observed. The scouts with good guesses survive and multiply; the ones with bad guesses die off. Repeat, and the surviving scouts crowd around the truth, wherever it is — even if the truth jumps around in wild ways.
It’s evolution, applied to estimation: survival of the fittest guess. Where the Kalman filter is the elegant GPS for smooth motion, the particle filter is the rugged tracker for unpredictable motion. Your team will learn both, because real markets do both: sometimes smooth, sometimes jumpy.
The payoff: volatility, and beyond
The concrete finance payoff in this phase is dynamic tracking of hidden, time-varying market volatility — the market’s shifting mood of fear and greed that no one can observe directly but everyone’s decisions depend on. With Kalman and particle filters, your team can estimate it continuously, in real time, from the noisy signals the market leaves behind, with an honest statement of how sure the estimate is at every moment.
And volatility is just the first target. The same machinery tracks any hidden, moving economic quantity: the “true” rate of inflation, the fair value of an asset whose worth is only glimpsed through noisy transactions, the health of a property market read through noisy data feeds. Any time the thing you care about is invisible but moving, this is the toolkit.