Evaluating a Factor Model: Is It Fit for Purpose?

15.1 “Is it a good model?” is the wrong question

A factor model is not good or bad in isolation. It is fit or unfit for a purpose. The four applications of Chapters 9–13 stress different properties of the same object:

• Risk monitoring needs calibrated risk forecasts on portfolios like yours, at your horizon.
• Performance attribution needs a factor structure that matches the investment process, so that skill and tilt land in the right buckets.
• Portfolio construction needs forecasts that stay calibrated under adversarial pressure from the optimizer, plus a well-conditioned covariance.
• Hedging needs accurate factor covariances and cross-sensitivities out of sample, at trading horizon.

A model can excel at one and fail another: a slow, stable long-horizon model is excellent for IC reporting and useless for sizing tomorrow’s hedge. A model missing a granular value factor can forecast total risk superbly while systematically misattributing a value manager’s returns. So evaluation starts by writing down the use case, universe, horizon, the portfolios that will actually be analyzed, and only then choosing the yardsticks. This chapter supplies the yardsticks.

15.2 Statistical quality of the fit

The first battery: does the model describe the cross-section it was built on? (Necessary, never sufficient.)

Cross-sectional $R^2$ over time: From each period’s regression (Chapter 6). Evaluate the average level against the model’s peer class (monthly fundamental models: ~0.2–0.4), the time variation (an $R^2$ that collapses in crises means the model loses explanatory power exactly when needed), and most importantly the trend: a multi-year decay signals a factor structure drifting away from how the market now moves.

Specific-variance share: The per-name complement of the cross-sectional $R^2$: the fraction of a stock’s variance the model calls specific, $\Delta_{ii} / \Sigma_{ii}$?ΔN×N diag.specific variance matrix (diagonal)ΣN×Nasset covariance, Σ = XFXᵀ + Δ model-implied or the residual share of trailing realized variance, averaged over names and dates. Expect well over half for a typical large cap in a monthly fundamental model, more for small caps. Compute it by size and liquidity band: the WLS fit is dominated by large names (Chapter 6), so the headline $R^2$ can hold steady while the model quietly stops explaining the tail. A share drifting up over the years is $R^2$ decay seen from the individual stock.

Factor return significance: Per factor: the fraction of periods with $|t_k| > 2$, the sign-consistency of $\hat f_{kt}$, and its volatility relative to its standard error. A factor significant in 30%+ of months is structurally real. One significant at the 5%-ish base rate is noise. Flag for the Chapter 16 removal process.

Factor return autocorrelation: First-order autocorrelation of each $\hat f_{kt}$ series at the model’s horizon. Near zero is the healthy reading. Persistent positive values usually mean stale or asynchronous prices leaking into the estimation (illiquid names, time-zone-offset closes), and they break simple horizon scaling: multi-period risk built by scaling up $F$ is understated unless the autocovariance terms are carried along (Newey–West, Chapter 8). Genuine factor momentum exists, but the evaluation question is narrower: is the autocorrelation small enough at the horizon of use that variance scaling holds? If not, fix the price inputs or the scaling before trusting any aggregated forecast.

Residual diagnostics, the missing-factor detector: Under the model’s assumptions residuals are cross-sectionally uncorrelated. Test it: average pairwise residual correlations within candidate groupings (sub-industries, regions, thematic baskets, ownership clusters). Run a PCA on the residual covariance (Chapter 4). An eigenvalue above the Marchenko–Pastur noise edge is systematic risk the model is calling “specific.” Consequences cascade: specific risk is understated for clustered names, diversification of them is overstated, and attribution misclassifies their common return as stock selection.

Exposure stability: Period-to-period autocorrelation of $X$ columns, at the horizon of use. Noisy exposures (from volatile descriptors or aggressive winsorization choices) churn everything downstream: factor returns pick up noise, optimized portfolios trade for no reason, hedges drift. Momentum is legitimately fast. Value should not be.

Implied-portfolio churn and purity: The model’s own portfolios are diagnostics too. The pure factor portfolios ($\hat f = Pr$?fK×1factor returns over one periodPK×Npure factor portfolio matrix, f̂ = PrrN×1asset returns over one period, Chapter 7) and the fully invested minimum-variance portfolio ($\Sigma^{-1}\mathbf{1} / \mathbf{1}^\top \Sigma^{-1} \mathbf{1}$, the characteristic portfolio of a vector of ones) are deterministic functions of the model, so rebuilding them each period measures the model, not any manager. Two readings:

• Turnover: pure momentum legitimately turns over several hundred percent a year (Chapter 7). Comparable churn in a slow factor like value, or in the minimum-variance portfolio, is estimation noise in $X$, $F$, or $\Delta$, the same noise an optimizer converts into transaction costs.
• Purity decay: factor portfolio $k$ formed at $t-1$ had unit exposure to factor $k$ and zero to the others ($PX = I$?PK×Npure factor portfolio matrix, f̂ = PrXN×Kfactor exposure (loading) matrix). Recompute its exposures one period later, $X_t^\top p_{k,t-1}$. The rate at which the off-target entries grow sets the shelf life of any hedge or factor bet held between rebalances, Chapter 13’s hedge-ratio churn measured at the source.

15.3 Risk forecast accuracy: the core test

The bias statistic (introduced in Chapter 8) is the central instrument.

The statistic: For a test portfolio with forecasts $\hat\sigma_{t-1}$ and realized returns $r_t$: $b = \mathrm{std}(r_t / \hat\sigma_{t-1})$?bscalarbias statistic, std(rₜ / σ̂ₜ₋₁)rN×1asset returns over one periodσscalarvolatility / standard deviation over a window of $T$ periods. Calibrated ⇒ $b \approx 1$. $b > 1$ ⇒ underforecast. $b < 1$ ⇒ overforecast. 95% acceptance band $\approx 1 \pm \sqrt{2/T}$ (T=12: [0.59,1.41], a year tells you little. T=120: [0.87,1.13]). Watch rolling bias for regime behavior. The classic signature is $b$ drifting up through a calm-then-volatile transition (the EWMA forgetting problem, Chapter 8) and down after crises.

The mini example can supply one honest data point. Its three monthly active returns against the standing TE forecast (5.42% annualized, 1.565% monthly) give $b = 0.74$. The T = 3 acceptance band is [0.18, 1.82], so this says nothing. The lesson is that a quarter of data cannot fail a risk model; bias statistics only become informative over years.

Test portfolio design: Each kind probes a different weakness:

Test portfolios	What they test
Random long-only baskets	overall covariance level, the “easy” test a model must not fail
Single-factor tilt portfolios (pure factor portfolios, Ch. 7)	each row/column of $F$: find which factor’s risk is mis-scaled
Industry & country portfolios	the dummies’ blocks of $F$
Long–short style portfolios	factor correlations (these portfolios live off the off-diagonals)
Optimized portfolios (minimum-variance, Ch. 12 style under varying constraints)	the matrix’s weakest directions: the optimizer hunts for under-priced risk, bias here predicts real-world optimization failure
Your actual portfolios/strategy backtests	the only test that directly answers “fit for my purpose”

A model can pass the first four and fail the last two. That is the alpha-misalignment / error-maximization pathology of Chapter 12 made measurable. A vendor bias study on random portfolios is not evidence the model prices your strategy.

Supporting tests: Exceedance/coverage counts ($|r_t| > 2\hat\sigma_{t-1}$?rN×1asset returns over one periodσscalarvolatility / standard deviation in ~5% of periods? clustered in time?). Q-statistics/log-scores (the average log-likelihood of realized returns under each model’s forecast distribution) for ranking candidate models on identical test sets. Bias by segment (size, volatility, liquidity bands) to expose structural biases that portfolio-level averages hide.

Horizon discipline: A bias test at the model’s native horizon validates that horizon only. Daily-validated $F$ says little about quarterly risk (autocorrelation scaling, Chapter 8). Evaluate at the horizon you will consume.

15.4 Suitability by purpose

The purpose-specific checklists, distilled:

Risk monitoring: Bias ≈ 1 on portfolios resembling yours, at your horizon, including stressed sub-periods. Responsiveness matched to your decision frequency (half-life dial, Chapter 8). Coverage of your actual holdings (Chapter 5) without excessive proxy/imputation rates for your names.

Performance attribution: Granularity matched to the process: a deep-value shop needs the model to distinguish kinds of cheapness (composite value vs. separate B/P, E/P, yield descriptors, Chapter 3). A sector specialist needs industries at least as fine as their decision units. The audit: run attribution (Chapter 10) on your own history. If “specific” return is large, persistent, and correlated with an identifiable theme, the model lacks a factor your process trades, and its attribution will systematically mislabel your skill (in either direction). Specific-purity check: your portfolios’ residuals against the model should look like noise to you.

Portfolio construction: Everything risk monitoring needs, plus: bias ≈ 1 on optimized portfolios specifically (the tell-tale failure: realized risk of optimized books exceeding forecast while unoptimized books are fine). A well-conditioned $F$ (eigenvalue floor, shrinkage, Chapter 8). Alpha-alignment compatibility: can the model be augmented with your signals (Chapter 16)? An unconditioned or unaugmentable model transfers its weaknesses straight into your weights.

Hedging: Out-of-sample hedge effectiveness: for representative books, compute model-implied hedge ratios, then measure realized variance reduction vs. predicted (Chapter 13). Stability of hedge ratios period-to-period (ratio churn = transaction costs = a real cost of model noise). Accuracy of the specific cross-sensitivities used (beta-to-instrument, factor cross-betas) at trading horizon. This is where short-horizon variants earn their keep.

15.5 Head-to-head comparison

Choosing between models (vendor A vs. B vs. in-house) is an experiment. Design it like one:

1. Same everything: universe intersection, period, horizon, test portfolio set. Differences must come from the models, not the setup.
2. Out of sample: score forecasts made at $t$ against returns after $t$. Never let either model see the scoring window (vendors’ published statistics are in-sample by construction of their estimation history, discount accordingly).
3. Score card across criteria, weighted by your use case: bias suite (by portfolio type), log-scores, attribution fidelity on your history, coverage/imputation rates on your holdings, conditioning, operational factors (latency, support, cost, transparency of methodology).
4. Expect non-domination. A typically rational outcome: model A wins bias on optimized portfolios, model B wins attribution granularity for your process. The weights on the score card, your purpose, break the tie. And a “statistically worse” model can still be the right choice on coverage of your markets, transparency (a model you can interrogate beats a black box you can’t), or compatibility with your construction stack.

15.6 Ongoing monitoring and governance

Evaluation keeps going after the model goes live. The standing apparatus at a well-run shop:

• Dashboards: rolling bias statistics (by portfolio family), $R^2$ trend, factor significance counts, residual-PCA top eigenvalue, implied-portfolio turnover, imputation rates. Alert thresholds pre-agreed, not improvised mid-crisis.
• Drift detection: slow degradations, $R^2$ decay, a factor’s significance fading (“factor death”), bias trending, trigger the Chapter 16 change process. The hand-off rule: evaluation finds the symptom, modification is a separate, governed act.
• Model risk management (formalized in regulated firms, wise everywhere): documented methodology and limitations. A stated domain of validity: universes, horizons, regimes where the model is trusted. Escalation protocol for when reality exits that domain (crises, structural breaks, new market regimes), typically: widen risk buffers, lean on stress tests (Chapter 9) over covariances, shorten review cycles.
• Version governance: when the vendor (or your team) ships a model update, re-run the score card across versions, quantify forecast jumps on standing portfolios, and communicate before the risk numbers change under your PMs’ feet (Chapter 16, Section on recalibration).

15.7 Worked example: two candidates for the running mandate

The running mandate: monthly-rebalanced, benchmark-relative, value-tilted long-only book (Chapters 9–12). Candidates: Model A, the MiniModel (7 factors, monthly horizon). Model B, a hypothetical 4-factor variant (market + 3 industries only, no styles). Stipulated evaluation results, illustrating the method (the $R^2$ figures are realistic multi-year averages for a production cross-section, not the 0.96 the 10-stock toy prints on a single month, Chapter 6):

Criterion	Model A	Model B
Bias, random portfolios (10y monthly, band [0.87, 1.13])	1.04 OK	1.06 OK
Bias, value-tilted portfolios	1.02 OK	1.31 FAIL
Bias, optimized portfolios	1.12 (marginal)	1.38 FAIL
Residual PCA top eigenvalue	at noise edge OK	well above (a “value-like” residual factor) FAIL
Attribution of the running book	VALUE/MOM lines explicit (Ch. 10)	style P&L lands in “specific”
$R^2$ (centered, avg)	0.36	0.29

Both models pass the generic test, and only one can serve this mandate. Model B cannot see the book’s defining bets: it prices a value tilt as diversifiable specific risk (hence the 1.31 bias on exactly the relevant portfolios, it underforecasts the strategy’s risk by ~30%), and its attribution would have reported Chapter 10’s −72bp momentum accident as stock-picking. The diagnosis that drove the Chapter 12 repair would never have been made. Decision: Model A, with two governance notes: watch the marginal optimized-portfolio bias (add shrinkage before heavy optimization use), and revisit if residual diagnostics ever show the momentum-adjacent clustering that would suggest A itself is missing something. That is what “evaluated” means: a purpose-specific decision, documented and monitored.

15.8 Summary

• Fitness is relative to purpose: write the use case down first. Pick yardsticks second.
• Core batteries: fit statistics ($R^2$, specific-variance share, factor significance and autocorrelation, residual structure, exposure stability, implied-portfolio churn and purity) and forecast calibration (bias statistics on a designed spectrum of test portfolios: random -> tilted -> optimized -> yours).
• Purpose-specific killers: attribution needs factor granularity matched to the process. Optimization needs calibration under adversarial selection. Hedging needs out-of-sample effectiveness at trading horizon.
• Compare models with controlled, out-of-sample, score-carded experiments. Govern the chosen model with dashboards, drift alerts, a stated domain of validity, and hand symptoms to the modification process of the next chapter.

Try it: in section 6 of mini_example.py, change month 2’s specific active return in spec_active from 0.30 to 3.30 and rerun the section-11 bias demo. b jumps to 1.85, finally poking past the band’s edge: it takes a single specific month twice the size of the entire monthly TE forecast before three observations can reject a model.