Feature Importance Explainer

Feature importance assigns each input variable a scalar score representing its contribution to a model prediction, but the three dominant definitions are not interchangeable. Mean decrease in impurity, often called Gini importance, sums the weighted impurity reduction a variable produces across every split in a tree ensemble; it is cheap because it falls out of training, but it is biased toward continuous and high-cardinality features. Permutation importance shuffles one column in held-out data and measures the resulting drop in a chosen metric such as R-squared or log loss; it is model-agnostic but understates correlated pairs, because a shuffled feature leaves its information intact in its partner. SHAP values distribute a single prediction among features using the Shapley solution from cooperative game theory, producing additive local attributions that sum exactly to the prediction minus the base value. The exponential decay profile used here encodes an assumption that importance in tabular models is heavily concentrated: with this page's defaults of ten features at 0.75 average correlation, the top three carry about 73 percent of the total. That shape is a modelling choice meant to resemble the concentration commonly seen in tabular problems, not a measurement taken from any particular collection of datasets. Reliability scales with the samples-per-feature ratio, where roughly 30 rows per feature is a floor for stable ranking and 100 or more is preferred before permutation scores are trusted. Average pairwise correlation above about 0.8 signals redundancy, the regime in which two collinear columns split credit and both look weak. Importance is not causation, and scores computed on training rather than held-out data mostly measure memorization.

The tree-based branch originated with Classification and Regression Trees, published in 1984 by Leo Breiman, Jerome Friedman, Richard Olshen and Charles Stone, which formalized impurity reduction as a byproduct of splitting and used it to rank variables. Breiman returned to the problem in his 2001 Random Forests paper, proposing that a variable be permuted in the out-of-bag samples and the increase in error measured - the first widely adopted model-agnostic importance measure. Carolin Strobl and colleagues documented the cardinality bias of impurity importance in a 2007 BMC Bioinformatics paper, which pushed practitioners toward permutation and conditional variants. The game-theoretic branch traces to Lloyd Shapley, who defined the Shapley value in 1953 as the unique fair allocation of a coalition payoff. Erik Strumbelj and Igor Kononenko applied it to prediction attribution around 2010, and Scott Lundberg and Su-In Lee unified it with several earlier methods in the 2017 NeurIPS paper introducing SHAP, together with TreeSHAP, an exact polynomial-time algorithm for tree ensembles that made the approach practical on real datasets. Marco Ribeiro, Sameer Singh and Carlos Guestrin had introduced LIME in 2016 using local surrogate models. Regulatory pressure, notably the EU General Data Protection Regulation taking effect in 2018, accelerated adoption of these methods in credit scoring and insurance underwriting.