Decision Trees & Feature Importance Interpretation

Decision Trees are non-parametric supervised learning models that partition feature space into rectangular regions using hierarchical if-then rules. Because they mimic human logical reasoning and automatically capture non-linear relationships and feature interactions, decision trees are one of the most interpretable tools in a business analyst's toolkit.

1. How Decision Trees Split: Gini Impurity vs Entropy

At each internal node, the decision tree algorithm tests all available features and numerical thresholds to find the split that maximizes information gain (reduction in impurity):

Gini Impurity

$$Gini = 1 - \sum_{i=1}^{C} p_i^2$$

• A pure node (100% class A) has $Gini = 0.0$.
• An evenly mixed binary node (50% A, 50% B) has maximum impurity $Gini = 0.50$.

Entropy & Information Gain

$$Entropy = -\sum_{i=1}^{C} p_i \log_2(p_i)$$

2. Preventing Overfitting: Regularization Hyperparameters

An unconstrained decision tree will continue splitting until every training leaf contains exactly 1 sample, memorizing noise and achieving 100% training accuracy but abysmal test accuracy.

Critical Hyperparameters:

• max_depth: Limits how deep the tree can grow (e.g., 3 to 6 levels for interpretability).
• min_samples_split: Minimum samples required to split an internal node.
• min_samples_leaf: Minimum samples required to exist at a leaf node (e.g., 20-50).

3. Extracting and Visualizing Feature Importance

Decision trees calculate MDI (Mean Decrease in Impurity), reflecting the total weighted reduction in Gini impurity brought by each feature across all splits:

4. Visualizing the Tree for Executive Presentations