Structure

Decision tree showing root, internal nodes, edges, and leaf nodes

Terminology

A decision tree asks a sequence of questions to make a prediction

Classification: a leaf predicts a class (e.g., \(+\) or \(-\)).

Regression: a leaf predicts a numeric value (typically the mean of the targets in that leaf).

CART Tree Construction Algorithm

• Goal: Create the smallest/simplest tree that fits the training data
• Doing this optimally is NP-Hard
• Greedy Algorithm:
  • Start with a root node containing the full data set.
  • If there is only one class, terminate
  • Else, check every possible split point for every attribute.
    • (Involves sorting the data using the attributes as the key)
  • Select the split that results in the greatest overall reduction in impurity
  • Distribute the data to the two children according to that split
  • Recursively build decision trees starting at the two children

A Training Set to Work With

Step 1: Enumerate the Candidate Splits

Sort by each attribute, then consider the midpoint between each pair of adjacent values.

Five split points per attribute, ten candidates in all.

Entropy

• The information associated with an event is defined as: \(I(E) = -\log_2 p(E)\)

• Entropy is expected information: \(\displaystyle -\sum_{i=1}^{c} \Big ( p_i \log_2 ( p_i ) \Big )\)

Gini Impurity

• The fraction of incorrect predictions in a node if the class of each element was predicted by randomly selecting a label according to the distribution of classes in the node:

  \(\displaystyle \phi(\mathbf{p}) = \sum_i p_i(1-p_i)\)

  • \((1-p_i)\) is the probability of making a mistake for class \(i\).

• Can be rewritten as: \(\displaystyle \phi(\mathbf{p}) = 1 - \sum_i p_i^2\)

Impurity Formulas: Gini vs. Entropy

Note: \(p_{ki}\) represents the proportion of instances of class \(k\) present at node \(i\).

What the Numbers Look Like

Four nodes ranging from pure to evenly mixed, each labeled with its Gini impurity and entropy

Both measures are \(0\) for a pure node and largest when the node is evenly mixed.

Entropy vs. Gini

Scoring a Split

• A split produces two children, so we need a single number for the pair. Weight each child by the fraction of examples that land in it:

  \(\displaystyle \phi_{\text{split}} = \frac{m_{\text{left}}}{m}\phi(\text{left}) + \frac{m_{\text{right}}}{m}\phi(\text{right})\)

• The gain is the reduction in impurity:

  \(\displaystyle \text{Gain} = \phi(\text{parent}) - \phi_{\text{split}}\)

• \(\phi(\text{parent})\) is the same for every candidate, so maximizing gain and minimizing \(\phi_{\text{split}}\) are the same thing.

Step 2: Score Every Candidate

Root: 4 +, 2 −, so \(\phi(\text{parent}) = 1 - \left(\tfrac{4}{6}\right)^2 - \left(\tfrac{2}{6}\right)^2 = 0.444\)

Step 2: The Answers

Winner: \(x_1 \le 4.5\).

Step 3: Take the Best Split

Left child is pure, so it becomes a leaf. Right child still holds 1 +, 2 −, so we recurse.

Step 4: Recurse on the Right Child

Two Views of the Finished Tree

Every internal node is an axis-aligned cut. Three leaves, three regions. A tree and a partition of the feature space are the same object.

MSE

• For regression trees, we are predicting a real value: \(\displaystyle\hat{y}_{\text{node}} = \frac{\sum_{i\in \text{node}}y^{(i)}}{m_{\text{node}}}\)
• MSE is then: \(\displaystyle MSE_{\text{node}} = \frac{\sum_{i\in \text{node}}(\hat{y}_{\text{node}} - y^{(i)})^2}{m_{\text{node}}}\)
• (Notice that this is exactly the sample variance)

MSE: A Worked Example

Same algorithm, same weighted average, different \(\phi\). Low MSE means a tight cluster of target values.

Scenario

Evaluating a binary split on a parent node with \(N = 10\) samples:

• 6 Arabica
• 4 Robusta

Step 1: Calculate Parent Node Impurity

\[ p_{\text{Arabica}} = \frac{6}{10} = 0.6, \qquad p_{\text{Robusta}} = \frac{4}{10} = 0.4 \]

\[ \text{Gini}_{\text{parent}} = 1 - (0.6^2 + 0.4^2) = 1 - (0.36 + 0.16) = \mathbf{0.480} \]

Step 2: Evaluate Split Outcome

Suppose the candidate feature \(X_1 \le 8.0\) splits the 10 samples into two child nodes:

• Left Child (\(N_L = 5\)): 5 Arabica, 0 Robusta

  \[ \begin{aligned} \text{Gini}_L &= 1 - \left(1.0^2 + 0.0^2\right) \\ &= \mathbf{0.000} \qquad \text{(Pure Node)} \end{aligned} \]

• Right Child (\(N_R = 5\)): 1 Arabica, 4 Robusta

  \[ \begin{aligned} \text{Gini}_R &= 1 - \left(0.2^2 + 0.8^2\right) \\ &= 1 - (0.04 + 0.64) \\ &= \mathbf{0.320} \end{aligned} \]

Step 3: Compute Weighted Child Impurity & Gain

\[ \text{Gini}_{\text{split}} = \frac{5}{10}(0.000) + \frac{5}{10}(0.320) = \mathbf{0.160} \]

\[ \Delta \text{Gini} = \text{Gini}_{\text{parent}} - \text{Gini}_{\text{split}} = 0.480 - 0.160 = \mathbf{0.320} \]

Decision Rule: CART selects the split that maximizes impurity reduction (\(\Delta \text{Gini}\)).

Issues to Consider

• Default algorithm builds tree until the leaves are 100% pure.
• Likely to result in overfitting.
• Solutions:
  • Pruning - build out the full tree, then combine leaves that don’t significantly improve classification
  • Restrict the size of the tree during construction - limit the depth, number of leaves, etc.
  • Looking ahead: Ensemble methods that combine multiple trees

What Overfitting Looks Like

Decision boundaries for a fully grown tree and a depth three tree on the same noisy data

The left tree gets every training point right. The extra slivers are chasing noise, and they will cost accuracy on data the tree has not seen.

Pros/Cons

• Pros:
  • Prediction is fast
  • Explainable
  • We don’t need to worry about the relative scale of the different attributes
• Cons
  • Sensitive to data orientation
  • Generally not very competitive in terms of accuracy

Sensitive to Data Orientation

The same two class data before and after a 45 degree rotation, with the tree decision boundary drawn on each

Every split is axis-aligned, so a boundary that is not axis-aligned has to be approximated by a staircase. Same data, same labels, much bigger tree.

Basic Syntax

In scikit-learn, decision tree classifiers are provided by the DecisionTreeClassifier class.

import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

# 1. Load dataset
data = pd.read_csv("coffee.csv")

X = data[["sweetness", "uniformity", "acidity"]]
y = data["species"]

# 2. Train-test split
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

# 3. Instantiate and fit model
dtc = DecisionTreeClassifier(random_state=42)
dtc.fit(X_train, y_train)

# 4. Predict classes and probabilities
y_pred = dtc.predict(X_test)
y_prob = dtc.predict_proba(X_test)
classes = dtc.classes_

5. Early Stopping (Pre-Pruning)

Unconstrained decision trees tend to grow until every leaf is 100% pure, memorizing training noise and leading to severe overfitting.

Early stopping restricts tree growth during training using stopping thresholds:

Early Stopping Code Example

from sklearn.tree import DecisionTreeClassifier

# Configure pre-pruning / early stopping parameters
dtc_constrained = DecisionTreeClassifier(
    criterion="gini",
    max_depth=3,             # Maximum tree depth of 3
    min_samples_split=10,    # Needs >= 10 samples to consider splitting
    min_samples_leaf=4,      # Every leaf must have >= 4 samples
    max_leaf_nodes=8,        # Maximum of 8 total leaves
    random_state=42
)

# Train model on training set
dtc_constrained.fit(X_train, y_train)

# Evaluate generalization
print("Train Score:", dtc_constrained.score(X_train, y_train))
print("Test Score: ", dtc_constrained.score(X_test, y_test))

Google Colab

Open the notebook in Google Colab