<aside> 💡

When we assume a Gaussian (normal) distribution for the predictors within each class, Bayes' theorem leads to Linear Discriminant Analysis (LDA) or Quadratic Discriminant Analysis (QDA).

</aside>

Bayes' Theorem for Classification

$$ P(Y=k \mid X=x) = \frac{P(Y=k) \times P(X=x \mid Y=k)}{P(X=x)} $$

$$ \text{Posterior} = \frac{\text{Prior} \times \text{Likelihood}}{\text{Evidence}} $$

Where:

• $\pi_k = P(Y=k)$ = prior probability of class $k$
• $f_k(x) = P(X=x \mid Y=k)$ = likelihood (density of X in class $k$)
• $P(X=x)$ = marginal probability (same for all classes, acts as normalizing constant)

Bayes Classifier Decision Rule

Assign observation to class $k$ that maximizes the posterior:

$$ \hat{k} = \arg\max_k \; \pi_k f_k(x) $$

For binary classification ($K=2$):

$$ \text{Predict class 1 if } \pi_1 f_1(x) > \pi_2 f_2(x), \quad \text{otherwise class 2} $$

LDA Discriminant Function

Assuming $f_k(x) \sim \mathcal{N}(\mu_k, \sigma^2)$ (same variance across classes), the log-posterior simplifies to the linear discriminant function:

$$ \delta_k(x) = x \cdot \frac{\mu_k}{\sigma^2} - \frac{\mu_k^2}{2\sigma^2} + \log(\pi_k) $$

Assign to class $k$ that maximizes $\delta_k(x)$.

<aside> 💡

The decision boundary between two classes is linear in $x$ — that's where LDA gets its name. If variances differ across classes, the boundary becomes quadratic → QDA.

</aside>

Why Use LDA Instead of Logistic Regression?