<aside> 💡 Even though the name suggests regression, this machine learning algorithm is used for classification based problems.

</aside>

• Generalized Logistic Function (Sigmoid Function):

  • Binary

    $$ p(X) = \frac{\beta_0 + \beta_1 X_1}{1 + e^{(\beta_0 + \beta_1 X_1)}} = \frac{1}{1 e^{-(\beta_0 + \beta_1 X_1)}} $$

  • Multi-class:

    $$ p(X) = \frac{z}{1 + e^{(z)}} = \frac{1}{1 + e^{-z}} $$

    • $z = \beta_0 + \beta_1 X_1 +....+\beta_nX_n$

• Logit Transformation:

  $$ \text{logit}(p) = \log\left(\frac{p(X)}{1-p(X)}\right)=z \newline \text{logit}(p) = log\; odds

  $$

• Logistic Regression Equation:

  $$ \text{logit}(\mathbb{P}(Y=1|\mathbf{X=x})) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \ldots + \beta_n x_n $$

  $$ \mathbb{P}(Y=y|\mathbf{X=x}) = [\;p(x)\;]^y\;[\;1-p(x)\;]^{1-y} $$

• Likelihood $\mathcal{L}(\beta_0,\beta_1,...,\beta_p)$

$$ \mathcal{L}(\beta_0, \beta_1, ..., \beta_p) = \prod_{i=1}^{n} \left( p(x_i) \right)^{y_i} \left( 1 - p(x_i) \right)^{1 - y_i} $$

• Log-Likelihood

  • $log(\mathcal{L}) = \mathcal{l}$

    $$ l(\beta_0, \beta_1, ..., \beta_p) = \sum_{i=1}^{n} \left( y_i \log \left( p(x_i ) \right) + (1 - y_i) \log \left( 1 - p(x_i) \right) \right) $$

  • Cross Entropy Function:

  $$ l(\beta_0, \beta_1, ..., \beta_p) = \sum_{i=1}^{n} -\left( y_i \log \left( p(x_i ) \right) + (1 - y_i) \log \left( 1 - p(x_i) \right) \right) $$

• Z-test:

  $$ H_0: \beta_i = \beta \newline H_A:B_i\ne\beta $$

  $$ \mathbf{z}=\frac{\hat\beta_i-\beta}{SE(\hat\beta_i)} $$

  • Pass/Fail Criteria:
    • Pass: $∣z∣>z_\text{n-p-1, α/2} or ∣z∣>z_\text{n-p-1,α}$ (two-tailed)
    • Fail: Otherwise