로지스틱 회귀 최적화에 Newton의 방법을 사용하는 것을 반복 재가 중 최소 제곱이라고하는 이유는 무엇입니까?

로지스틱 회귀 최적화에 Newton의 방법을 사용하는 것을 반복 재가 중 최소 제곱이라고하는 이유는 무엇입니까?

물류 손실과 최소 제곱 손실이 완전히 다르기 때문에 나에게 분명하지 않은 것 같습니다.

요약 : GLM은 Dimitriy V. Masterov가 지적한 것처럼 대신 Hessian이 예상되는 Newton-Raphson 인 Fisher 점수 를 통해 적합 합니다 (즉, 관측 된 정보 대신 Fisher 정보의 추정치를 사용합니다). 표준 링크 함수를 사용하는 경우 관찰 된 Hessian이 예상 Hessian과 같으므로 NR과 Fisher 점수는 동일합니다. 어느 쪽이든, 우리는 Fisher 점수가 실제로 가중 최소 제곱 선형 모형에 적합하다는 것을 알 수 있으며, 이로부터의 계수 추정치는 최대 로지스틱 회귀 가능성에 대한 수렴 *입니다. 로지스틱 회귀 분석을 이미 해결 된 문제에 맞추는 것 외에도 최종 WLS 적합에 대해 선형 회귀 진단을 사용하여 로지스틱 회귀에 대해 배울 수 있다는 이점도 얻습니다.

나는 이것을 로지스틱 회귀에 중점을 두겠다. 그러나 GLM의 최대 가능성에 대한보다 일반적인 관점을 위해이 장의 섹션 15.3을 권장 하고이를 통해보다 일반적인 환경에서 IRLS를 도출한다 (존 폭스의 Applied 에서 온 것으로 생각한다) 회귀 분석 및 일반 선형 모형 ).

$^*$ 끝에 주석을 참조하십시오

---

가능성과 점수 함수

우리는 폼의 것을 반복하여 우리 GLM 피팅 될 ℓ가 로그 가능도이고 J의 m이 어느 것 대수 가능성의 관측되거나 예상되는 Hessian.

$$b^{(m+1)} = b^{(m)} - J^{-1}_{(m)}\nabla \ell(b^{(m)})$$ $\ell$$J_{m}$

우리의 링크 함수는 조건부 평균 μ i = E ( y i | x i ) 를 선형 예측 변수에 매핑 하는 함수 이므로 평균에 대한 모델은 g ( μ i ) = x T i β 입니다. h를 선형 예측 변수를 평균에 매핑하는 역 링크 함수라고 합시다 .$g$$\mu_i = E(y_i | x_i)$$g(\mu_i) = x_i^T\beta$$h$

로지스틱 회귀 분석의 경우 독립 관측 값을 갖는 Bernoulli 가능성이 있으므로

$$\ell(b; y) = \sum_{i=1}^n y_i\log h(x_i^T b) + (1 - y_i) \log(1 - h(x_i^Tb)).$$ 파생 상품 가져 오기, = n ∑ i=1xijh'(x T i b)( y $$\frac{\partial \ell}{\partial b_j} = \sum_{i=1}^n \frac{y_i}{h(x_i^T b)} h'(x_i^T b) x_{ij} - \frac{1 - y_i}{1 - h(x_i^T b)} h'(x_i^T b) x_{ij}$$ =∑ixijh′(x T i b $$= \sum_{i=1}^n x_{ij} h'(x_i^T b) \left(\frac{y_i}{h(x_i^T b)} - \frac{1 - y_i}{1 - h(x_i^T b)} \right)$$ $$= \sum_i x_{ij} \frac{h'(x_i^T b)}{h(x_i^T b)(1 - h(x_i^T b))}(y_i - h(x_i^T b)).$$

---

정식 링크 사용

이제 표준 링크 함수 사용한다고 가정 해 봅시다 . 그런 다음 g − 1 c ( x ) : = h c ( x ) = $g_c = \text{logit}$ 그래서h ′ c =hc⋅(1−hc)이것은∂ℓ로단순화 함을 의미합니다 $g^{-1}_c(x) := h_c(x) = \frac{1}{1+e^{-x}}$$h_c' = h_c \cdot (1-h_c)$ 정도로 ∇ℓ(B,Y)=XT(Y - Y ). 또한 여전히hc, ∂2

$$\frac{\partial \ell}{\partial b_j} = \sum_i x_{ij} (y_i - h_c(x_i^T b))$$ $$\nabla \ell (b; y) = X^T (y - \hat y).$$ $h_c$ $$\frac{\partial^2 \ell}{\partial b_k \partial b_j} = - \sum_i x_{ij} \frac{\partial}{\partial b_k} h_c(x_i^T b) = - \sum_i x_{ij}x_{ik} \left[h_c(x_i^T b) (1 - h_c(x_i^T b))\right].$$

하자 그리고 우리는이 H=- X T WX 및 참고이 어떤없는 방법 Y 내가 더 이상 거기에 그렇게E(H)=H(우리의 함수로이를보고있는B유일한 무작위 것은 그래서Y그 자체). 따라서 로지스틱 회귀 분석에서 정식 링크를 사용하는 경우 Fisher 점수가 Newton-Raphson과 같습니다. 또한의 미덕

$$W = \text{diag}\left(h_c(x_1^T b)(1 - h_c(x_1^T b)), \dots, h_c(x_n^T b)(1 - h_c(x_n^T b))\right) = \text{diag}\left(\hat y_1(1 - \hat y_1), \dots, \hat y_n (1 - \hat y_n)\right).$$ $$H = -X^TWX$$ $y_i$$E(H) = H$$b$$y$-XTWX수치 경우더라도 항상 일정한 엄격 부정적인 것 Y 제가너무 가까이 가져0또는1, 그리고 우리는 가중치가 둥글게 가질 수0수있는H에따라서 계산 마이너스의 semidefinite 및 단수형.$\hat y_i \in (0,1)$ $-X^TWX$$\hat y_i$$0$$1$$0$$H$

$z = W^{-1}(y - \hat y)$

$$\nabla \ell = X^T(y - \hat y) = X^T W z.$$

모두 함께 이것은 를 반복하여 로그 우도를 최적화 할 수 있음을 의미합니다

$$b^{(m+1)} = b^{(m)} + (X^T W_{(m)} X)^{-1}X^T W_{(m)} z_{(m)}$$ and $(X^T W_{(m)} X)^{-1}X^T W_{(m)} z_{(m)}$ is exactly $\hat \beta$ for a weighted least squares regression of $z_{(m)}$ on $X$.

Checking this in R:

set.seed(123)
p <- 5
n <- 500
x <- matrix(rnorm(n * p), n, p)
betas <- runif(p, -2, 2)
hc <- function(x) 1 /(1 + exp(-x)) # inverse canonical link
p.true <- hc(x %*% betas)
y <- rbinom(n, 1, p.true)

# fitting with our procedure
my_IRLS_canonical <- function(x, y, b.init, hc, tol=1e-8) {
  change <- Inf
  b.old <- b.init
  while(change > tol) {
    eta <- x %*% b.old  # linear predictor
    y.hat <- hc(eta)
    h.prime_eta <- y.hat * (1 - y.hat)
    z <- (y - y.hat) / h.prime_eta

    b.new <- b.old + lm(z ~ x - 1, weights = h.prime_eta)$coef  # WLS regression
    change <- sqrt(sum((b.new - b.old)^2))
    b.old <- b.new
  }
  b.new
}

my_IRLS_canonical(x, y, rep(1,p), hc)
# x1         x2         x3         x4         x5 
# -1.1149687  2.1897992  1.0271298  0.8702975 -1.2074851

glm(y ~ x - 1, family=binomial())$coef
# x1         x2         x3         x4         x5 
# -1.1149687  2.1897992  1.0271298  0.8702975 -1.2074851 

and they agree.

---

Non-canonical link functions

Now if we're not using the canonical link we don't get the simplification of $\frac{h'}{h(1-h)} = 1$ in $\nabla \ell$ so $H$ becomes much more complicated, and we therefore see a noticeable difference by using $E(H)$ in our Fisher scoring.

Here's how this will go: we already worked out the general $\nabla \ell$ so the Hessian will be the main difficulty. We need

$$\frac{\partial^2 \ell}{\partial b_k \partial b_j} = \sum_i x_{ij} \frac{\partial}{\partial b_k}h'(x_i^T b) \left(\frac{y_i}{h(x_i^T b)} - \frac{1 - y_i}{1 - h(x_i^T b)} \right)$$ $$= \sum_i x_{ij}x_{ik} \left[h''(x_i^T b) \left(\frac{y_i}{h(x_i^T b)} - \frac{1 - y_i}{1 - h(x_i^T b)} \right) - h'(x_i^T b)^2\left(\frac{y_i}{h(x_i^T b)^2} + \frac{1-y_i}{(1-h(x_i^T b))^2} \right)\right]$$

Via the linearity of expectation all we need to do to get $E(H)$ is replace each occurrence of $y_i$ with its mean under our model which is $\mu_i=h(x_i^T\beta)$. Each term in the summand will therefore contain a factor of the form

$$h''(x_i^T b) \left(\frac{h(x_i^T \beta)}{h(x_i^T b)} - \frac{1 - h(x_i^T \beta)}{1 - h(x_i^T b)} \right) - h'(x_i^T b)^2\left(\frac{h(x_i^T \beta)}{h(x_i^T b)^2} + \frac{1-h(x_i^T \beta)}{(1-h(x_i^T b))^2} \right).$$ But to actually do our optimization we'll need to estimate each $\beta$, and at step $m$ $b^{(m)}$ is the best guess we have. This means that this will reduce to $$h''(x_i^T b) \left(\frac{h(x_i^T b)}{h(x_i^T b)} - \frac{1 - h(x_i^T b)}{1 - h(x_i^T b)} \right) - h'(x_i^T b)^2\left(\frac{h(x_i^T b)}{h(x_i^T b)^2} + \frac{1-h(x_i^T b)}{(1-h(x_i^T b))^2} \right)$$ $$= - h'(x_i^T b)^2\left(\frac{1}{h(x_i^T b)} + \frac{1}{1-h(x_i^T b)} \right)$$ $$= -\frac{h'(x_i^T b)^2}{h(x_i^T b)(1-h(x_i^T b))}.$$ This means we will use $J$ with $$J_{jk} = -\sum_i x_{ij}x_{ik} \frac{h'(x_i^T b)^2}{h(x_i^T b)(1-h(x_i^T b))}.$$

Now let

$$W^* = \text{diag}\left(\frac{h'(x_1^T b)^2}{h(x_1^T b)(1-h(x_1^T b))} ,\dots, \frac{h'(x_n^T b)^2}{h(x_n^T b)(1-h(x_n^T b))}\right)$$ and note how under the canonical link $h_c' = h_c \cdot (1-h_c)$ reduces $W^*$ to $W$ from the previous section. This lets us write $$J = -X^TW^*X$$ except this is now $\hat E(H)$ rather than necessarily being $H$ itself, so this can differ from Newton-Raphson. For all $i$ $W_{ii}^* > 0$ so aside from numerical issues $J$ will be negative definite.

We have

$$\frac{\partial \ell}{\partial b_j} = \sum_i x_{ij} \frac{h'(x_i^T b)}{h(x_i^T b)(1 - h(x_i^T b))}(y_i - h(x_i^T b))$$ so letting our new working response be $z^* = D^{-1}(y-\hat y)$ with $D=\text{diag}\left(h'(x_1^T b), \dots, h'(x_n^T b)\right)$, we have $\nabla \ell = X^TW^*z^*$.

All together we are iterating

$$b^{(m+1)} = b^{(m)} + (X^T W_{(m)}^* X)^{-1}X^T W_{(m)}^* z_{(m)}^*$$ so this is still a sequence of WLS regressions except now it's not necessarily Newton-Raphson.

---

I've written it out this way to emphasize the connection to Newton-Raphson, but frequently people will factor the updates so that each new point $b^{(m+1)}$ is itself the WLS solution, rather than a WLS solution added to the current point $b^{(m)}$. If we wanted to do this, we can do the following:

$$b^{(m+1)} = b^{(m)} + (X^T W_{(m)}^* X)^{-1}X^T W_{(m)}^* z_{(m)}^*$$ $$= (X^T W_{(m)}^* X)^{-1}\left(X^T W_{(m)}^* Xb^{(m)}+ X^TW^*_{(m)}z_{(m)}^* \right)$$ $$= (X^T W_{(m)}^* X)^{-1}X^TW_{(m)}^*\left(Xb^{(m)}+ z_{(m)}^* \right)$$ so if we're going this way you'll see the working response take the form $\eta^{(m)} + D^{-1}_{(m)}(y - \hat y^{(m)})$, but it's the same thing.

---

Let's confirm that this works by using it to perform a probit regression on the same simulated data as before (and this is not the canonical link, so we need this more general form of IRLS).

my_IRLS_general <- function(x, y, b.init, h, h.prime, tol=1e-8) {
  change <- Inf
  b.old <- b.init
  while(change > tol) {
    eta <- x %*% b.old  # linear predictor
    y.hat <- h(eta)
    h.prime_eta <- h.prime(eta)
    w_star <- h.prime_eta^2 / (y.hat * (1 - y.hat))
    z_star <- (y - y.hat) / h.prime_eta

    b.new <- b.old + lm(z_star ~ x - 1, weights = w_star)$coef  # WLS

    change <- sqrt(sum((b.new - b.old)^2))
    b.old <- b.new
  }
  b.new
}

# probit inverse link and derivative
h_probit <- function(x) pnorm(x, 0, 1)
h.prime_probit <- function(x) dnorm(x, 0, 1)

my_IRLS_general(x, y, rep(0,p), h_probit, h.prime_probit)
# x1         x2         x3         x4         x5 
# -0.6456508  1.2520266  0.5820856  0.4982678 -0.6768585 

glm(y~x-1, family=binomial(link="probit"))$coef
# x1         x2         x3         x4         x5 
# -0.6456490  1.2520241  0.5820835  0.4982663 -0.6768581 

and again the two agree.

---

Comments on convergence

Finally, a few quick comments on convergence (I'll keep this brief as this is getting really long and I'm no expert at optimization). Even though theoretically each $J_{(m)}$ is negative definite, bad initial conditions can still prevent this algorithm from converging. In the probit example above, changing the initial conditions to b.init=rep(1,p) results in this, and that doesn't even look like a suspicious initial condition. If you step through the IRLS procedure with that initialization and these simulated data, by the second time through the loop there are some $\hat y_i$ that round to exactly $1$ and so the weights become undefined. If we're using the canonical link in the algorithm I gave we won't ever be dividing by $\hat y_i (1 - \hat y_i)$ to get undefined weights, but if we've got a situation where some $\hat y_i$ are approaching $0$ or $1$, such as in the case of perfect separation, then we'll still get non-convergence as the gradient dies without us reaching anything.