역 전파를 사용하여 신경망을 훈련시키는 데 시간이 얼마나 복잡합니까?

NN 에 각 레이어에 $n$ 숨겨진 레이어, $m$ 교육 예제, $x$ 기능 및 $n_i$ 노드가 포함되어 있다고 가정합니다 . 역 전파를 사용하여이 NN을 교육하는 데 시간이 얼마나 복잡합니까?

알고리즘의 시간 복잡성을 찾는 방법에 대한 기본 아이디어가 있지만 여기서는 반복, 레이어, 각 레이어의 노드, 학습 예제 및 더 많은 요소와 같이 여기에서 고려해야 할 4 가지 요소가 있습니다. 나는 여기 에서 답을 찾았 지만 충분히 명확하지 않았다.

위에서 언급 한 것 외에 NN 교육 알고리즘의 시간 복잡성에 영향을 미치는 다른 요소가 있습니까?

신뢰할 수있는 출처의 답변을 보지 못했지만 간단한 예 (현재 알고있는)로 직접 대답하려고합니다.

일반적으로 역 전파를 사용하여 MLP를 훈련하는 것은 일반적으로 행렬로 구현됩니다.

행렬 곱셈의 시간 복잡성

$M_{ij} * M_{jk}$ 대한 행렬 곱셈의 시간 복잡도 는 단순히 $\mathcal{O}(i*j*k)$ 입니다.

여기서 우리는 가장 간단한 곱셈 알고리즘을 가정합니다. 시간 복잡성이 약간 더 좋은 다른 알고리즘이 있습니다.

피드 포워드 패스 알고리즘

피드 포워드 전파 알고리즘은 다음과 같습니다.

먼저, 레이어 $i$ 에서 $j$ 로 이동 하려면

$$S_j = W_{ji}*Z_i$$

그런 다음 활성화 기능을 적용합니다

$$Z_j = f(S_j)$$

우리가 있다면 $N$ (입출력 층을 포함한다) 층을,이 실행 $N-1$ 회.

예

예를 들어, $4$ 레이어가 있는 MLP의 순방향 패스 알고리즘에 대한 시간 복잡도를 계산해 봅시다 . 여기서 $i$ 는 입력 레이어 의 노드 수 , $j$ 는 두 번째 레이어 의 노드 수 , $k$ 는 노드 수 세 번째 계층과 $l$ 출력 계층의 노드 수.

이 때문에 $4$ 층, 당신은 필요한 $3$ 이 개 사이의 가중치를 나타내는 행렬. 그럼으로 그들을 나타낸다하자 $W_{ji}$ , $W_{kj}$ 및 $W_{lk}$ 여기서, $W_{ji}$ 가진 행렬 $j$ 행 $i$ 열 ( $W_{ji}$ 따라서 층으로부터가는 가중치 포함 $i$ 층으로 $j$ ).

당신이 가정 $t$ 훈련 예. 레이어 $i$ 에서 $j$ 전파 하기 위해 먼저

$$S_{jt} = W_{ji} * Z_{it}$$

and this operation (i.e. matrix multiplcation) has $\mathcal{O}(j*i*t)$ time complexity. Then we apply the activation function

$$Z_{jt} = f(S_{jt})$$

and this has $\mathcal{O}(j*t)$ time complexity, because it is an element-wise operation.

So, in total, we have

$$\mathcal{O}(j*i*t + j*t) = \mathcal{O}(j*t*(t + 1)) = \mathcal{O}(j*i*t)$$

Using same logic, for going $j \to k$, we have $\mathcal{O}(k*j*t)$, and, for $k \to l$, we have $\mathcal{O}(l*k*t)$.

In total, the time complexity for feedforward propagation will be

$$\mathcal{O}(j*i*t + k*j*t + l*k*t) = \mathcal{O}(t*(ij + jk + kl))$$

I'm not sure if this can be simplified further or not. Maybe it's just $\mathcal{O}(t*i*j*k*l)$, but I'm not sure.

Back-propagation algorithm

The back-propagation algorithm proceeds as follows. Starting from the output layer $l \to k$, we compute the error signal, $E_{lt}$, a matrix containing the error signals for nodes at layer $l$

$$E_{lt} = f'(S_{lt}) \odot {(Z_{lt} - O_{lt})}$$

where $\odot$ means element-wise multiplication. Note that $E_{lt}$ has $l$ rows and $t$ columns: it simply means each column is the error signal for training example $t$.

We then compute the "delta weights", $D_{lk} \in \mathbb{R}^{l \times k}$ (between layer $l$ and layer $k$)

$$D_{lk} = E_{lt} * Z_{tk}$$

where $Z_{tk}$ is the transpose of $Z_{kt}$.

We then adjust the weights

$$W_{lk} = W_{lk} - D_{lk}$$

$l \to k$$\mathcal{O}(lt + lt + ltk + lk) = \mathcal{O}(l*t*k)$.

Now, going back from $k \to j$. We first have

$$E_{kt} = f'(S_{kt}) \odot (W_{kl} * E_{lt})$$

Then

$$D_{kj} = E_{kt} * Z_{tj}$$

And then

$$W_{kj} = W_{kj} - D_{kj}$$

where $W_{kl}$ is the transpose of $W_{lk}$. For $k \to j$, we have the time complexity $\mathcal{O}(kt + klt + ktj + kj) = \mathcal{O}(k*t(l+j))$.

And finally, for $j \to i$, we have $\mathcal{O}(j*t(k+i))$. In total, we have

$$\mathcal{O}(ltk + tk(l + j) + tj (k + i)) = \mathcal{O}(t*(lk + kj + ji))$$

which is same as feedforward pass algorithm. Since they are same, the total time complexity for one epoch will be

$$O(t*(ij + jk + kl)).$$

This time complexity is then multiplied by number of iterations (epochs). So, we have

$$O(n*t*(ij + jk + kl)),$$ where $n$ is number of iterations.

Notes

Note that these matrix operations can greatly be paralelized by GPUs.

Conclusion

We tried to find the time complexity for training a neural network that has 4 layers with respectively $i$, $j$, $k$ and $l$ nodes, with $t$ training examples and $n$ epochs. The result was $\mathcal{O}(nt*(ij + jk + kl))$.

We assumed the simplest form of matrix multiplication that has cubic time complexity. We used batch gradient descent algorithm. The results for stochastic and mini-batch gradient descent should be same. (Let me know if you think the otherwise: note that batch gradient descent is the general form, with little modification, it becomes stochastic or mini-batch)

Also, if you use momentum optimization, you will have same time complexity, because the extra matrix operations required are all element-wise operations, hence they will not affect the time complexity of the algorithm.

I'm not sure what the results would be using other optimizers such as RMSprop.

Sources

The following article http://briandolhansky.com/blog/2014/10/30/artificial-neural-networks-matrix-form-part-5 describes an implementation using matrices. Although this implementation is using "row major", the time complexity is not affected by this.

If you're not familiar with back-propagation, check this article:

http://briandolhansky.com/blog/2013/9/27/artificial-neural-networks-backpropagation-part-4

For the evaluation of a single pattern, you need to process all weights and all neurons. Given that every neuron has at least one weight, we can ignore them, and have $\mathcal{O}(w)$ where $w$ is the number of weights, i.e., $n * n_i$, assuming full connectivity between your layers.

The back-propagation has the same complexity as the forward evaluation (just look at the formula).

So, the complexity for learning $m$ examples, where each gets repeated $e$ times, is $\mathcal{O}(w*m*e)$.

The bad news is that there's no formula telling you what number of epochs $e$ you need.

A potential disadvantage of gradient-based methods is that they head for the nearest minimum, which is usually not the global minimum.

This means that the only difference between these search methods is the speed with which solutions are obtained, and not the nature of those solutions.

An important consideration is time complexity, which is the rate at which the time required to find a solution increases with the number of parameters (weights). In short, the time complexities of a range of different gradient-based methods (including second-order methods) seem to be similar.

Six different error functions exhibit a median run-time order of approximately O(N to the power 4) on the N-2-N encoder in this paper:

Lister, R and Stone J "An Empirical Study of the Time Complexity of Various Error Functions with Conjugate Gradient Back Propagation" , IEEE International Conference on Artificial Neural Networks (ICNN95), Perth, Australia, Nov 27-Dec 1, 1995.

Summarised from my book: Artificial Intelligence Engines: A Tutorial Introduction to the Mathematics of Deep Learning.