Deep Learning

Deep Learning

Cost for sample $x$

total Cost:

the objective function:

define:

to z $$ \frac {\partial C_x} {\partial z_k^l} = \sum_j { \frac {\partial C_x} {\partial z_j^{l+1} } \cdot \frac {\partial z_j^{l+1}} {\partial z_k^{l}} }\

\delta^l_k = \sum_j \delta^{l+1}_j \cdot {\frac {\partial z_j^{l+1}} {\partial z_k^{l} }}

\delta^L_j \equiv \frac {\partial C} {\partial z_j^L}
=\frac {\partial C} {\partial a_j^L} \cdot \sigma^{\prime}(z_j^L)
= ( a_j^L - y_j) \cdot \sigma(z_j^L) \cdot (1 - \sigma(z_j^L)) $$

matrix based: back propagation:

$$


\delta^l_k = \sum_j {\delta^{l+1}_k}

\delta^L = \nabla_aC \odot {\sigma^{\prime}(z^L)} \

\delta^l = ((w^{l+1})^T \delta^{l+1}) \odot {\sigma^{\prime}(z^l)} \

$$

with $w_{jk}$ $$ \nabla_{b^l}C = \delta^l
\nabla_{w^l}C = \delta^l \sigma^{l-1} \

$$