GRADIENT DESCENT
1. Mathematical Introduction.
Consider a function $y=f(x_1, \ldots, x_n)$ evatuated at a point P:$x_1^0, \ldots, x_n^0$:
\begin{eqnarray}
y^0=f(x_1^0, \ldots, x_n^0)
\end{eqnarray}
Let us take a step of the “size” $\Delta x_1, \ldots, \Delta x_n$ from the point P
and evaluate the function again. Note that $y^0$ will also change to $y^0+ \Delta y$ in response to this step.
\begin{eqnarray}
y^0+\Delta y=f(x_1^0+\Delta x_1, \ldots, x_n^0+\Delta x_n)
\end{eqnarray}
Assume that $\Delta x_1, \ldots, \Delta x_n$ are all small quantities and apply our calculus knowledge:
\begin{eqnarray}
y^0+\Delta y&=&f(x_1^0+\Delta x_1, \ldots, x_n^0+\Delta x_n) \\
&=& f(x_1^0, \ldots, x_n^0)+\left[ \frac{\partial f}{\partial x_1} \right]_{x_1^0, \ldots, x_n^0} \Delta x_1+\ldots+\left[ \frac{\partial f}{\partial x_n} \right]_{x_1^0, \ldots, x_n^0} \Delta x_n + O(\Delta x^2)
\end{eqnarray}
Do the cancellations using (*). $\Delta x$’s are chossen to be small, so neglecting terms of $O(\Delta x^2)$, we get
\begin{eqnarray}
\Delta y &=& \left[ \frac{\partial f}{\partial x_1} \right]_{x_1^0, \ldots, x_n^0} \Delta x_1+\ldots+\left[ \frac{\partial f}{\partial x_n} \right]_{x_1^0, \ldots, x_n^0} \Delta x_n
\end{eqnarray}
We can write this expression as a dot product. onsider two vectors:
\begin{eqnarray}
\vec{\Delta x} &=& \begin{bmatrix}
\Delta x_1\\
\Delta x_2 \\
\vdots\\
\Delta x_n
\end{bmatrix} \\ \\ \\ \nabla f &=&
\begin{bmatrix}
\left[ \frac{\partial f}{\partial x_1} \right]_{x_1^0, \ldots, x_n^0} \\
\left[ \frac{\partial f}{\partial x_2} \right]_{x_1^0, \ldots, x_n^0} \\
\vdots \\
\left[ \frac{\partial f}{\partial x_n} \right]_{x_1^0, \ldots, x_n^0}
\end{bmatrix}.= \begin{bmatrix}
\frac{\partial f}{\partial x_1}\\
\frac{\partial f}{\partial x_2}\\
\vdots\\
\frac{\partial f}{\partial x_n}
\end{bmatrix}_{x_1^0, \ldots, x_n^0}.
\end{eqnarray}