[PUBLISHER] Merge #22

* PUSH NOTE : GBO.md

* PUSH ATTACHMENT : GBO-slope.png

content/NN101/GBO.md

@@ -2,18 +2,23 @@
 share: "true"
 title: Gradient Based Optimization
 ---
+<style>
+!p img {
+text-align:center center;
+}
+</style>
 [Link](https://www.youtube.com/watch?v=atXrbIY6iyQ) for the video of Gradient Based Optimization.
 
-### Introduction
-##### What is optimization?
+## Introduction
+#### What is optimization?
 Optimization is selecting the best element with some criterion. Optimization problem is maximizing or minimizing functions by choosing inputs.[^1]
 
 > [!Example] Real-life Optimizations Example
 > - Optimizing the design to maximize the performance.
 > - Optimizing the recommendation algorithm to show best personalized advertisement.
 > - Optimizing the factory to spend less time and money.
 
-### Mathematical Optimization
+## Mathematical Optimization
 This is just finding $x$ that maximizes or minimizes some function $f(x)$. It is easy for some simple functions, i.e. quadratic functions.
 $$
 f(x)= ax^2 + bx+c \quad (a>0)
@@ -23,15 +28,32 @@ To find $x$ that minimizes function $f(x)$, we simply differentiate $f(x)$.
 $$
 f'(x)=2ax+b=0
 $$
+$f(x)$ is minimized when $x=-\frac{b}{2a}$.
+However, we want to find parameter values that minimizes the *complex* function, harder to differentiate symbolically.
+
+## Gradient Based Optimization
+### Observations
+![[GBO-slope.png|200]]
+<br/>
+
+Assume that global minimum is a only minimum, and $x'$ is a value that makes $f(x)$ minimum.
+Then, slope of the function $f(x)$ indicates where minimum exists.
+- If slope $f'(x_0)$ is positive, it means $x' < x_0$.
+- If slope $f'(x_0)$ is positive, it means $x_0 < x'$.
+
+Therefore, whatever the value of $x_0$ is, $x_1 = x_0 - f'(x_0)$ will move value to the direction of $x'$.
+We can do this iteratively, and expect that sequence $\{x_n\}$ will be converge to $x'$.
+$$
+x_{n+1} = x_n - f'(x_n)
+$$
+### Problem: To steep slope
+If function is to steep, sequence $\{x_n\}$ can diverge. Therefore, we add one hyperparameter, learning rate $\alpha\ (0<\alpha<1)$. Hyperparameter is set before optimization algorithm.[^2]
+$$
+x_{n+1} = x_n - \alpha f'(x_n)
+$$
+### Problem: Local minimum
 
-<script type="text/tikz">
-  \begin{tikzpicture}[domain=-2:2, scale=1.4]
-  \draw[very thin,color=gray] (-2,0) grid (2,3.9);
-  \draw[->, color=gray] (-2.0,0) -- (2.0,0) node[right] {\large $x$};
-  \draw[->, color=gray] (0,0) -- (0,4.2) node[above] {\large $f(x)$};
-  \draw[color=red] plot (\x,\x*\x) node[right] {\large $f(x) =x ^ 2$};;
-  \end{tikzpicture}
-</script>
 
 ---
-[^1]: https://en.wikipedia.org/wiki/Mathematical_optimization
+[^1]: https://en.wikipedia.org/wiki/Mathematical_optimization
+[^2]: https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)