Some interesting videos on Deep Learning [Work in Progress]

This post shall contain some interesting videos on Deep Learning. The list shall be continuously updated [Last Update 20 Aug. 2017].

Deep Learning

• Deep Learning Summer School 2015 - Deep Learning Summer School 2015
• Deep Learning Summer School 2016 - Deep Learning Summer School 2016
• Deep Learning Summer School 2017 - Deep Learning and Reinforcement Learning Summer School 2017

• Bay Area School 2016 Day1 - Bay Area Deep Learning School Day 1
• Bay Area School 2016 Day2 - Bay Area Deep Learning School Day 2

MapReduce in Apache Spark

Based on the Course CS120x Distributed Machine Learning with Apache Spark.

Basically we can summarize the map/reduce paradigm as following:

• Map: transforms a series of elements by applying a function individually to each element in the series. It then returns the series of transformed elements.
• Filter: applies a function individually to each element in a series but, the function evaluates to True or False and only elements that evaluate to True are retained.
• Reduce: operates on pairs of elements in a series. It applies a function that takes in two values and returns a single value. Using this function, reduce is able to, iteratively, “reduce” a series to a single value.

We have define an array of 10 elements and transform it in a Resilient Distributed Dataset (RDD)

numberRDD = range(0,10)
numberRDD = sc.parallelize(numberRDD, 4)
numberRDD.collect()
> Out[1]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] 

Map the numberRDD using a lambda function that mulitplies each element by 5

numberRDD.map(lambda x:x*5).collect()
> Out[2]: [0, 5, 10, 15, 20, 25, 30, 35, 40, 45] 

Filter the numberRDD in order to obtain only the number multiple of 2

numberRDD.filter(lambda x:x%2==0).collect()
> Out[3]: [0, 2, 4, 6, 8]

Reduce the numberRDD summing pairs of numbers

numberRDD.reduce(lambda x1,x2:x1+x2)
> Out[4]: [45]

Putting all together we obtain the sum of the numbers in the even positions

numberRDD.map(lambda x:x*5).filter(lambda x:x%2==0).reduce(lambda x1,x2:x1+x2)
> Out[5]: [100]

This post has been written using Markdown and Dillinger. Here an interesting Markdown Cheatsheet

Convolutional Layer of CNN in one Picture

A complete course at Stanford has devoted to Convolutional Neural Network.
The Course Notes (by Andrej Karpathy) are well written and they worth a look.

That course notes have inspired me to create a picture for summarising some concepts.

An interesting summary (adapted from here ) is the following:

Input Layer

• Size: $W_1 \times H_1 \times D_1$
• Hyperparameters:
• Number of filters $K$
• Dimension of the filter $F \times F \times D_1$
• Stride: $S$
• Amount of Zero Padding: $P$

Output Layer

• Size: $W_2 \times H_2 \times D_2$
• $W_2 = \frac{W_1 - F + 2P}{S} + 1$
• $H_2 = \frac{H_1 - F + 2P}{S} + 1$
• $D_2 = K$

The parameter sharing introduces $F \times F \times D_1$ per filter, for a total of $(F \times F \times D_1) \times K$ weights and $K$ biases

In the output volume, the d-th depth slice (of size $W_2 \times H2$) is the result of performing a valid convolution of the d-th filter over the input volume with a stride of $S$, and then offset by d-th bias.

Another interesting post on the Convolutional Neural Network is here

TensorFlow on Databricks

TensorFlow is an Open Source Software Library for Machine Learning and AI tasks.

In these months is becoming a widely used tool in the AI community (and not only).

Databricks is an interesting Cluster Manager based on Apache Spark. It offers a Community Edition for free (pricing).

Since some ML tasks can be very computational intensive (e.g. training of the Deep Networks) could be a good idea to have a Cluster on Databricks and use it.

You can run this Notebook on your Databricks cluster (or import it).

Even though the Notebook says that "It is not required for the Databricks Community Edition", I experimented that it is necessary for the Community Edition as well.

Variable Function Name in MATLAB

Suppose we want to call four different type of ODE solver, we can define a cell containing the 4 function handler as below:

functionName = {@ode45 @ode23 @ode113 @ode15s};

 and we can call in a cycle the desired function:

 for iFunction = 1 : size(functionName,2)  
   f = functionName{iFunction};  
   tol = 1e-4;  
   options = odeset ('RelTol',tol,'AbsTol',tol);  
   [t,y] = f('function_rhs', xspan, ic, options); % SOLVE ODEs 

   figure  
   plot(t,y)  
   title(func2str(f))  
 end 

Where function_rhs is the right side of the ODE we want to solve.
In this way we'll obtain 4 figures with the ODE solver name in the title.

2. Discrete Time Fourier Transform

Recall that if x(n) is absolutlely summable, that is, $\sum_{-\infty}^{+\infty} |x(n)| < \infty$ than its Discrete-Time Fourier Transform is given by:
\[
X(e^{j \omega})=\sum_{n=- \infty}^{+ \infty} x(n) \cdot e^{-j \cdot \omega \cdot n} \tag{3.1}
\]
This function transforms a discrete signal $x(n)$ into a complex-valued continuous function $X(e^{j \omega})$.

${\bf Matrix Implementation}$
If $x(n)$ is a finite duration sequence, we can use MATLAB to compute the Discrete Time Fourier Transform (DTFT) $X(e^{j\omega})$ numerically at any frequency $\omega$.

Let assume that the sequence x(n) has N samples between $n_1 \leq n \leq n_N$ we can define:
\[\omega_k =\frac{\pi}{M}k, \qquad k=0,1,....,M\]
which are (M+1) frequencies between $[0, \pi]$. Then (1) can be written as:

\[
X(e^{j\omega_k})=\sum_{r=1}^{N} x(n_r) \cdot e^{-j \cdot \omega_k \cdot n_r}  = \sum_{r=1}^{N} x(n_r) \cdot e^{-j \cdot \frac{\pi}{M} k \cdot n_r}, \qquad k=0,...,M \tag{3.2}
\]

Let's define the vector $\mathbf{X} = \begin{bmatrix} X(e^{j\omega_0}) \\ \vdots \\ X(e^{j\omega_M}) \end{bmatrix}$ where the k-th element $X(e^{j\omega_k})$ has the below expression:

\[X(e^{j\omega_k})= [ x(n_1) \cdot e^{-j \cdot \frac{\pi}{M}\cdot k \cdot n_1} + \ldots + x(n_N) \cdot  e^{-j \cdot \frac{\pi}{M}\cdot k \cdot n_N} ] \]

if we define the matrix $\mathbf{W}=\begin{bmatrix} W_0\\ \vdots \\W_M \end{bmatrix}$ where the k-th row $W_k$ has the below expression:

\[ W_k = \begin{bmatrix} e^{-j \cdot \frac{\pi}{M}\cdot k \cdot n_1}, \ldots, e^{-j \cdot \frac{\pi}{M}\cdot k \cdot n_N} \end{bmatrix} \]

and $\mathbf{x(n)} = \begin{bmatrix} x(n_1) \\ \vdots  \\ x(n_N) \end{bmatrix}$, we can write (3.2) as follow:

\[\mathbf{X} = \mathbf{W} \cdot \mathbf{x}\]

Let's consider the row vectors ${\bf k}$ and ${\bf n}$ that represent, respectively, the index of the frequency and for the time.
\[ \begin{matrix}
\mathbf{k}= \begin{bmatrix} 0, 1, \ldots, M\end{bmatrix}\\
\mathbf{n}= \begin{bmatrix} n_1, n_2, \ldots, n_N\end{bmatrix}
\end{matrix} \]
the product $\mathbf{k^T \cdot n}$ is equal to:
\[ \begin{bmatrix} 0 \\1 \\ \vdots \\M\end{bmatrix} \cdot \begin{bmatrix} n_1 & n_2 & \ldots & n_N\end{bmatrix} = \begin{bmatrix} 0 & 0 & \ldots & 0 \\ n_1 & n_2 & \ldots & n_N \\ \vdots & \vdots & \ddots & \vdots \\ M \cdot n_1 & M \cdot n_2 & \ldots & M \cdot n_N \end{bmatrix} \]
so we can rewrite $\mathbf{W}$

\[ \mathbf{W} = \begin{bmatrix} W_0 \\ W_1 \\ \vdots \\ W_M \end{bmatrix} = \begin{bmatrix} e^{-j \cdot \frac{\pi}{M}\cdot 0 \cdot n_1} \ldots e^{-j \cdot \frac{\pi}{M}\cdot 0 \cdot n_N} \\ \vdots \\ e^{-j \cdot \frac{\pi}{M}\cdot M \cdot n_1} \ldots e^{-j \cdot \frac{\pi}{M}\cdot M \cdot n_N} \end{bmatrix} =
 exp[-j \cdot \frac{\pi}{M} \cdot \mathbf{k^T \cdot n}] \]

finally the complete matrix form for DTFT is
\[\mathbf{X} = exp[-j \cdot \frac{\pi}{M} \cdot \mathbf{k^T \cdot n}] \cdot \mathbf{x} \tag{3.3} \]

Using of filter command in the convolution computation MATLAB

In the MATLAB exists the command filter that permits the resolution of difference equation numerically.
An important difference equation that express a link between input ad output in a LTI is
\[\sum_{k=0}^N a_ky(n-k) = \sum_{m=0}^M b_mx(n-m)\]
this equation can be solved using filter(b,a,x) where $b=[b_0 ... b_m],\; a=[a_0 ... a_k]$.
\[z(n) = conv(x,y) = x(n)*y(n) = \sum_{k=-\infty}^{+\infty} x(k)y(n-k)\]
Considering two sequence with finite length we can observe that:
\[filter(x,1,y)\Rightarrow 1\cdot z(n) =\sum_{m=0}^M x(m)y(n-m)\]
from below expression we can see that using filter function we can compute convolution of x and y but the result will have the dimension of x

1. Discrete Time Sinusoidal Signal

A discrete time sinusoid has this expression:
\[A\cos(\omega_0n + \theta),\qquad n \in]-\infty, +\infty[\tag{1.1}\]
where:
$A$ is the Amplitude
$\omega_0$ is the Frequency in radians per sample
$\theta$ is the Phase in radians.
We can express frequency with the variable $f_0$ defined by $\omega_0 = 2\pi f_0$
and the (1) becomes:
\[A\cos(2\pi f_0 n + \theta),\qquad n \in]-\infty, +\infty[\tag{1.2}\]
Now the frequency $f_0$ has dimensions of cycles per samples.

A discrete time sinusoid is periodic only if its frequency $f_0$ is a rational number.

A discrete time signal x(n) is periodic with period N (N>0) if and only if:
\[x(n+N) = x(n), \qquad \forall n\tag{1.3}\]
The smallest value of N is called fundamental period.For a sinusoid with frequency $f_0$ we should have:
\[\cos[2\pi f_0(N+n) + \theta] = \cos(2\pi f_0n+\theta)\]
This relation is true if and only if
\[2\pi f_0N=2k\pi\Rightarrow f_0=\frac{k}{N}\]
in other words the sinusoid is periodic if we can express the frequency as a rational.
The fundamental period is N (the value we obtain cancelling common factors so that k and N are relatively prime).
The absolute value of k is the number of cycles the phasor makes before going to the initial position.

EXAMPLE 1.a
Let's consider the discrete sequence $x(n) = \cos(0.3\pi n)$, we can see that $\omega_0=0,3\pi$ and so since $f_0=\frac{\omega_0}{2\pi}$ we obtain that $f_0=\frac{0.3\pi}{2\pi}\Rightarrow f_0=\frac{3}{20}$
The sequence x(n) is periodic and N=20

EXAMPLE 1.b
Let's consider the discrete sequence $x(n) = \cos(0.3n)$, we can see that $\omega_0=0,3$ and so since $f_0=\frac{\omega_0}{2\pi}$ we obtain that $f_0=\frac{0.3}{2\pi}\Rightarrow f_0=\frac{3}{20\pi}$
The sequence x(n) is not periodic

A discrete time sinusoid is periodic in frequence with period of $2\pi$
To prove this assertion, let us consider a sinusoid $\cos[\omega_0n+ \theta]$ and add $2\pi$ to $\omega_0$, we obtain:
\[\cos[(\omega_0+2\pi)n+ \theta]=\cos[\omega_0n+2\pi n+ \theta]=\cos[\omega_0n+ \theta] \]
in fact n is an integer number.
This result is more important because it permits us to study the sinusoid only in the frequency range of $2\pi$ (i.e. $-\pi\leq\omega\leq\pi$ or $-\frac{1}{2}\leq f \leq\frac{1}{2}$)