2.1. Linear Algebra

In deep learning, operations like dot product and matrix multiplication are fundamental building blocks. Additionally, the Hadamard product finds application in specific cases.

References

Textbooks

Tutorials

2.1.1. Dot-product

$$ a \cdot b = \sum^{n}_{i=1} a_{i}b_{i} = a_{1}b_{1} + a_{2}b_{2} + \cdots + a_{n}b_{n} $$
Example:
$$ a = (1, 2, 3, 4), \ \ b = (5, 6, 7, 8) $$ $$ \Downarrow $$ $$ a \cdot b = 1 \cdot 5 + 2 \cdot 6 + 3 \cdot 7 + 4 \cdot 8 = 5 + 12 + 21 + 32 = 70 $$
>>> import numpy as np
>>> a = np.arange(1, 5).reshape(4)
>>> a
array([1, 2, 3, 4])
>>> b = np.arange(5, 9).reshape(4)
>>> b
array([5, 6, 7, 8])
>>> np.dot(a, b)
70
2.1.1.1. Dot Product Similarity

Dot-product is often used to measure the similarity between two vectors.

References

For example, Vectors $A$ and $B$ are same, so the similarity of them is $|A|\cdot|B| = 1$.

>>> import numpy as np
>>> A = np.array([1, 0])
>>> B = A
>>> A
array([1, 0])
>>> B
array([1, 0])
>>> np.dot(A, B)
1

In contrast, Vector $C$ is orthogonal to $A$, so the similarity of $A$ and $C$ is $0$.

>>> C = np.array([0,1])
>>> np.dot(A, C)
0

2.1.2. Matrix multiplication

$$ A = \begin{pmatrix} a_{11} & a_{12} & \ldots & a_{1n} \\ a_{21} & a_{22} & \ldots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \ldots & a_{mn} \end{pmatrix}, B = \begin{pmatrix} b_{11} & b_{12} & \ldots & b_{1p} \\ b_{21} & b_{22} & \ldots & b_{2p} \\ \vdots & \vdots & \ddots & \vdots \\ b_{n1} & b_{n2} & \ldots & b_{np} \end{pmatrix} $$ $$ \Downarrow $$ $$ C = AB = \begin{pmatrix} c_{11} & c_{12} & \ldots & c_{1p} \\ c_{21} & c_{22} & \ldots & c_{2p} \\ \vdots & \vdots & \ddots & \vdots \\ c_{m1} & c_{m2} & \ldots & c_{mp} \end{pmatrix} $$

where $ c_{ij} = a_{i1}b_{1j} + a_{i2}b_{2j} + \cdots + a_{in}b_{nj} = \sum_{k=1}^{n} a_{ik}b_{kj} $.

Info

The complexity of the matrix multiplication of $AB$, where $A \in \mathbb{R}^{m \times n} $ and $B \in \mathbb{R}^{n \times p} $, is $\mathcal{O}(m \cdot n \cdot p)$.

Example
$$ A = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}, B = \begin{pmatrix} 5 & 6 & 7 \\ 8 & 9 & 10 \end{pmatrix} $$ $$ \Downarrow $$ $$ \begin{align} C &= AB = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \begin{pmatrix} 5 & 6 & 7 \\ 8 & 9 & 10 \end{pmatrix} = \begin{pmatrix} 1 \cdot 5 + 2 \cdot 8 & 1 \cdot 6 + 2 \cdot 9 & 1 \cdot 7 + 2 \cdot 10 \\ 3 \cdot 5 + 4 \cdot 8 & 3 \cdot 6 + 4 \cdot 9 & 3 \cdot 7 + 4 \cdot 10 \end{pmatrix} \\ &= \begin{pmatrix} 21 & 24 & 27 \\ 47 & 54 & 61 \end{pmatrix} \end{align} $$
>>> import numpy as np
>>> A = np.arange(1, 5).reshape(2,2)
>>> A
array([[1, 2],
       [3, 4]])
>>> B = np.arange(5, 11).reshape(2,3)
>>> B
array([[ 5,  6,  7],
       [ 8,  9, 10]])
>>> np.dot(A,B)
array([[21, 24, 27],
       [47, 54, 61]])

2.1.3. Hadamard product

$$ A = \begin{pmatrix} a_{11} & a_{12} & \ldots & a_{1n} \\ a_{21} & a_{22} & \ldots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \ldots & a_{mn} \end{pmatrix}, B = \begin{pmatrix} b_{11} & b_{12} & \ldots & b_{1n} \\ b_{21} & b_{22} & \ldots & b_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ b_{m1} & b_{m2} & \ldots & b_{mn} \end{pmatrix} $$ $$ \Downarrow $$ $$ C = A \odot B = \begin{pmatrix} c_{11} & c_{12} & \ldots & c_{1n} \\ c_{21} & c_{22} & \ldots & c_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ c_{m1} & c_{m2} & \ldots & c_{mn} \end{pmatrix} \\ $$

where $ c_{ij} = a_{ij}b_{ij} $.

Example
$$ A = \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix}, B = \begin{pmatrix} 7 & 8 & 9 \\ 10 & 11 & 12 \end{pmatrix} $$ $$ \Downarrow $$ $$ C = A \odot B = \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{pmatrix} \odot \begin{pmatrix} 7 & 8 & 9 \\ 10 & 11 & 12 \end{pmatrix} = \begin{pmatrix} 1 \cdot 7 & 2 \cdot 8 & 3 \cdot 9 \\ 4 \cdot 10 & 5 \cdot 11 & 6 \cdot 12 \end{pmatrix} = \begin{pmatrix} 7 & 16 & 27 \\ 40 & 55 & 72 \end{pmatrix} $$
>>> import numpy as np
>>> A = np.arange(1, 7).reshape(2, 3)
>>> A
array([[1, 2, 3],
       [4, 5, 6]])
>>> B = np.arange(7, 13).reshape(2, 3)
>>> B
array([[ 7,  8,  9],
       [10, 11, 12]])
>>> A*B
array([[ 7, 16, 27],
       [40, 55, 72]])

2.1.4. Transpose

$$ (AB)^{T} = B^{T} A^{T} $$
Example
$$ (AB)^{T} = \left( \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \begin{pmatrix} 5 & 6 & 7 \\ 8 & 9 & 10 \end{pmatrix} \right)^{T} = \begin{pmatrix} 21 & 24 & 27 \\ 47 & 54 & 61 \end{pmatrix}^{T} = \begin{pmatrix} 21 & 47 \\ 24 & 54 \\ 27 & 61 \end{pmatrix} $$ $$ \Updownarrow $$ $$ B^{T} A^{T} = \begin{pmatrix} 5 & 6 & 7 \\ 8 & 9 & 10 \end{pmatrix}^{T} \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}^{T} = \begin{pmatrix} 5 & 8 \\ 6 & 7 \\ 7 & 10 \end{pmatrix} \begin{pmatrix} 1 & 3 \\ 2 & 4 \end{pmatrix} = \begin{pmatrix} 21 & 47 \\ 24 & 54 \\ 27 & 61 \end{pmatrix} $$
>>> import numpy as np
>>>
>>> A = np.arange(1, 5).reshape(2,2)
>>> B = np.arange(5, 11).reshape(2,3)
>>> A
array([[1, 2],
       [3, 4]])
>>> B
array([[ 5,  6,  7],
       [ 8,  9, 10]])

>>>
>>> np.dot(A,B).T
array([[21, 47],
       [24, 54],
       [27, 61]])
>>>
>>> np.dot(B.T,A.T)
array([[21, 47],
       [24, 54],
       [27, 61]])