Finding the feature map that approximates the first three terms of Maclaurin series

120 Views Asked by At

Consider the Maclaurin series expansion for the exponential function,

\begin{equation} e^t = \sum_{n=0}^{\infty} \frac{t^n}{n!} = 1 + t + \frac{t^2}{2!} + \frac{t^3}{3!} + \ldots \end{equation}

Using only the first three terms in this series, we obtain $\exp(t) \approx 1 + t + \frac{t^2}{2}$. Use this approximation to obtain a feature map $\phi: \mathbb{R}^D \rightarrow \mathbb{R}^M$ such that, for arbitrary $q \in \mathbb{R}^D$ and $k \in \mathbb{R}^D$, we have $\exp(q^\top k) \approx \phi(q)^\top \phi(k)$. Express the dimensionality of the feature space $M$ as a function of $D$. What would be the dimensionality if you used $K \geq 3$ terms in the Maclaurin series expansion (as a function of $D$ and $K$)?

After putting some thought in this problem, I got the following feature map:

\begin{equation} \boldsymbol{\phi}(q) = \begin{bmatrix} 1 \\ q \\ \frac{q^2}{\sqrt{2}} \\ \end{bmatrix} \end{equation}

Is this solution correct? If yes, what is the dimensionality of the feature space?

1

There are 1 best solutions below

2
On BEST ANSWER

Your solution is not correct. In order to build the correct feature map, it helps to consider a particular low-dimensional example in detail.

Take $D = 2$. Our approximation for $\exp(q^Tk)$ can be written as follows. \begin{align} 1 + (q^Tk) + \frac{(q^Tk)^2}{2!} &= 1+(q_1k_1 + q_2k_2) + \frac{(q_1k_1 + q_2k_2)^2}{2!} \\ & = 1 + q_1k_1 + q_2k_2 + \frac 12 q_1^2k_1^2 + \frac 12 q_2^2 k_2^2 + q_1q_2 k_1k_2 \end{align} With each of the terms written out like this, it is easy to see that the following feature map will work: $$ \phi(x) = (1,x_1,x_2,\frac 1{\sqrt{2}} x_1^2, \frac{1}{\sqrt{2}}x_2^2, x_1x_2). $$ I recommend that you verify with a detailed calculation that $\phi(q)^T\phi(k) = \exp(q^Tk)$.

To generalize this, we can take $\phi:\Bbb R^D \to \Bbb R^M$ to be something like $$ \phi(x) = (1,x_1,\dots,x_D, x_1^2/\sqrt{2},\dots,x_D^2/\sqrt{2},x_1x_2, x_1x_3,\dots,x_2x_3,x_2x_4,\dots,x_{D-1}x_D). $$ In this case, we have $$ M = 1 + D + D + \frac{D(D-1)}{2} = \frac{D^2 + 3D + 2}{2}. $$