Formulating the polynomial regression

Question

I'm trying to formulate a regression problem such that $y=ax^{b}$. Previously, I formulated the $y=ax+b$ like $y=Ac+e$ where $c= \begin{bmatrix} a\\ b \end{bmatrix}$ and $A= \begin{bmatrix} x_1 & 1\\ x_2 & 1\\ . & \\ . & \\ x_n & 1 \end{bmatrix}$

Can I only change the A matrix such that $A= \begin{bmatrix} x_1^{b} & 1\\ x_2^{b} & 1\\ . & \\ . & \\ x_n^{b} & 1 \end{bmatrix}$ and change c as $c= \begin{bmatrix} a\\ 0 \end{bmatrix}$ and write the same formula $y=Ac+e$

Am I correct? Any help would be appreciated. Many thanks

Answer 1

No. You need all the powers in the columns of A.

Write the problem as minimizing $D=\sum_{k=1}^m (y_k-\sum_{j=0}^n a_jx_k^j)^2$.

Set $\partial D/\partial a_j =0$ for each $j$ and see what you get.

Answer 2

You have three alternatives to solve this problem.

1. Alternative: As Jens Schwaiger proposed you can rewrite the equation as $\ln y_i = \ln a + b \ln x_i+ \tilde{\varepsilon}_i$. If we introduce the coefficient $\tilde{a}=\ln a$ and the transformed outputs $\tilde{y}_i=\ln y_i$ then it is possible to perform a standard linear regression. the coefficients $\boldsymbol{w}=[\tilde{a},b]^T=[\ln a, b]^T$ can be estimated by the least squares estimate

$$\hat{\boldsymbol{w}}=\left[\boldsymbol{\Phi}^T\boldsymbol{\Phi} \right]^{-1}\boldsymbol{\Phi}^T\tilde{\boldsymbol{y}},\qquad (*)$$

in which $\tilde{\boldsymbol{y}}=\left[\ln y_1, \ln y_2, \ldots, \ln y_n\right]^T$ is the transformed output vector and

$$\boldsymbol{\Phi} = \begin{bmatrix}1 & \ln x_1\\1 & \ln x_2 \\ \vdots & \vdots \\1 & \ln x_n \end{bmatrix}.$$

2. Alternative: We have $y_i = ax_i^{b}+\varepsilon_i=a\exp(b\ln x_i)+\varepsilon_i$. We can now expand the exponential function by a taylor series.

$$y_i=a\left[1+b\ln x_i + b^2/2!\left[\ln x_i\right]^2+b^3/3![\ln x_i]^3+\cdots \right]+\varepsilon_i $$

If we truncate the series to the $m^{\text{th}}$ power then we can approximate $y_i$ as

$$y_i \approx \left[a+ab\ln x_i + ab^2/2!\left[\ln x_i\right]^2+ab^3/3![\ln x_i]^3+\cdots +ab^m/m!\left[\ln x_i \right]^m\right]+\varepsilon_i.$$

By introducing the coefficients $w_l = ab^l/l!$ we can rewrite the previous equation as

$$y_i \approx \left[w_0+w_1\ln x_i + w_2\left[\ln x_i\right]^2+w_3[\ln x_i]^3+\cdots +w_m\left[\ln x_i \right]^m\right]+\varepsilon_i.$$

The coefficients are given by equation $(*)$ but now $\tilde{\boldsymbol{y}}=\boldsymbol{y}=\left[y_1, y_2, \ldots,y_n \right]^T$ and

$$\boldsymbol{\Phi} = \begin{bmatrix}1 & \ln x_1 & [\ln x_1]^2 & \cdots & [\ln x_1]^m\\1 & \ln x_2 & [\ln x_2]^2 & \cdots & [\ln x_2]^m\\ \vdots & \vdots & \vdots & \ddots & \vdots \\1 & \ln x_n & [\ln x_n]^2 & \cdots & [\ln x_n]^m\\\end{bmatrix}.$$

After having obtained the coefficients $\boldsymbol{w}=[a, ab, ab^2, ..., ab^m]$ you can determine $a=w_0$ and $b=w_1/w_0$ and so forth.

3. Alternative: Full nonlinear least squares (proposed by Marty Cohen), which does not have a closed form solution. Here we use the objective function $E(\boldsymbol{w}=[a,b]^T)$ which is

$$E(\boldsymbol{w})=\sum_{i=1}^{n}[y_i-ax_i^b]^2.$$

The partial derivatives are given by

$$\dfrac{\partial E}{\partial a} = \sum_{i=1}^n2[y_i-ax_i^b](-x_i^b)$$ $$\dfrac{\partial E}{\partial b} = \sum_{i=1}^n2[y_i-ax_i^b](-ax_i^b\ln x_i).$$

After setting these partial derivatives equal to zero you will have to solve a nonlinear equation in the coefficients $a$ and $b$. You can try to numerically solve this equation by using Newton-Raphson.