Here $B_t$ is Brownian motion and $P(x)$ is a polynomial of degree $n, n \ge 1$
Assuming $f(t,B_t) = X_t$, I used Ito lemma and used $f_t + f_{xx} = 0$ to equate the drift part of the SDE to $0$. So I get something like this:
$$ \frac{a}{t}P(\frac{B_t}{t^b}) - \frac{b}{t^{b+1}}P'(\frac{B_t}{t^b}) + \frac{1}{t^{2b}}P''(\frac{B_t}{t^b}) = 0 $$
because $$ f_t = at^{a-1}P(\frac{B_t}{t^b}) - b\frac{t^a}{t^{b+1}}P'(\frac{B_t}{t^b})$$ and $$f_{xx} = \frac{t^a}{ t^{2b}}P''(\frac{B_t}{t^b}) $$
What should a and b be from this equation so that $X_t = t^a P(\dfrac{B_t}{t^b})$ is a martingale?
Earlier I tried using the general form of a polynomial like this
$P(x) = c_n x^n + c_{n-1} x^{n-1} + \dots c_1 x + c_0$
and tried differentiating that. But it gave me extremely complicated terms when I differentiate it twice.
Edit: I further simpified the DE as follows:
$$at^{2b -1}P(\frac{B_t}{t^b}) - b t^{b-1}P'(\frac{B_t}{t^b}) + P''(\frac{B_t}{t^b}) = 0 $$
A trivial solution would be that the polynomial $P$ is identically zero.
Another solution is that $a=0, b=1$ along with $P' = 0$ Is that all?
The second part of the question asks what $X_t$ looks like for $n = 2,3,4$. Shouldn't $n$ be equal to $1$ for $P' = 0$?
Guys, anyone? I am probably doing this wrong but can't figure out what exactly.
For the partial derivative wrt. $t$ you missed one inner derivative $$ f_t(t,u)=at^{a-1}P(t^{-b}u)-bt^{a-b-1}uP'(t^{-b}u) $$ so that the drift term in the Ito formula is $$ 0=f_t+\tfrac12f_{uu}=t^{a-1}(aP(t^{-b}u)-bt^{-b}uP'(t^{-b}u)+\tfrac12t^{1-2b}P''(t^{-b}u)) $$ As $t$ and $u$ are independent, either $P''=0$ or $b=\frac12$. In the more interesting second case setting $w=t^{-b}u$ gives the equation $$ 0=2aP(w)-wP'(w)+P''(w) $$ In the highest degree coefficient of the degree $n$ polynomial one gets $$ 0=2ac_n-nc_n $$ so that $a=\frac{n}2$. As also $0=2ac_{n-1}-(n-1)c_{n-1}$ one gets $c_{n-1}=0$ (and then $c_{n-3}=0$, $c_{n-5}=0$,...). With $c_n=1$ the other coefficients are uniquely determined.
For $n=2$ then $0=2c_0+2c_2$ giving $f(t,B_t)=B_t^2-t$.
For $n=3$ further $0=3c_1-c_1+6c_3\implies c_1=-3$ and $0=3c_0+2c_2$ so that $c_0=0$ and $$f(t,B_t)=t^{3/2}((B_t/\sqrt{t})^3-3(B_t/\sqrt{t}))=B_t^3-3tB_t.$$
...