Need help simplifying an iterative diffusion process to a 1-step process

73 Views Asked by At

We have the following function that adds noise to an input, $x_{t-1}$:

$$ q(x|x_{t-1}) = \mathcal{N}(x_t, \sqrt{1-\beta_t}x_{t-1},\beta_t\mathbb{I}) $$

This process is iterative, to reach $x_5$ from $x_0$ we must call $q$ 5 times.

Using a bit of math, we can transform this into a 1-step process (from here):

derivation

Re-written in $\LaTeX$:

$$ \begin{align} q(x|x_{t-1})& = \mathcal{N}(x_t, \sqrt{1-\beta_t}x_{t-1},\beta_t\mathbb{I})\\ & = \sqrt{1 - \beta_t}x_{t-1} + \sqrt{\beta_t}\epsilon\\ & = \sqrt{a_t}x_{t-1} + \sqrt{1-a_t}\epsilon\\ & = \sqrt{a_ta_{t-1}}x_{t-2} + \sqrt{1-a_ta_{t-1}}\epsilon\\ & = \ldots \end{align} $$

where $$ a_t = 1 - \beta_t $$

I'm having trouble understanding how we got from the 2nd to last step to the last step in the simplification. It does not seem true that $a_ta_{t-1} = 1-\beta_{t-1}$.

Would appreciate an explanation. Thanks!

Edit

Using a suggestion from @sku, I tried the following:

$$ \begin{align} x_{t-1} &= \sqrt{1-\beta_{t-1}}x_{t-2} + \sqrt{\beta_{t-1}}\epsilon\\ \sqrt{a_t}x_{t-1} + \sqrt{1-a_t}\epsilon &= \sqrt{a_t}(\sqrt{1-\beta_{t-1}}x_{t-2} + \sqrt{\beta_{t-1}}\epsilon) + \sqrt{1-a_t}\epsilon\\ &= \sqrt{a_ta_{t-1}}x_{t-2} + \sqrt{a_t}\sqrt{\beta_{t-1}}\epsilon + \sqrt{1-a_t}\epsilon \end{align} $$

However, I'm a bit stuck from here. I can't figure out how to simplify further.

1

There are 1 best solutions below

0
On BEST ANSWER

They are not using straightforward algebra.

We start with:

$x_t = \sqrt{1 - \beta_t}x_{t-1} + \sqrt{\beta_t}\epsilon$

$= \sqrt{\alpha_t}x_{t-1} + \sqrt{1 - \alpha_t}\epsilon$

Substitute $x_{t-1} = \sqrt{\alpha_{t-1}}x_{t-2} + \sqrt{1 - \alpha_{t-1}}\epsilon$

we get:

$x_t = \sqrt{\alpha_t}\left(\sqrt{\alpha_{t-1}}x_{t-2} + \sqrt{1 - \alpha_{t-1}}\epsilon\right) + \sqrt{1 - \alpha_t}\epsilon$

$ = \sqrt{\alpha_t\alpha_{t-1}}x_{t-2} + \sqrt{\alpha_t}\sqrt{1 - \alpha_{t-1}}\epsilon + \sqrt{1 - \alpha_t}\epsilon$

Now the last two terms are noise terms which are gaussian with mean $0$ and standard deviation $\sqrt{\alpha_t}\sqrt{1 - \alpha_{t-1}}$ and $ \sqrt{1 - \alpha_t}$. So the sum of those terms is a gaussian with mean $0$ and variance equal to sum of $\alpha_t(1 - \alpha_{t-1}) + (1 - \alpha_t) = 1 - \alpha_t\alpha_{t-1}$

So now we have:

$x_t = \sqrt{\alpha_t\alpha_{t-1}}x_{t-2} + \sqrt{1 - \alpha_t\alpha_{t-1}}\epsilon$

and so on.