I am going through Michael Jordan's notes on exponential families and an exponential family is defined w.r.t. functions $h(\cdot), T(\cdot)$, and parameter $\eta$ such that $$ p(x | \eta) = h(x) \exp\bigl\{\eta^\top T(x) - A(\eta)\bigr\} $$ with $$ A(\eta) = \log \int h(x) \exp\left(\eta^\top T(x) \right) \, dx. $$
The thing that strikes me as odd about this definition is that it appears too flexible due to the choice of "$h$". For instance, say we define our family in terms of $h_1, \eta_1$, and $T_1$. We could then reparameterize our family in terms of $h_2, \eta_2$, and $T_2$ such that
- $\eta_2 = 0$
- or alternatively, $T_2(x) = 0$
- $h_2(x) = h_1(x) \exp\left(\eta_1^\top T_1(x)\right)$
so the choices of $\eta$ and $T(\cdot)$ are not important (at least in terms of defining the density).
Moreover, it seems we might use this trick to mimic any density (not necessarily an exponential family), simply by letting $\eta = 0$ and making a "sneaky" choice for $h(\cdot)$.
What am I missing here?