I have a bilevel optimization problem where I'm solving an outer optimization problem and for every iteration of the outer problem $t$, I'm interested in solving a quadratic minimization problem where the matrix $Q_t$ and the vector $b_t$ change with every iteration $t$, specifically, the inner optimization problem is given by:
$$\underset{x \in \mathbb{R}^d}{\min} \left\{f_t(x) = \frac{1}{2} x^{T} Q_t x - x^{T}b_t\right\}$$
For computation reasons, I want to run a few (ideally one) steps in the inner problem without solving it explicitly and then run a step in the outer problem. My question is which algorithm can give me a good approximation of the optimal solution of the inner problem with a few iterations?
This is an unconstrained nonlinear optimization (minimization) problem. If your coefficients change the problem is no longer quadratic.
When you say $Q$ and $b$ change with every iteration, this is not very precise as the mathematical formulation of the optimization problem has no concept of "iteration". Perhaps it is more appropriate to say $Q$ and $b$ are dependent on $x$. Then you arrive at the correct problem formulation.
For nonlinear optimization there exist solvers, but perhaps not as efficient as for the quadratic case.