Generally speaking, this is a question about when we can swap $A \rightarrow\lambda$.
In problem 2-5(c) of Applied Optimal Estimation, we are asked to consider the matrix $$A = \left[\begin{matrix} 1 & 2 \\ 3 & 4\end{matrix}\right],$$ whose characteristic polynomial is $$p(\lambda) = \lambda^2-5\lambda-2.$$ We then Taylor expand $e^{At}$ and use Cayley-Hamilton to simplify, yielding: $$e^{At} = a_1(t)I + a_2(t)A.$$
Part (c) of the question is getting a closed form expression for $a_1(t)$ and $a_2(t)$. To do that, this solution pdf swaps out $\lambda_1$ and then $\lambda_2$ for $A$, which gives two equations for the two unknowns.
Why can one swap out $\lambda_i$ for $A$ in that expression, if it is not the characteristic polynomial? Does this hold in general, that while we can't necessarily swap $A$ for $\lambda$, we can always do the reverse?
You obtained that expression using only the fact that $A$ is a "zero" of the characteristic polynomial. Similarly $\lambda_1$ and $\lambda_2$ are zeros of the characteristic polynomial.
If you are still not convinced, just expand $e^{\lambda_1 t}$ and use what you know about $\lambda_1$ to simplify in exactly the same manner as you did with $A$.