I have a function of a matrix and a vector $f(A,b)=y^\top (I-A)^{-1} b$ and I want to know the conditions under which it is convex.
For functions of a vector, the positive definiteness of the Hessian is sufficient to claim convexity. How do we extend this to functions of matrices? I know the first derivative $\frac{\partial f}{\partial A}=(I-A)^{-\top}y\ b^\top(I-A)^{-\top}$, but how to extend this for finding convexity?
Convexity is the exception, not the rule. In my experience, nearly every question "is this function convex?" ends up being answered in the negative---because the cases where convexity is present tend to be somewhat obvious.
For general questions of computing derivatives involving vectors and matrices, the Matrix Cookbook is an essential resource.
This function is not convex, however, for any interesting case. Consider the scalar case $f(a,b)=by/(1-a)$. It is well known that the linear fractional function is neither convex nor concave, even when restricted to domains like $1-a>0$. This can be readily verified by examining the Hessian, which is indefinite.