Suppose $A,H$ are positive definite matrices. How do I find $\alpha$ which minimizes the following?
$$\|(I-\alpha H) A(I-\alpha H)^T\|_\text{op}$$
It can be written as minimizing linear function with semidefinite cone constraint, however this constraint has an $\alpha^2$ entry, so SDP solvers won't take it.
$$\begin{align} \text{minimize}_{\alpha,t}\ & t \\ \text{subject to } & A-\alpha AH - \alpha HA + \alpha^2 HAH \prec t I \end{align}$$
For matrices below I can use visual inspection to check that $\alpha\approx 3.2$ $$\text{H=}\left( \begin{array}{cc} 4 & 0 \\ 0 & 1 \\ \end{array} \right)\\ \text{A=}\left( \begin{array}{cc} 11 & 9 \\ 9 & 11 \\ \end{array} \right) $$
This has already been noted in the comments above but for completeness here is the SDP solving the above problem
$$ \begin{aligned} \min_{t,\alpha} & \quad t \\ \mathrm{s.t.} & \quad \begin{pmatrix} t\, \mathbb{I} & (\mathbb{I} - \alpha H) \\ (\mathbb{I} - \alpha H) & A^{-1} \end{pmatrix} \succeq 0 \end{aligned} $$
This SDP representation can be derived in two steps. Firstly note that the SDP $$ \begin{aligned} \min_{t} & \quad t \\ \mathrm{s.t.} & \quad t\, \mathbb{I} - X \succeq 0 \end{aligned} $$ is equal to $\|X\|_{\mathrm{op}}$ (largest eigenvalue of $X$) for any positive semidefinite $X$. Secondly we have by the Schur complement form of block PSD matrices $$ \begin{pmatrix} A & B \\ B^T & C \end{pmatrix} \succeq 0 \quad \iff \quad C \succeq 0, \quad A - B C^{-1} B^T \succeq 0, \quad (\mathbb{I} - C C^{-1} ) B^T = 0\,. $$ Using this result we see that the block positive semidefinite constraint in the first SDP is equivalent to the constraint $$ t\,\mathbb{I} - (\mathbb{I}- \alpha H) A (\mathbb{I} - \alpha H) \succeq 0 $$ and hence the first SDP is computing $\min_{\alpha} \|(\mathbb{I}- \alpha H) A (\mathbb{I} - \alpha H)\|_{\mathrm{op}}$.