I am trying to solve $$\frac{d^2y}{dx^2} + 17\frac{dy}{dx} + 16y = e^x+\sec^2(x)$$ I get the complementary solution to be $y_c=C_1e^{-x} + C_2e^{-16x}$. The first complementary solution, which is for $$\frac{d^2y}{dx^2} + 17\frac{dy}{dx} + 16y = e^x$$ is $y_{p1}=C_3e^x$.
To find the second complementary solution, which is for $$\frac{d^2y}{dx^2} + 17\frac{dy}{dx} + 16y = \sec^2(x)$$ we compute the Wronskian, we have $$W(e^{-x},e^{-16x}) = e^{-x} (-16e^{-16x}) - e^{-16x}(-e^{-x})=-15e^{-17x}$$ So, the particular solution is : . $$y_{p2}=-e^{-x}\int \frac{e^{-16x}\sec^2(x)}{-15e^{-17x}} dx + e^{-16x}\int\frac{e^{-x}\sec^2(x)}{-15e^{-17x}} dx$$ which simplifies to $$y_{p2}=\frac{1}{15}e^{-x}\int e^x\sec^2(x)dx -\frac{1}{15} e^{-16x}\int e^{16x}\sec^2(x)dx$$
I am stuck here to find the integral. Any help appreciated, or is there another method to solve this ODE?
First of all $C_3=\frac 1{34}$.
Then, as I suspected (refer to my first comment), I think that you face a terrible problem with the $\sec^2(x)$ since the solution involves hypergeometric functions.
Using a CAS, I received (please fasten you seat belt) $$\frac{1}{975} \left((8-i) e^{2 i x} \left((3+2 i) \, _2F_1\left(1,1-\frac{i}{2};2-\frac{i}{2};-e^{2 i x}\right)-8 i \, _2F_1\left(1,1-8 i;2-8 i;-e^{2 i x}\right)\right)-65 i \left(\, _2F_1\left(-\frac{i}{2},1;1-\frac{i}{2};-e^{2 i x}\right)-\, _2F_1\left(-8 i,1;1-8 i;-e^{2 i x}\right)\right)\right)$$ Even with $\sec(x)$ the same kind of monster appears.
I am pretty sure that the equation is in fact $$\frac{d^2y}{dx^2} + 17\frac{dy}{dx} + 16y = e^x+\color{red}{\sin^2(x)}$$
Probably one more typo in a textbook.