Could anyone suggest me how to linearize the following system of nonlinear odes (special attention to (2) \begin{align} -cU'&=-U''+UV\tag{1}\\ -cV'&=-k(k+1)V^{k-1}(V')^2+(k+1)V^k V''+UV-\mu V-\gamma V^2\tag{2} \end{align}
?
Take $V=V(x)$, $U=U(x)$, $\mu$, $\gamma$ arbitrary constants. $k$ takes integer values.
using \begin{align} V(x) = V_0 + \delta V,\\ U(x) = U_0 + \delta U. \end{align}
we find the linearized equations are $$ -c\delta U' = -\delta U'' + U_0\delta V + V_0\delta U + U_0V_0,\\ -c\delta V' = \left(U_0V_0 - \mu V_0 -\gamma V_0^2\right) + \left(U_0 -\mu -2V_0\right)\delta V\\ +(k+1)V_0^k\delta V'' + V_0\delta U $$ From the first equation the equilibrium point occurs at $U_0V_0=0$ and therefore the first linearized equation is $$ -c\delta U' = -\delta U'' + U_0\delta V + V_0\delta U. $$ The second equation the eqluibrium points is given by $$ U_0V_0 - \mu V_0 -\gamma V_0^2 =0 $$ which leads to the linearized equation $$ -c\delta V' = \left(U_0 -\mu -2V_0\right)\delta V +(k+1)V_0^k\delta V'' + V_0\delta U $$ where the $V_0,U_0$ are constants. Though if you mean using $$ V(x) = \tilde{V}(x) + \delta V $$ and linearize in that fashion then you can go through that yourself following the same convention with the exception that $\tilde{V}(x),\tilde{U}(x)$ are solutions of the original equation.
$\textbf{Appendix}$ To further expand on how I arrived at the above linear form. lets take $$ cU' = -U'' + UV (1). $$ we find the equilibrium points by setting the derivatives to zero as follows $$ 0 = -0 + UV\implies UV = U_0V_0 = 0 $$ The other equation will also yield an equilibrium equation which is set to zero. In the above I have been cheeky and not explicitly determined what the equilibrium points are, but in most cases I have come across you do not need to know that explicitly to form the linearised equations. so continuing we have equations of the form $f = f_0 + \delta f$ (as above) inserting into the Eq.(1) we find \begin{align} -c\left(U_0'+\delta U'\right) = -\left(U_0''+\delta U''\right)\\ +\left(U_0+\delta U\right)\left(V_0+\delta V\right) \end{align} But the terms $f_0'$ are zero(since $f_0$ are constants) so we can further reduce. $$ -c\delta U' = -\delta U''\\ +U_0V_0 +U_0\delta V + \delta U V_0 + \delta U\delta V $$ Now we linearise by neglecting terms of $O(\delta f^2)$ and higher $$ -c\delta U' = -\delta U'' + U_0V_0 +U_0\delta V + \delta U V_0 $$
Can you take it from here? and also do the second equation?
Analogously, the second equation would become $$ -c\delta V'=(k+1)V_0^k\delta V''+U_0 V_0-\mu V_0-\gamma V_0^2+(U_0-\mu-2V_0)\delta V+\delta UV_0 $$ ?
Note that in the first term of rhs, we have $(k+1)V_0^k\delta V''$ instead of $(k+1)(V_0+\delta V)^k\delta V''$ since we are ignoring the higher power terms in the binomial expansion.