I have a nondimensionalized linear perturbation system relevant to the appropriate pure conduction solution for Rayleigh-Bénard convection in upper planetary atmospheres under the compressible gas version of the Boussinesq approximation: $$ \frac{\partial u}{\partial x} + \frac{\partial w}{\partial z} = 0\\ \frac{\partial u}{\partial t} = -\frac{\partial p}{\partial x} + (m\frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial z^2})u\\ \frac{\partial w}{\partial t} = -\frac{\partial p}{\partial z} + R\theta +(m\frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial z^2})w\\ \frac{\partial \theta}{\partial t} = w + (m\frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial z^2})\theta\\ w = \theta = \frac{\partial u}{\partial z} + \frac{\partial w}{\partial x} = 0 \text{ at } z = 0,1 $$ Here the gas has perturbation velocity components $(u,v)$, reduced pressure $p$, and temperature $\theta$ which are functions of the spatial coordinates $(x,z)$ and time $t$. Further, $R$ is the compressible fluid Rayleigh number and to account for the role of eddies an eddy kinematic viscosity and thermometric conductivity, which are assumed equal, have been introduced where $m$ is the ratio of the horizontal to the vertical diffusivity.
Seeking a normal mode solution of this system of the form: $$ u(x,z,t) = A\sin(nx)\cos(\pi z)e^{\sigma t}\\ [w,\theta](x,z,t) = [B,C]\cos(nx)\sin(\pi z)e^{\sigma t}\\ p(x,z,t) = D\cos(nx)\cos(\pi z)e^{\sigma t} $$ obtain the secular equation $$ k_1^2\sigma^2 + 2k_1^2k_m^2\sigma + k_1^2k_m^4 - n^2R = 0 $$ where $$ k_m^2 = mn^2 + \pi^2 = \pi^2(1 + \frac{4m}{\lambda^2}) \text{ for } \lambda = \frac{2\pi}{n} $$ I'm not quite sure on where to begin. All of the other parts of the question are fairly simple, but this one is stumping me completely. I'd really just need a little help getting pointed in the right direction, if that wouldn't be too much to ask for.
Substitute the normal mode solution into the partial differential equations. The boundary conditions at $z = 0,1$ are already satisfied.
This leads to the following homogeneous system of algebraic equations for $A$, $B$, $C$, and $D$.
$$nA + \pi B = 0\\(\sigma + k_m^2)A - nD = 0\\ (\sigma + k_m^2)B -RC - \pi D = 0\\-B + (\sigma + k_m^2)C = 0.$$
A solution is obtained when the determinant of this system equals zero -- generating the secular equation.
$$\begin{vmatrix}n & \pi & 0 & 0 \\ \sigma + k_m^2 & 0 & 0 & -n \\ 0 & \sigma + k_m^2 & -R & -\pi \\ 0 & -1 & \sigma + k_m^2 & 0\end{vmatrix}= 0$$
Expanding the determinant leads to
$$n^2[(\sigma+ k_m^2)^2 -R] + \pi^2 (\sigma + k_m^2)^2 = 0,$$
which can be rearranged into your form:
$$(n^2+\pi^2)\sigma^2 + 2(n^2+\pi^2)k_m^2\sigma+ (n^2+\pi^2)k_m^4-n^2R = 0.$$
With $k_1^2 = n^2 + \pi^2$ this is
$$k_1^2\sigma^2 + 2k_1^2k_m^2\sigma+ k_1^2k_m^4-n^2R = 0.$$