Let's say I have $\widehat{\sigma} \in \mathbb{R}^{p \times 1}$, $X \in \mathbb{R}^{p \times 1}$, $\sigma \in \mathbb{R}^{p \times 1}$, $k \in \mathbb{R}$, $c \in \mathbb{R}$, $H \in \mathbb{R}^{p \times p}$
Is the following equation true?
$$\widehat{\sigma}^T H X - kc\sigma^T X = \begin{pmatrix} \widehat{\sigma} \\ k \end{pmatrix}^T \begin{pmatrix} H \\ -c \sigma^T \end{pmatrix} X.$$ I mean, the dimensions are correct, because both sides are numbers.
Yes, that's true. To parse things out, note that for a block matrix $A = \begin{bmatrix}A_1 \\ A_2\end{bmatrix}$ with $A\in\mathbb{R}^{m\times n}$, $A_1\in\mathbb{R}^{m_1\times n}$, $A_2\in\mathbb{R}^{m_2\times n}$, and $m_1+m_2=m$, we have that \begin{equation*} Ax = \begin{bmatrix}A_1 \\ A_2\end{bmatrix}x = \begin{bmatrix}A_1x \\ A_2x\end{bmatrix} \end{equation*} for a vector $x\in\mathbb{R}^n$. (Convince yourself of this by checking dimensions, computing a simple example, or by writing out the actual multiplication.) Therefore, in your example, \begin{equation*} \begin{bmatrix} H \\ -c\sigma^\top \end{bmatrix}X = \begin{bmatrix} HX \\ -c\sigma^\top X \end{bmatrix}, \end{equation*} which is a $(p+1)\times 1$ vector. Left multiplying by the transpose of your other $(p+1)\times 1$ vector with the same block dimensions, we obtain \begin{equation*} \begin{bmatrix} \hat{\sigma} \\ k \end{bmatrix}^\top \begin{bmatrix} HX \\ -c\sigma^\top X \end{bmatrix} = \begin{bmatrix} \hat{\sigma}^\top & k \end{bmatrix}\begin{bmatrix} HX \\ -c\sigma^\top X \end{bmatrix} = \hat{\sigma}^\top Hx - kc\sigma^\top X. \end{equation*}