Given the matrices $\mathbf{A} \in \mathbb{R}^{N \times N}$ and $\mathbf{B} \in \mathbb{R}^{M \times N}$,
\begin{align} \arg \min_{\mathbf{X} \in \mathbb{R}^{N \times M}} \| \mathbf{A} - \mathbf{X} \mathbf{B} \|_\mathrm{F} \end{align}
where $\|\cdot\|_\mathrm{F}$ is the Frobenius norm operator. The above problem can be rewritten as
\begin{align} \arg \min_{\mathrm{vec}(\mathbf{X})} \mathrm{vec}(\mathbf{X})^T (\mathbf{B}\mathbf{B}^T \otimes \mathbf{I}) \mathrm{vec}(\mathbf{X}) - 2 \mathrm{vec}(\mathbf{A} \mathbf{B}^T)^T \mathrm{vec}(\mathbf{X}). \end{align}
where $\otimes$ is the Kronecker product. The above optimization can be solved easily as it is a quadratic program with no constraints. Suppose, we are given prior information that $\mathbf{X}$ is a lower-triangular matrix, how do I impose it as an equality constraint in the form of $\mathbf{C} \mathrm{vec}(\mathbf{X}) = \mathrm{vec}(\mathbf{Y})$ where $\mathbf{C} \in \mathbb{R}^{MN \times MN}$ and $\mathrm{vec}(\mathbf{Y})$ is the vectorized lower-triangular entries of $\mathbf{X}$? In other words, how to determine the entries of matrix $\mathbf{C}$?
Note that I can use CVX in MATLAB to solve this but when the dimensions of the matrices are large, then CVX takes a lot of time for computing.
The problem is given by:
$$ \arg \min_{X \in \mathcal{T} } \frac{1}{2} {\left\| X B - A \right\|}_{F}^{2} $$
Where $ \mathcal{T} $ is the set of Lower Triangular Matrices.
The set $ \mathcal{T} $ is a Convex Set.
Moreover, the orthogonal projection onto the set of a given matrix $ Y \in \mathbb{R}^{m \times n} $ is easy:
$$ X = \operatorname{Proj}_{\mathcal{T}} \left( Y \right) = \operatorname{tril} \left( Y \right) $$
Namely, zeroing all elements above the main diagonal of $ Y $.
By utilizing the Projected Gradient Descent it is easy to solve this problem:
$$ \begin{align*} {X}^{k + 1} & = {X}^{k} - \alpha \left( X B {B}^{T} - A {B}^{T} \right) \\ {X}^{k + 2} & = \operatorname{Proj}_{\mathcal{T}} \left( {X}^{k + 1} \right)\\ \end{align*} $$
The full MATLAB code with CVX validation is available in my StackExchnage Mathematics Q2876283 GitHub Repository.
The solution is very similar to the solution in Q2421545 - Solve Least Squares (Frobenius Norm) Problem with Diagonal Matrix Constraint.
Remark
I think you can also get a closed form solution for each element in $ X $ if you go through deriving the derivative with respect to each element $ X $.
Another approach would be developing the Linear Operator which operates on $ \frac{ \left( n - 1 \right) n }{2} $ elements and creates an $ n \times n $ Triangular Matrix.