I have got three matrices: $A$ (dimension $n \times n$), $B$ (dimension $m \times m$) and $M$ (dimension $m \times n$). We have $m > n$.
This is the relation between these three matrices: $A = M^t B M$
$M$ is lower triangular, $B$ is symmetric.
Is there a way to get the unknown matrix $B$ from the known matrices $A$ and $M$, without explicitly inverting $M$ or $M^t$? What I mean is that I cannot write $B = (M^t)^{-1} A M^{-1}$, I would rather need a sort of backsubstitution method, with an explicit formula for a generic element $B_{i j}$ of $B$.
The way I started thinking about this is that one could start by noticing, after some algebra, that
$B_{m m} = \frac{A_{n n}}{M^t_{nm} M_{m n}} = \frac{A_{n n}}{M_{m n}^2}$
but then I am not able to provide a generic formula for $B_{i j}$.
As Matthias Klubsch pointed out: If $M$ is not square, we cannot find $B$ in a unique way.
Let's split up $M$ and $B$ into blocks: As $M$ is $m \times n$ and lower triangular, we can split it into a lower $n \times n$ block $M_2$ which is lower triangular and an upper $(m-n)\times n$ block which consists entirely of zeros. We split $B$ into blocks of according size:
$$B= \begin{pmatrix}B_{11} & B_{12} \\B_{21} & B_{22}\end{pmatrix}, \ M=\begin{pmatrix}0 \\ M_2 \end{pmatrix},$$
where $B_{11}$ is $(m-n)\times(m-n) $, $B_{12}$ is $n\times(m-n) $, $B_{21}$ is $(n-m)\times n $ and $B_{22}$ is $n\times n$, see the edit below for the definition of the blocks.
Now we can multiply:
$$A=M^tBM = \begin{pmatrix}0 & M_2^t \end{pmatrix}\begin{pmatrix}B_{11} & B_{12} \\B_{21} & B_{22}\end{pmatrix}\begin{pmatrix}0 \\ M_2 \end{pmatrix} = M_2^t B_{22}M_2.$$
This means that the equation $A=M^tBM$ does not contain any information to determine the parts of $B$ which are not in the lower right $n\times n$ corner.
Addtionally, if the square matrix $M_2$ is not of full rank (i.e. has zeros on the diagonal) then we can't even get unique values for $B_{22}$, since with any solution of $M_2^tC_{22}M_2=0$ we would also get a solution $B_{22} + C_{22}$ of $$A= M^t B M = M_2^t (B_{22}+ C_{22})M_2$$
Edit: This splitting is done by defining
$(B_{11})_{ij} = (B)_{ij},\ i,j= 1,\dots m-n$
$(B_{12})_{ij} = (B)_{ij}, \ i= m-n + 1, \dots, m ,\ j= 1,\dots m-n $
$(B_{21})_{ij} = (B)_{ij}, \ j= 1,\dots m-n, \ i= m-n + 1, \dots, m \ , $
$(B_{22})_{ij} = (B)_{ij}, i,j= m-n + 1, \dots, m $
$(M_2)_{ij}= (M)_{ij}, \ i= m-n +1 , \dots, m, \ j=1,\dots, n$
Edit 2: Suppose that $M$ has a different shape than outlined above, namely the upper square matrix $(M)_{ij}, \ i,j =1, \dots, n$ is lower triangular and the residual lower block is not of any particular shape. By the comments, this is what OP meant with lower triangular. (I'd call this lower trapeziodal. But this is semantics.)
We can find a $QR-$decomposition for $M$ such that
$$M = Q R = Q \begin{pmatrix} R_1 \\ 0 \end{pmatrix},$$
where $Q$ is an $m \times m$ orthogonal matrix, $R$ is an $m \times n$ upper triangular matrix and its upper square block $R_1$ is an $n \times n$ upper triangular matrix.
Instead of $A= M^t B M$, we write
$$A = (QR)^t B (QR) = \begin{pmatrix} R_1^t & 0 \end{pmatrix} Q^t B Q \begin{pmatrix} R_1 \\ 0 \end{pmatrix}.$$
We define $B' = Q^t B Q$ (since $Q$ is orthogonal, the task of finding $B$ is equivalent to finding $B'$ and calculating $B = Q B' Q^t$) and split $B'$ into square blocks and find, as above:
$$A = R_1^t B'_{11} R_1,$$
where $(B_{11}')_{ij} = (Q^t B Q)_{ij}, \ i,j, = 1, \dots,n$
Again, with this information, and supposing that $R_1$ has full rank, we can only uniquely determine $B_{11}'$, but not any of the other blocks of $B'$. Thus, we cannot determine $B$ uniquely.