I have been using this answer as a guide to find specific point within a shape.
I am using a camera to read the area that is projected on a wall. This first drawing shows the boundary of the image as detected by the camera. Using the coordinates p1, I want to figure out what the coordinate would be for p2 (in the second image).
This is my work. Something is wrong, and I am not sure where I went off the rails. You can see that the values for p2 are negative. That is definitely not anticipated.
Here is more detail on how I calculated lambda, mu, and tao for Gameboard. The formula is repeated for the next two variables.
The matrix for the first image is what I refer to as the boundary (b). The matrix for the second image is referred to as gameboard (g). From there, hopefully you can see where I tried to follow the process listed in the fore mentioned answer.
I am in way over my head on this problem. Can someone please help me find the issue in my logic?
Update:
I found several errors in my matrix multiplication. I have followed the steps listed in the original answer and suggestions from this question, and while I am getting closer, I am still not there. As you can see in the following image, my coordinates are at least in the range of the gameboard, but they appear to be inverse of the expected results.
Your second screenshot shows that to compute $\lambda$ you multiply the first coordinate of the fourth (bottom left) point with each element in the first row of the inverse matrix. That's not how matrix times vector multiplication works: you need to multiply the first column entry with the first point coordinate, the second column entry with the second point coordinate and the third column entry with the third point coordinate. For $\mu$ and $\tau$ you do the same with the second and third row of the matrix.
As I have described in my answer, you can avoid the division by the determinant, and just work with the transposed cofactor matrix (adjugate matrix). So at that point I would compute
$$ \begin{pmatrix}\lambda\\\mu\\\tau\end{pmatrix} = \begin{pmatrix} -667 & -296 & 912919 \\ 528 & -297 & -124410 \\ 139 & 593 & -434122 \end{pmatrix}\cdot \begin{pmatrix}497\\1145\\1\end{pmatrix} = \begin{pmatrix}242500\\-202059\\313946\end{pmatrix} \\ A = \begin{pmatrix} 572\lambda & 1165\mu & 869\tau \\ 598\lambda & 459\mu & 1126\tau \\ 1\lambda & 1\mu & 1\tau \end{pmatrix} = \begin{pmatrix} 138710000 & -235398735 & 272819074 \\ 145015000 & -92745081 & 353503196 \\ 242500 & -202059 & 313946 \end{pmatrix} $$
Any multiple of the above matrix will work just as well. In particular if you divide by the determinant to get the actual inverse instead of the adjugate, you get that division applied to all the entries and end up with just a different multiple of that matrix.
At that point you can verify that all four basis points map as expected. $A\cdot(1,0,0)^T$ must be a multiple of your first point, for $(0,1,0)$ a multiple of the second, and for $(0,0,1)$ a multiple of the third. The most interesting one is probably $(1,1,1)$ yielding a multiple of the fourth. In a spreadsheet you could check this by computing these:
$$ \frac1{A_{31}}\left(A\cdot\begin{pmatrix}1\\0\\0\end{pmatrix}\right) = \frac1{A_{31}}\begin{pmatrix}A_{11}\\A_{21}\\A_{31}\end{pmatrix} = \begin{pmatrix}572\\598\\1\end{pmatrix} \\ \frac1{A_{32}}\left(A\cdot\begin{pmatrix}0\\1\\0\end{pmatrix}\right) = \frac1{A_{32}}\begin{pmatrix}A_{12}\\A_{22}\\A_{32}\end{pmatrix} = \begin{pmatrix}1165\\459\\1\end{pmatrix} \\ \frac1{A_{33}}\left(A\cdot\begin{pmatrix}0\\0\\1\end{pmatrix}\right) = \frac1{A_{33}}\begin{pmatrix}A_{13}\\A_{23}\\A_{33}\end{pmatrix} = \begin{pmatrix}869\\1126\\1\end{pmatrix} \\ \frac1{A_{31}+A_{32}+A_{33}}\left(A\cdot\begin{pmatrix}1\\1\\1\end{pmatrix}\right) = \frac1{A_{31}+A_{32}+A_{33}}\begin{pmatrix} A_{11}+A_{12}+A_{13}\\ A_{21}+A_{22}+A_{23}\\ A_{31}+A_{32}+A_{33} \end{pmatrix} = \begin{pmatrix}497\\1145\\1\end{pmatrix} $$
Mind you, all of the above is just the matrix $A$ from step 2 of the answer you're citing. You still have to do the same for the target points, then multiply the adjugate of one with the other.
I'm willing to share a spreadsheet with the computation for $A$, including the checks.